Pillar 1 · Model Selection — family deep dive

Types of Diffusion Models

All diffusion models share one trick — start from noise and remove it step by step until content matching the prompt emerges. The types differ in what they generate (image, video, audio) and how much control you get over the result. The step count is your quality-vs-latency-vs-cost knob.

How generation works — and where the cost comes from

The denoising pipeline

"Latent diffusion" (the Stable Diffusion trick): denoise in a compressed latent space instead of raw pixels — that's why it runs on a single consumer GPU.

The step-count knob: distilled "fast" variants (SDXL-Turbo, FLUX.1 schnell, LCM) generate in 1–4 steps instead of 30–50 — near-real-time and much cheaper, at some quality cost. For interactive UX, preview with a fast variant and upscale/refine the chosen result with the full model.

The types you'll actually choose between

Text-to-image

the headline act

text prompt image

Named models

Stable Diffusion 1.5 / SDXL / SD3.5 (open-weight)
FLUX.1 (open-weight, current quality leader)
DALL·E 3 (API), Midjourney (service)
Imagen (Google), Firefly (Adobe, licensed training data)

Choose it when

Marketing visuals, concept art, mockups
Open-weight (SD/FLUX) if you need self-hosting or NSFW-filter control
Firefly-class when legal wants clean training-data provenance

Watch out

Text rendering inside images is still hit-and-miss
IP/brand review loops belong in the workflow

Image editing: img2img, inpainting, outpainting

generation constrained by an existing image

image + mask + prompt edited image

Named models

SDXL / FLUX inpainting variants
Adobe Generative Fill (Firefly)
DALL·E editing API

Choose it when

Product-shot variants: swap backgrounds, extend scenes
Retouching at scale (e-commerce catalogs)

Watch out

Mask quality drives result quality — pair with segmentation (SAM)
Edited regions can drift from brand reality — human review for claims-sensitive imagery

Controlled generation: ControlNet, LoRA, IP-Adapter

how brands get consistency

prompt + structure (pose/edges/depth) or learned style on-brand image

Named tools

ControlNet (follow a pose, sketch, depth map)
LoRA adapters (a learned style/character/product, trained from ~20–50 images)
IP-Adapter (match a reference image)

Choose it when

Same product/mascot rendered consistently across campaigns
Layouts must follow a template or pose
This is the diffusion equivalent of fine-tuning — cheap and swappable

Watch out

Requires open-weight base models (SD/FLUX) and a pipeline (ComfyUI-class tooling)
LoRA on a person's likeness = consent & rights management

Video generation

frames with temporal consistency

text or image short video clip

Named models

Sora (OpenAI), Veo (Google)
Runway Gen-3, Kling, Pika
Stable Video Diffusion (open-weight)

Choose it when

Short-form ads, product spins, storyboards/previz
Animating existing stills

Watch out

Seconds of output = minutes of compute — budget accordingly
Physics/continuity glitches; plan human curation
Deepfake & disclosure policies (EU AI Act labeling)

Audio, music & speech

the same trick on waveforms

text (± reference voice) speech · music · sound effects

Named models

Stable Audio, AudioLDM (sound/music, diffusion)
Suno, Udio (music services)
ElevenLabs, OpenAI TTS (speech — diffusion + other techniques)

Choose it when

Voice-overs, IVR voices, localized audio at scale
Background music/SFX without licensing friction

Watch out

Voice cloning requires consent workflows
Music outputs: check commercial-use terms per service

Cheat sheet: diffusion type → when to buy it

Type	Named models	Buy it for	Ops note
Text-to-image	SDXL, FLUX.1, DALL·E 3, Midjourney, Firefly	Visuals from briefs	Open-weight = self-host + customize; APIs = zero infra
Editing / inpainting	Generative Fill, SDXL-inpaint	Catalog retouching, background swaps	Pair with SAM for masks
Controlled (ControlNet/LoRA)	ControlNet, LoRA, IP-Adapter	Brand/product consistency	Needs open-weight base + pipeline tooling
Video	Sora, Veo, Runway Gen-3, Kling	Short clips, previz	Expensive per second; curate outputs
Audio / speech	Stable Audio, Suno, ElevenLabs	Voice-overs, music, SFX	Consent + licensing terms per service
Fast/distilled variants	SDXL-Turbo, FLUX schnell, LCM	Interactive previews, high volume	1–4 steps: cheap & quick, slight quality dip

Diffusion vs GAN in one line: diffusion wins on quality and prompt control; GANs win on single-pass speed and niche pipelines (upscaling, synthetic tabular data) — see the GAN types page.