Diffusion Models — All Variants & Tricks

Updated July 2026 with 2025–2026 SOTA additions — new entries marked ★. Algorithm names link to their papers (arXiv / project page).

July 2026 · Updated Edition

Foundations: Forward and Reverse Processes
Parameterizations and Loss Weighting
Variance Schedules
Sampling Algorithms
Conditioning and Guidance
Architectures: U-Net, DiT, MM-DiT
Latent Diffusion
Flow Matching and Rectified Flow
Distillation: Many-Step to One-Step
ControlNet, Adapters, and Conditioning Architectures
Personalization and Customization
Image Editing
The Image-Generation Model Lineup (2021–2026)
Video Diffusion
3D Generation: SDS, VSD, Native 3D Diffusion
Audio and Music Diffusion
Diffusion Beyond Images: Robotics, Motion, Discrete
RL and Alignment for Diffusion
Inference Acceleration and Caching
Specialized Variants and Theory
Watermarking, Safety, Provenance
Evaluation Metrics
Production Stack: 2026 Defaults

Appendix A: Twenty-Five Equations to Memorize

Appendix B: Year-by-year Highlights

1. Foundations: Forward and Reverse Processes

1.1 The forward process (DDPM, Ho et al. 2020)

A fixed Markov chain that progressively adds Gaussian noise:

\[q(\mathbf{x}_t|\mathbf{x}_{t-1}) = \mathcal{N}(\mathbf{x}_t; \sqrt{1-\beta_t}\,\mathbf{x}_{t-1},\, \beta_t I).\]

With \(\alpha_t = 1-\beta_t\) and \(\bar\alpha_t = \prod_{s\le t}\alpha_s\), by induction:

\[q(\mathbf{x}_t|\mathbf{x}_0) = \mathcal{N}(\mathbf{x}_t; \sqrt{\bar\alpha_t}\,\mathbf{x}_0,\, (1-\bar\alpha_t)I).\]

Equivalently:

\[\mathbf{x}_t = \sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1-\bar\alpha_t}\,\boldsymbol\epsilon, \qquad \boldsymbol\epsilon \sim \mathcal{N}(0,I).\]

1.2 The reverse posterior (closed form)

\[q(\mathbf{x}_{t-1}|\mathbf{x}_t,\mathbf{x}_0) = \mathcal{N}(\tilde\mu_t,\, \tilde\beta_t I),\]

\[\tilde\mu_t = \frac{\sqrt{\bar\alpha_{t-1}}\,\beta_t}{1-\bar\alpha_t}\mathbf{x}_0 + \frac{\sqrt{\alpha_t}\,(1-\bar\alpha_{t-1})}{1-\bar\alpha_t}\mathbf{x}_t, \qquad \tilde\beta_t = \frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\beta_t.\]

1.3 Variational lower bound (ELBO)

\[\mathcal{L}_{\text{vlb}} = \mathbb{E}_q\Big[\underbrace{\mathrm{KL}\big(q(\mathbf{x}_T|\mathbf{x}_0)\,\|\,p(\mathbf{x}_T)\big)}_{L_T} + \sum_{t>1}\underbrace{\mathrm{KL}\big(q(\mathbf{x}_{t-1}|\mathbf{x}_t,\mathbf{x}_0)\,\|\,p_\theta(\mathbf{x}_{t-1}|\mathbf{x}_t)\big)}_{L_{t-1}} - \underbrace{\log p_\theta(\mathbf{x}_0|\mathbf{x}_1)}_{L_0}\Big].\]

1.4 The simplified objective

With \(p_\theta(\mathbf{x}_{t-1}|\mathbf{x}_t) = \mathcal{N}(\mu_\theta, \sigma_t^2)\):

\[\mu_\theta(\mathbf{x}_t,t) = \frac{1}{\sqrt{\alpha_t}}\left(\mathbf{x}_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\,\boldsymbol\epsilon_\theta(\mathbf{x}_t,t)\right),\]

the per-step KL reduces to (up to coefficients) \(\|\boldsymbol\epsilon - \boldsymbol\epsilon_\theta\|^2\). Dropping the \(t\)-dependent weighting yields:

Key

Simplified DDPM training objective:

\[\mathcal{L}_{\text{simple}} = \mathbb{E}_{t,\mathbf{x}_0,\boldsymbol\epsilon}\left\|\boldsymbol\epsilon - \boldsymbol\epsilon_\theta\big(\sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1-\bar\alpha_t}\,\boldsymbol\epsilon,\; t\big)\right\|^2.\]

Random \(t \in \{1,\dots,T\}\), random \(\boldsymbol\epsilon\), MSE on noise.

1.5 Score matching equivalence

The score is \(s(\mathbf{x}_t) = \nabla_{\mathbf{x}_t}\log p_t(\mathbf{x}_t)\). For Gaussian \(q(\mathbf{x}_t|\mathbf{x}_0)\):

\[\nabla_{\mathbf{x}_t}\log q(\mathbf{x}_t|\mathbf{x}_0) = -\frac{\mathbf{x}_t - \sqrt{\bar\alpha_t}\,\mathbf{x}_0}{1-\bar\alpha_t} = -\frac{\boldsymbol\epsilon}{\sqrt{1-\bar\alpha_t}}.\]

Therefore: \(\boldsymbol\epsilon_\theta = -\sqrt{1-\bar\alpha_t}\cdot s_\theta(\mathbf{x}_t, t)\). Diffusion = denoising score matching.

1.6 Continuous-time SDE/ODE view (Song et al. 2021)

Forward SDE:

\[d\mathbf{x} = f(\mathbf{x},t)\,dt + g(t)\,d\mathbf{w}.\]

Reverse SDE (Anderson):

\[d\mathbf{x} = \big[f(\mathbf{x},t) - g(t)^2\,\nabla_x\log p_t(\mathbf{x})\big]\,dt + g(t)\,d\bar{\mathbf{w}}.\]

Probability flow ODE (deterministic; same marginals):

\[d\mathbf{x} = \big[f(\mathbf{x},t) - \tfrac{1}{2}g(t)^2\,\nabla_x\log p_t(\mathbf{x})\big]\,dt.\]

1.7 VP vs VE SDE

Variance-Preserving (VP, DDPM): \(f = -\tfrac{1}{2}\beta(t)\mathbf{x}\), \(g = \sqrt{\beta(t)}\). Marginals stay \(\mathcal{N}(\sqrt{\bar\alpha_t}\mathbf{x}_0, (1-\bar\alpha_t)I)\).

Variance-Exploding (VE, NCSN): \(f = 0\), \(g(t)\) grows; marginals are \(\mathcal{N}(\mathbf{x}_0, \sigma_t^2 I)\) with \(\sigma_t\) growing without bound.

Sub-VP: \(f\) as VP, \(g = \sqrt{\beta(t)\big(1 - e^{-2\int\beta}\big)}\).

2. Parameterizations and Loss Weighting

2.1 Three equivalent targets

\(\boldsymbol\epsilon\)-prediction: predict the added noise.
\(\mathbf{x}_0\)-prediction: predict the clean image.
v-prediction: \(v_t = \sqrt{\bar\alpha_t}\,\boldsymbol\epsilon - \sqrt{1-\bar\alpha_t}\,\mathbf{x}_0\).

Conversion:

\[\mathbf{x}_0 = \frac{\mathbf{x}_t - \sqrt{1-\bar\alpha_t}\,\boldsymbol\epsilon}{\sqrt{\bar\alpha_t}}, \qquad \boldsymbol\epsilon = \frac{\mathbf{x}_t - \sqrt{\bar\alpha_t}\,\mathbf{x}_0}{\sqrt{1-\bar\alpha_t}}.\]

2.2 Why v-prediction?

Target's scale doesn't blow up at \(t\to 0\) (\(v\approx -\mathbf{x}_0\)) or \(t\to T\) (\(v\approx\boldsymbol\epsilon\)). Stable across noise levels; the standard for distillation and modern video diffusion.

2.3 SNR-aware loss weighting

Define \(\mathrm{SNR}(t) = \bar\alpha_t/(1-\bar\alpha_t)\). The simplified loss is biased toward high-SNR (low \(t\)). Reweight:

Min-SNR-\(\gamma\) (Hang et al.): scale loss by \(\min(\mathrm{SNR}(t), \gamma)/\mathrm{SNR}(t)\), \(\gamma\in[1,5]\). Faster convergence.
Karras EDM: continuous-\(\sigma\) formulation with explicit \(c_{\text{skip}}\), \(c_{\text{out}}\), \(c_{\text{in}}\), \(c_{\text{noise}}\) pre-conditioning so the network outputs are scale-invariant.
P2 weighting: emphasizes mid-noise regimes that contribute most perceptually.

2.4 Karras EDM preconditioning

\[F_\theta(\mathbf{x},\sigma) = c_{\text{skip}}(\sigma)\,\mathbf{x} + c_{\text{out}}(\sigma)\,\hat F_\theta\big(c_{\text{in}}(\sigma)\,\mathbf{x},\; c_{\text{noise}}(\sigma)\big),\]

with \(c_{\text{skip}} = \sigma_{\text{data}}^2/(\sigma^2 + \sigma_{\text{data}}^2)\), \(c_{\text{out}} = \sigma\sigma_{\text{data}}/\sqrt{\sigma^2+\sigma_{\text{data}}^2}\), \(c_{\text{in}} = 1/\sqrt{\sigma^2+\sigma_{\text{data}}^2}\), \(c_{\text{noise}} = \tfrac{1}{4}\log\sigma\).

3. Variance Schedules

3.1 Linear (DDPM original)

\(\beta_t\) linear from \(10^{-4}\) to \(0.02\) over \(T = 1000\) steps. Simple but oversamples high-noise.

3.2 Cosine (Improved DDPM, Nichol & Dhariwal)

\[\bar\alpha_t = \frac{f(t)}{f(0)}, \qquad f(t) = \cos\!\left(\frac{t/T + s}{1+s}\cdot\frac{\pi}{2}\right)^2, \quad s = 0.008.\]

Smoother low-noise transitions; better for high-resolution and complex distributions.

3.3 Sigmoid, EDM, continuous-σ

Sigmoid: \(\bar\alpha_t = \sigma(\text{linear in }t)\), used in some recent papers. EDM: continuous-\(\sigma\) with \(\sigma \sim \mathrm{LogNormal}(P_{\text{mean}}, P_{\text{std}}^2)\) during training; default \(P_{\text{mean}} = -1.2\), \(P_{\text{std}} = 1.2\).

3.4 Shifted schedules for higher resolution

Higher-resolution images need lower SNR at the same time \(t\) to look "equally noisy". Shifted schedules apply a constant SNR offset \(\log \mathrm{SNR}(t) \to \log \mathrm{SNR}(t) - 2\log(s)\) where \(s\) is the spatial scale ratio. Used in SD3 and FLUX for multi-aspect-ratio training.

4. Sampling Algorithms

4.1 Ancestral sampling (DDPM)

\[\mathbf{x}_{t-1} = \mu_\theta(\mathbf{x}_t,t) + \sigma_t\,\mathbf{z}, \quad \mathbf{z}\sim\mathcal{N}(0,I).\]

\[\sigma_t^2 = \beta_t \;\text{ or }\; \sigma_t^2 = \tilde\beta_t.\]

4.2 DDIM (deterministic / partially stochastic)

Non-Markov forward with the same marginals. Update:

\[\mathbf{x}_{t-1} = \sqrt{\bar\alpha_{t-1}}\,\hat{\mathbf{x}}_0 + \sqrt{1-\bar\alpha_{t-1}-\sigma_t^2}\,\boldsymbol\epsilon_\theta(\mathbf{x}_t,t) + \sigma_t\mathbf{z},\]

where \(\hat{\mathbf{x}}_0 = (\mathbf{x}_t - \sqrt{1-\bar\alpha_t}\,\boldsymbol\epsilon_\theta)/\sqrt{\bar\alpha_t}\). \(\sigma_t = 0\): deterministic, supports inversion. \(\eta\in[0,1]\) controls stochasticity (\(\sigma_t = \eta\,\tilde\beta_t^{1/2}\)).

4.3 PNDM, DPM-Solver, DPM-Solver++

PNDM (Pseudo Numerical Methods): Adams–Bashforth on the diffusion ODE.

DPM-Solver / DPM-Solver++ (Lu et al.): high-order ODE solvers exploiting the semi-linear analytic structure of the diffusion ODE. Multistep (\(k\)th order) variants reach high quality in \(\sim 10\text{–}20\) steps.

DPM-Solver-3M-SDE: stochastic third-order multistep; better quality than DDIM at very few steps for some models.

4.4 UniPC

Unified predictor-corrector: \(\sim 5\text{–}10\) steps to match many-step quality. State of the art among non-distilled samplers.

4.5 Euler, Heun, Karras schedule

Euler is first-order; Heun second-order. Karras et al. EDM proposes:

\[\sigma_i = \left(\sigma_{\max}^{1/\rho} + \frac{i}{N-1}\big(\sigma_{\min}^{1/\rho} - \sigma_{\max}^{1/\rho}\big)\right)^\rho, \quad \rho = 7,\]

which produces near-optimal step spacing. Heun + Karras schedule is competitive at \(\sim 30\) NFEs.

4.6 Restart sampling

After the ODE samples down to \(\sigma_{\min}\), re-noise back to \(\sigma_{\max'} > \sigma_{\min}\) and re-sample. Restarts inject controlled stochasticity, improving FID at additional cost.

4.7 Few-step distilled samplers

Covered in §9: LCM, DMD, DMD2, Hyper-SD, SDXL Turbo, Lightning, Phased Consistency.

4.8 NFE cheat table

Sampler	Typical NFEs	Notes
DDPM	1000	original, slow
DDIM	20–50	deterministic, supports inversion
PNDM	50	legacy SD default
DPM-Solver++	10–20	good speed/quality balance
UniPC	5–10	state-of-art non-distilled
Heun + Karras	EDM 25–50	SDXL-class quality
LCM	4–8	distilled
DMD2	1–4	one-step possible
Hyper-SD / Lightning	1–8	distilled SDXL/FLUX

5. Conditioning and Guidance

5.1 Class conditioning

Add label embedding to time embedding; inject via FiLM (scale+shift) or adaLN.

5.2 Cross-attention to text

Text tokens encoded by frozen CLIP / T5 / SigLIP; image tokens cross-attend at each U-Net / DiT block.

Standard since GLIDE / Stable Diffusion.

5.3 adaLN, adaLN-Zero (DiT)

Predict per-block scale + shift from condition \(c\) (timestep + class / text):

\[\mathrm{adaLN}(x,c) = \gamma(c)\frac{x-\mu}{\sigma} + \beta(c).\]

adaLN-Zero: also predict \(\alpha(c)\) on the residual; initialize to 0:

\[y = x + \alpha(c)\cdot f(\mathrm{adaLN}(x,c)).\]

Block is identity at init; conditioning is learned smoothly.

5.4 Classifier guidance

Use a separately-trained classifier \(p(y|\mathbf{x}_t)\):

\[\nabla_{\mathbf{x}_t}\log p(\mathbf{x}_t|y) = \nabla_{\mathbf{x}_t}\log p(\mathbf{x}_t) + \nabla_{\mathbf{x}_t}\log p(y|\mathbf{x}_t).\]

Augment the score with \(w\,\nabla\log p(y|\mathbf{x}_t)\). Requires a noise-aware classifier; cumbersome.

5.5 Classifier-free guidance (CFG)

Train a single network \(\boldsymbol\epsilon_\theta(\mathbf{x}_t, t, c)\), randomly drop \(c\to\varnothing\) with prob \(p_{\text{drop}}\sim 0.1\). At inference:

\[\tilde{\boldsymbol\epsilon}(\mathbf{x}_t,t,c) = \boldsymbol\epsilon_\theta(\mathbf{x}_t,t,\varnothing) + w\big(\boldsymbol\epsilon_\theta(\mathbf{x}_t,t,c) - \boldsymbol\epsilon_\theta(\mathbf{x}_t,t,\varnothing)\big).\]

Equivalent (asymptotically) to sampling \(p(\mathbf{x}|c)\propto p(\mathbf{x})\,p(c|\mathbf{x})^w\) via Bayes log trick. \(w > 1\) amplifies conditioning, sharper images, less diversity. Default in essentially every modern diffusion model.

5.6 Modern CFG variants

CFG++: corrects DDIM step using guidance only on the score component, not on the manifold projection step. Less over-saturation.
Autoguidance (Karras et al. 2024): replace the unconditional with a smaller / weaker model. Improves quality with no \(w>1\) artifacts.
APG (Adaptive Projected Guidance): project the guidance direction to fix the negative-prompt artifact.
PAG (Perturbed-Attention Guidance): replace the attention matrix with the identity in one layer; use the difference as guidance. Works without CFG.
SAG (Self-Attention Guidance): blur high-attention regions; guide away from the blurred prediction.

5.7 Negative prompts

At inference, encode a negative prompt \(c^-\) in place of the unconditional. The CFG step then steers toward \(c\) and away from \(c^-\):

\[\tilde{\boldsymbol\epsilon} = \boldsymbol\epsilon_\theta(\mathbf{x}_t,t,c^-) + w\big(\boldsymbol\epsilon_\theta(\mathbf{x}_t,t,c) - \boldsymbol\epsilon_\theta(\mathbf{x}_t,t,c^-)\big).\]

Used routinely in SD-class models.

5.8 Dynamic thresholding (Imagen)

At each step, clamp \(\hat{\mathbf{x}}_0\) to a percentile range to prevent saturation at high CFG. Specifically,

\[\hat{\mathbf{x}}_0 \leftarrow \mathrm{clamp}(\hat{\mathbf{x}}_0,\, -s,\, s)\,/\,\max(1,\, s/p)\]

with \(s = \text{percentile}\).

6. Architectures: U-Net, DiT, MM-DiT

6.1 U-Net (DDPM, SD 1/2/XL)

Encoder downsampling + decoder upsampling with skip connections; each block has self-attention + cross-attention to text. Standard for SD 1/2/XL.

SDXL specifics: 2.6B U-Net parameters, additional pooled-CLIP context (size + crop conditioning), Stage 1 + Stage 2 refiner.

6.2 Cascaded diffusion (Imagen, Stable Cascade)

Generate at low resolution first; then super-resolve with conditional diffusion. Imagen: \(64\to256\to1024\) cascade.

Stable Cascade: text \(\to\) low-dim latent \(\to\) image latent \(\to\) image. Reduces training cost; high-resolution stage is conditional and shallower.

6.3 DiT (Diffusion Transformer)

Replace U-Net with Transformer over noised latent tokens. adaLN-Zero conditioning. Cleaner scaling than U-Net; matches SD U-Net at \(\sim 2\times\) params.

6.4 PixArt-α / Σ / δ

DiT with cross-attention to T5. PixArt-\(\alpha\): small + efficient; \(\Sigma\): 4K resolution; \(\delta\): LCM distilled. Open frontier line.

6.5 MM-DiT (SD3, FLUX)

Two parallel streams (text, image), each with own \(W_Q, W_K, W_V, W_O\) and MLP, interacting through one joint self-attention over concatenated tokens:

\[[Q_{\text{txt}}; Q_{\text{img}}],\quad [K_{\text{txt}}; K_{\text{img}}],\quad [V_{\text{txt}}; V_{\text{img}}].\]

Cleaner cross-modal coupling than U-Net cross-attention. Standard for new diffusion releases.

6.6 Lumina-T2I, Hunyuan-DiT

LLaMA-style backbone (Lumina) or Hunyuan's own backbone, both DiT-class. 2D-RoPE on image tokens.

6.7 Sana (NVIDIA, 2024)

DiT with linear attention + deep-compression VAE (\(32\times\) spatial vs SD's \(8\times\)). 4K text-to-image at sub-second on consumer GPU. Aggressive efficiency play.

6.8 Stable Cascade vs SD3 vs FLUX

Model	Architecture	Loss	Highlight
SD 1.5	U-Net + CLIP	DDPM (\(\boldsymbol\epsilon\))	860M U-Net
SDXL	U-Net + 2×CLIP	DDPM (\(\boldsymbol\epsilon\))	2.6B + refiner
Stable Cascade	3-stage cascade	DDPM (\(\boldsymbol\epsilon\))	efficient training
SD 3.5	MM-DiT improvements	RF	better data, bigger
FLUX.1	MM-DiT	RF + few-step distill	dev/schnell/pro/Kontext
Sana	Linear DiT	RF	4K, fast

★ 2026 SOTA update — Autoregressive image / next-scale prediction

VAR (Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction): Recasts image AR as coarse-to-fine 'next-scale (next-resolution) prediction' instead of raster next-token prediction, letting GPT-style AR transformers surpass DiT on ImageNet 256x256 (FID 18.65 -> 1.73, IS 80.4 -> 350.2) with ~20x faster inference. Exhibits LLM-like power-law scaling and zero-shot inpainting/outpainting/editing. The reference point for the 2025-2026 autoregressive-image / next-scale line that now sits alongside diffusion (and underpins instruction-edit AR variants); the sheet only covers MaskGIT/MAGVIT-style token models.

7. Latent Diffusion

7.1 The latent diffusion idea (LDM, Rombach 2021)

Train an autoencoder \(E, D\) to compress images to low-dim latents (typically \(8\times\) spatial \(\to 64\times\) fewer FLOPs in diffusion). Run diffusion in latent space:

\[\mathcal{L} = \mathbb{E}_{z,t,\boldsymbol\epsilon,c}\left\|\boldsymbol\epsilon - \boldsymbol\epsilon_\theta\big(\sqrt{\bar\alpha_t}\,z + \sqrt{1-\bar\alpha_t}\,\boldsymbol\epsilon,\; t,\; c\big)\right\|^2.\]

Decode: \(\hat{x} = \mathcal{D}(\hat{z})\).

7.2 Autoencoder regularization

KL-regularized: \(E\) outputs \((\mu, \sigma)\); sample \(z \sim \mathcal{N}(\mu, \sigma^2)\). Used by SD's KL-VAE.
VQ-regularized: VQ-GAN-style discrete codebook on latents. Used in early LDM, Parti.
Both with adversarial + perceptual (LPIPS) losses for sharpness.

7.3 SD's autoencoder (the f = 8, C = 4)

\(8\times\) spatial downsampling, 4 latent channels. SD3 / FLUX use \(f = 8\), \(C = 16\) (more latent channels for higher fidelity).

7.4 Pixel-space vs latent

Latent: standard for new image / video diffusion. Cheaper, higher quality at same compute.

Pixel-space: occasionally used for very small images / scientific data where the autoencoder loss matters.

8. Flow Matching and Rectified Flow

8.1 Continuous Normalizing Flow refresher

A vector field \(u_t : \mathbb{R}^d \to \mathbb{R}^d\) generates a probability path \(p_t\) via the ODE:

\[d\mathbf{x}_t = u_t(\mathbf{x}_t)\,dt, \quad \mathbf{x}_0 \sim p_0.\]

Continuity equation: \(\partial_t p_t + \nabla\cdot(p_t u_t) = 0\). Densities computable via instantaneous change of variables.

8.2 Conditional Flow Matching (Lipman et al. 2023)

Given a marginal path between \(p_0\) (prior) and \(p_1\) (data), pick conditional paths \(p_t(x|x_1)\) with known conditional vector field \(u_t(x|x_1)\). Train:

\[\mathcal{L}_{\text{CFM}} = \mathbb{E}_{t,\mathbf{x}_1,\mathbf{x}\sim p_t(\cdot|\mathbf{x}_1)}\left\|v_\theta(\mathbf{x},t) - u_t(\mathbf{x}|\mathbf{x}_1)\right\|^2.\]

8.3 Affine paths ⇒ the simple FM loss

For \(\mathbf{x}_t = (1-t)\mathbf{x}_0 + t\mathbf{x}_1\) with \(\mathbf{x}_0 \sim \mathcal{N}(0,I)\):

\[u_t(\mathbf{x}|\mathbf{x}_1) = \mathbf{x}_1 - \mathbf{x}_0.\]

Key

Flow matching training objective:

\[\mathcal{L}_{\text{FM}} = \mathbb{E}_{t\sim U(0,1),\, \mathbf{x}_0\sim p_0,\, \mathbf{x}_1\sim p_{\text{data}}}\left\|v_\theta\big((1-t)\mathbf{x}_0 + t\mathbf{x}_1,\; t\big) - (\mathbf{x}_1 - \mathbf{x}_0)\right\|^2.\]

Same architecture as diffusion, simpler target.

8.4 Rectified Flow (Liu et al.)

Iteratively re-pair \((\mathbf{x}_0, \mathbf{x}_1)\) along the learned ODE, then re-train. Trajectories straighten; few-step (or one-step) sampling becomes possible. Underlies SD3 and FLUX.

8.5 Stochastic interpolants

General framework subsuming diffusion and FM as special cases. Unified treatment of various noise / interpolation schedules.

8.6 Why FM is replacing diffusion

Cleaner straight-line sampling, fewer NFEs at high quality.
Easier distillation (rectified-flow pairs are direct teachers).
Same architecture; simpler loss and noise schedule.
No need for many-step ODE solvers; Euler often suffices.

By 2026, FM is the default for new image/video model releases (SD3, FLUX, MovieGen, Cosmos).

9. Distillation: Many-Step to One-Step

9.1 Progressive distillation (Salimans & Ho)

Train a student to imitate two teacher steps in one. Iterate \(\log_2 T\) times to halve steps. Slow but reliable.

9.2 Consistency Models (Song et al.)

Train \(f_\theta(\mathbf{x}_t, t)\approx\mathbf{x}_0\) for any \(t\) along the same ODE trajectory:

\[\mathcal{L}_{\text{CM}} = \mathbb{E}\,d\big(f_\theta(\mathbf{x}_{t_{n+1}}, t_{n+1}),\; f_{\theta^-}(\hat{\mathbf{x}}_{t_n}, t_n)\big),\]

EMA target \(\theta^-\). 1–4 step inference. LCM (Latent Consistency Models): same in latent space; the standard distillation for SDXL through 2024.

9.3 DMD / DMD2 (Distribution Matching Distillation)

Train a one-step student by matching the score field of the teacher:

\[\mathcal{L}_{\text{DMD}} = \mathbb{E}\Big[\big(s_{\text{real}}(\hat{x}) - s_{\text{fake}}(\hat{x})\big)\cdot\partial\hat{x}/\partial\theta\Big],\]

plus a regression loss in DMD; in DMD2 the regression is dropped and a GAN-style discriminator is added.

Reaches teacher quality in one step.

9.4 Hyper-SD, Phased Consistency Models (PCM)

Hyper-SD: blend consistency + adversarial + ODE-trajectory losses; few-step SDXL/FLUX with high quality.

PCM: divide the ODE into \(K\) segments; apply consistency within each. Recovers quality at very few NFEs without quality cliff.

9.5 Score Identity Distillation (SiD)

One-step distillation that doesn't need a trajectory teacher. Uses a Stein-style score-matching identity to derive the loss.

9.6 InstaFlow / Reflow

Use the rectified-flow recipe to straighten then distill. 1–2 step text-to-image at SDXL quality.

9.7 LADD (Latent Adversarial Diffusion Distillation, SD Turbo)

Train a 1–4 step student with discriminator in latent space; teacher is the original SDXL.

9.8 Few-step model lineup (2026)

SDXL Turbo / Lightning / DMD2 / Hyper-SD / PCM-SDXL: 1–4 step SDXL.
FLUX schnell: 4-step FLUX out of the box.
LCM-FLUX, Hyper-FLUX: distilled FLUX variants.
One-step text-to-image is increasingly viable for production.

Watch out

Distilled models are lower-diversity than their teachers. For best aesthetic / creative range, sample with the original. For latency-critical apps (chat, real-time editing), distilled wins.

★ 2026 SOTA update — Few-step / one-step generation

MeanFlow (Mean Flows for One-step Generative Modeling): Introduces the average-velocity (mean flow) field vs. the instantaneous velocity of Flow Matching, with a closed-form MeanFlow identity relating the two used directly as a training target. Self-contained: no pre-training, distillation, or curriculum. Reaches FID 3.43 at 1-NFE on ImageNet 256x256 trained from scratch, closing much of the gap between one-step and multi-step models and spawning follow-ups (pixel-space, discrete, rectified-trajectory variants). A key 2025 native few-step trick to add alongside the sheet's Consistency/LCM/DMD2 coverage.

★ 2026 SOTA update — Consistency / few-step

sCM / TrigFlow (Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models): TrigFlow unifies EDM and Flow Matching in a trigonometric parameterization, then stabilizes continuous-time consistency model (sCM) training via identity time transforms, positional time embeddings, and adaptive normalization -- eliminating the discretization error of prior discrete-step CMs. Scales stably to 1.5B params; two-step sampling reaches FID 1.48 on ImageNet 64x64 and 1.88 on ImageNet 512x512, comparable to the teacher diffusion model. The theoretical backbone behind the current wave of continuous-time few-step distillation.

★ 2026 SOTA update — Few-step / flow-map

Shortcut Models (One Step Diffusion via Shortcut Models): Conditions the velocity network on step size as well as noise level, so a single network trained in one phase can sample in 1 step or many -- no multi-phase schedule, extra networks, or fragile distillation. A precursor of the 'flow-map' family (any-to-any noise-level jumps); consistently beats consistency models and reflow across step budgets. Missing from the sheet's distillation section.

★ 2026 SOTA update — Flow-map distillation

Align Your Flow (Scaling Continuous-Time Flow Map Distillation): Defines flow maps that connect any two noise levels in one step, generalizing both consistency and flow-matching objectives, and introduces two new continuous-time flow-map training losses. Unlike consistency models (whose quality degrades as steps increase), flow maps stay strong across all step counts. Adds autoguidance-during-distillation and adversarial finetuning for a further boost. Represents the 2025 SOTA framing of the 'flow-map / shortcut / sCM' cluster the task asks for.

10. ControlNet, Adapters, and Conditioning Architectures

10.1 ControlNet (Zhang & Agrawala 2023)

Clone the encoder of the diffusion U-Net into a trainable copy; feed the control input (canny, depth, pose, segmentation, scribble, normal map, etc.). Connections back to the base U-Net use zero-initialized \(1\times1\) convs so the base model is undisturbed at start.

10.2 T2I-Adapter

Lightweight: a few conv blocks predict feature offsets added to the base U-Net's intermediate features. Less power than ControlNet, much fewer parameters.

10.3 ControlNet++, ControlNeXt, Uni-ControlNet

ControlNet++: cycle-consistency loss (re-extract control from generation, match to input). ControlNeXt: lighter ControlNet with curriculum training. Uni-ControlNet: one model with multiple control modalities.

10.4 IP-Adapter

Image as condition. Decoupled cross-attention: parallel image cross-attn path with own \(W_K, W_V\), conditioned on CLIP-encoded reference. Personalization without fine-tuning.

10.5 ReferenceNet (AnimateAnyone, MagicAnimate)

Trainable U-Net copy on the reference image; KVs of the main U-Net's self-attention concatenated with reference KVs. Identity preservation for video.

10.6 Single-shot face personalization

InstantID, PhotoMaker, PuLID: face encoder + attention / feature injection. No fine-tuning per person.

10.7 ControlNet for video (CogVideoX-Control, others)

Apply ControlNet-style branch to video diffusion U-Net / DiT, with control video (depth maps over time, pose sequences).

11. Personalization and Customization

11.1 DreamBooth

Fine-tune the full diffusion model on \(\sim 5\) images of a subject with a unique identifier token ([V] dog) and a prior-preservation loss to avoid forgetting:

\[L = L_{\text{recon}}(\text{subject}) + \lambda\, L_{\text{recon}}(\text{class images, generated by base model}).\]

Slow but high quality. Works on SDXL, FLUX, etc.

11.2 Textual Inversion

Learn only a new embedding for a new token; freeze the model. Cheap; lower fidelity than DreamBooth.

11.3 LoRA

Low-rank update \(W \to W + BA\), \(A\in\mathbb{R}^{r\times d}\), \(B\in\mathbb{R}^{d\times r}\), \(r\ll d\). Train only \(A, B\). Standard for SDXL / FLUX customization. Stackable: combine multiple LoRAs at inference (with weights).

11.4 OFT, BOFT

Orthogonal Fine-Tuning: \(W \to RW\) with \(R\in SO(d)\) parameterized by Cayley transform. Preserves spectrum, better identity preservation. BOFT: block-diagonal \(R\) for compute savings.

11.5 Custom Diffusion, Mix-of-Show

Multi-concept personalization: train multiple LoRAs, then merge or compose at inference. Mix-of-Show handles attribute leakage.

11.6 HyperDreamBooth

A hypernetwork predicts personalization parameters from a reference image in one shot. No per-subject finetuning.

12. Image Editing

12.1 SDEdit (img2img)

Add noise to source image up to step \(t^*\), then denoise with new prompt. Higher \(t^*\): more change vs source.

Default for "img2img" in every diffusion UI.

12.2 Inpainting / outpainting

Mask-conditioned variant: condition on (image, mask, prompt). At inference, denoise only inside mask region; blend with original outside via noise composition. SD Inpainting, FLUX Fill are dedicated checkpoints.

12.3 Prompt-to-Prompt (P2P)

Manipulate cross-attention maps to do edits:

Word swap: replace cross-attn of one word with another, keep rest.
Re-weight: scale cross-attn map of a target token.
Add token: inject cross-attn for a new token.

12.4 Null-text inversion

Fix DDIM inversion drift by per-step optimization of the unconditional embedding so that DDIM forward + reverse exactly reconstructs. Combined with P2P for high-quality real-image editing.

12.5 InstructPix2Pix (instruction-based)

Train a conditional diffusion model on \((I_{\text{src}}, \text{instruction}, I_{\text{target}})\) triples generated by GPT-3 + Stable Diffusion.

At inference: condition on source image and natural-language instruction.

12.6 Imagic

Per-image fine-tune the model on the source + interpolate text embeddings between source caption and edit caption.

12.7 MagicBrush, InstructEdit, OmniEdit, AnyEdit

Larger / cleaner instruction-edit datasets (MagicBrush manually curated, OmniEdit programmatic + curated).

2024–2025 generation pushes edit quality to commercial-grade.

12.8 FLUX Kontext (2025)

First-class image editing in FLUX backbone: instruction + source image \(\to\) edited image, end-to-end without inversion gymnastics. Sets new bar for editing quality.

13. The Image-Generation Model Lineup (2021–2026)

13.1 Era I: pixel-space & early latent

GLIDE (OpenAI 2021): pixel-space diffusion + CLIP guidance / classifier-free guidance. Established CFG.
DALL-E 2 / unCLIP (2022): CLIP latent prior + decoder.
Imagen (Google 2022): pixel-space cascade with T5-XXL text encoder; demonstrated text encoder > image-only.
LDM / Stable Diffusion 1.4/1.5 (2022): latent diffusion, open-sourced. Catalyzed the open-source ecosystem.

13.2 Era II: SD-class scaling

SD 2.0/2.1: OpenCLIP text encoder, v-prediction.
SDXL (2023): 2.6B U-Net + refiner, multi-aspect-ratio training, size + crop conditioning.
SDXL Turbo / Lightning: distilled few-step variants.
DALL-E 3 (2023): GPT-4 prompt rewriter + improved diffusion backbone.

13.3 Era III: DiT and flow matching

PixArt-\(\alpha/\Sigma\) (2023–24): DiT + T5; open frontier.
SD3 / 3.5 (2024): MM-DiT + rectified flow + CLIP+T5; 2B / 8B variants.
FLUX.1 (Black Forest Labs, 2024): MM-DiT + RF; dev / schnell / pro / Kontext editor. Dominant open frontier.
Stable Cascade: 3-stage cascade for efficient training.
Sana (NVIDIA 2024): linear-attention DiT \(+32\times\) VAE for 4K speed.
Lumina-T2I, Hunyuan-DiT, CogView: open Chinese-led DiT variants.

13.4 Era IV: closed frontier

DALL-E 3, GPT Image (OpenAI).
Imagen 2 / 3 / 4 (Google).
Midjourney v5/v6/v7 (Midjourney): closed, top-of-class aesthetic.
Ideogram 1/2/3: typography-strong closed.
Adobe Firefly: licensed-data trained, enterprise.
Recraft V3, Flux Pro Ultra: high-end specialty.

★ 2026 SOTA update — Latest image model + editing

Qwen-Image (and Qwen-Image-Edit / Qwen-Image-Edit-2509): 20B MM-DiT foundation model (Aug 2025) with SOTA complex text rendering -- notably strong on logographic Chinese via a curriculum from no-text to paragraph-level rendering -- and a multi-task paradigm (T2I + TI2I + I2I reconstruction) for high-consistency editing. Ships Qwen-Image-Edit, and Qwen-Image-Edit-2509 adds multi-image editing (person+person, person+product, person+scene). A major open 2025 model/editor absent from both the lineup and editing sections (which stop at FLUX Kontext / SD 3.5).

★ 2026 SOTA update — Latest efficient image model

Lumina-Image 2.0 (A Unified and Efficient Image Generative Framework): Unified Next-DiT that treats text and image tokens as one joint sequence for native cross-modal interaction, plus a Unified Captioner (UniCap) for dense captions and multi-stage progressive training with inference acceleration. Matches much larger models with only 2.6B params -- an efficiency-focused open successor to Lumina-T2I/Next that the sheet doesn't list.

14. Video Diffusion

14.1 Architecture pattern

Causal 3D VAE encodes \(V\in\mathbb{R}^{T\times H\times W\times 3}\) to latents \(Z\in\mathbb{R}^{T'\times H'\times W'\times C}\) (typically \(T' = T/4\), \(H' = H/8\)).

Spatiotemporal MM-DiT operates on \(Z\) with 3D positional encodings. Loss: same diffusion / FM target.

14.2 Joint image + video training

A still image is the \(T = 1\) case. With token packing, the same model trains on both. Often weighted:

\[L = \lambda_I\,\mathcal{L}_{\text{img}} + \lambda_V\,\mathcal{L}_{\text{video}}.\]

14.3 The open lineup

ModelScope T2V (2023): early open.
AnimateDiff (2023): plug temporal attention modules into SD U-Net; uses pretrained SD knowledge.
SVD / SVD-XT (Stability 2023): image-to-video with SD backbone + temporal modules.
ZeroScope: ModelScope improvements.
CogVideoX / 1.5 (Tsinghua 2024): DiT, T5 conditioning, 5–10 second clips.
Open-Sora / Open-Sora-Plan: community Sora replications.
Mochi 1 (Genmo 2024): fully open; 10B AsymmDiT.
Hunyuan Video (Tencent 2024): 13B, frontier-open quality.
LTX-Video (Lightricks 2024): real-time on A100.
Wan 2.1 (Alibaba 2025): open frontier.
Step-Video, Allegro: other open frontier.

14.4 Closed frontier

Sora / Sora 2 (OpenAI 2024–25): 60s+ clips, reportedly DiT.
Veo 2 / Veo 3 (Google 2024–25): with audio gen.
Runway Gen-3 / Gen-4 (2024–25).
Kling 1.6 / 2 (Kuaishou 2024–25): photoreal motion.
Pika 1 / 2.
MovieGen (Meta 2024): joint video + audio.
MiniMax Hailuo, Luma Dream Machine, Vidu.

14.5 Image-to-video, video-to-video, animation

AnimateAnyone: drive a static character with a pose video.
MimicMotion: motion transfer.
Champ, MagicAnimate: subject-driven video.
Live Portrait: image + driving video \(\to\) animated portrait.
EMO, Audio2Photoreal: audio-driven talking head.

14.6 Long-video and consistency

Long videos > 10 s are the open problem. Strategies:

Chunked autoregressive: generate clip; condition next on last frames.
Causal temporal training: train with causal mask so model can extend.
Anchor-frame conditioning: keyframes provided; in-between generated.
World-model framing: train on action-conditioned video for closed-loop simulation.

14.7 Audio-video joint generation

MovieGen audio model, Veo 3 (with audio), MMAudio, V2A models: video and audio jointly diffused, with cross-modal coupling. By 2026 the new release norm.

14.8 Video evaluation

FVD: \(\|\mu_r - \mu_g\|^2 + \mathrm{tr}\big(\Sigma_r + \Sigma_g - 2(\Sigma_r\Sigma_g)^{1/2}\big)\) on I3D / InternVideo features. VBench / VBench-2: 16+ axes (motion smoothness, dynamic degree, subject consistency, etc.). T2V-CompBench: compositional. VideoScore: learned reward. Human preference still dominant.

★ 2026 SOTA update — Video diffusion frontier

Wan 2.2 (Open MoE Video Diffusion): First open-source Mixture-of-Experts video diffusion model: a high-noise expert handles early denoising and a low-noise expert refines details, raising capacity at ~constant per-step compute (A14B = 27B total / 14B active). Adds curated cinematic aesthetic labels (lighting, composition, color tone), a 5B TI2V model that runs on consumer GPUs, and audio-driven (speech-to-video) generation. Successor to the sheet's Wan 2.1 and a key 2025 open-video frontier release; released as a model repo without a standalone arXiv paper (URL is the official repo, verified).

15. 3D Generation: SDS, VSD, Native 3D Diffusion

15.1 Score Distillation Sampling (DreamFusion, Poole et al.)

Optimize a parametric scene \(\theta\) (NeRF or 3DGS) by passing renders \(\mathbf{x} = g(\theta)\) through a frozen 2D diffusion teacher:

\[\nabla_\theta\mathcal{L}_{\text{SDS}} = \mathbb{E}_{t,\boldsymbol\epsilon}\big[w(t)(\boldsymbol\epsilon_\phi(\mathbf{x}_t,t,c) - \boldsymbol\epsilon)\,\partial\mathbf{x}/\partial\theta\big].\]

Slow (hours per scene), prone to Janus problem (multi-face), saturated colors.

15.2 Variational SDS (VSD, ProlificDreamer)

Replace noise target \(\boldsymbol\epsilon\) by a learned variational distribution \(\hat{q}\). Reduces mode collapse; sharper details.

15.3 Multi-view diffusion: MVDream, Wonder3D, Zero123/++

Train a 2D diffusion model that generates multiple consistent views of an object given one image (or text). Use multi-view outputs to optimize 3D representation. Dramatically reduces SDS Janus problem.

Zero-1-to-3 / Zero123++ / Stable Zero123 / SyncDreamer: variants on novel-view synthesis with diffusion priors.

15.4 LRM family (feed-forward triplane)

LRM (2023): single image \(\to\) triplane neural rep in one transformer pass. Trained on Objaverse + Objaverse-XL.

InstantMesh, MeshLRM, GS-LRM, Long-LRM: variants producing meshes / Gaussians, single or multiview.

15.5 Native 3D diffusion (the 2025 wave)

Trellis (Microsoft): structured 3D latent + flow matching; seconds per asset.
Hunyuan3D-2: native 3D diffusion, mesh + texture in one pass.
CLAY: volumetric latent diffusion.
Direct3D, SF3D, SPAR3D: variants.
Rodin Gen-1.5: 3D-asset generation product.

15.6 Mesh generation Transformers (autoregressive)

MeshGPT, MeshXL, MeshAnything V2, EdgeRunner, BPT: autoregressive face-vertex generation with VQ tokens or sequence-style. Native mesh topology output without isosurface extraction.

Key

By 2025, native 3D diffusion / FM (Trellis, Hunyuan3D-2) replaces SDS optimization for production 3D asset generation. SDS remains useful for scene-level / unbounded outputs.

16. Audio and Music Diffusion

16.1 AudioLDM / AudioLDM 2

Latent diffusion on audio; text conditioning. Two-stage pretraining (alignment + generation). Good general TTA (text-to-audio) baseline.

16.2 Stable Audio / Stable Audio Open

Latent diffusion DiT conditioned on CLAP embeddings + duration. Stable Audio Open released for community use.

16.3 DiffWave, Grad-TTS

Diffusion vocoders / TTS. DiffWave: predict raw waveform conditioned on mel. Grad-TTS: continuous-time diffusion TTS.

16.4 Diff-SVC, VC variants

Singing-voice / voice conversion via diffusion.

16.5 MusicGen, AudioGen (LM, not diffusion)

Token-based AR Transformers on EnCodec tokens. Listed for contrast: not all generative audio is diffusion.

17. Diffusion Beyond Images: Robotics, Motion, Discrete

17.1 Diffusion Policy (robotics)

Generate the action sequence \(a_{t:t+H}\) with a conditional diffusion model on observation history:

\[\mathcal{L} = \mathbb{E}_{k,\boldsymbol\epsilon,a_0,o}\left\|\boldsymbol\epsilon - \boldsymbol\epsilon_\theta\big(\sqrt{\bar\alpha_k}\,a_0 + \sqrt{1-\bar\alpha_k}\,\boldsymbol\epsilon,\; k,\; o\big)\right\|^2.\]

Multi-modal action distributions (which unimodal Gaussian heads can't represent) are critical for real manipulation.

17.2 3D Diffuser Actor, RDT-1B

3D Diffuser Actor: condition on 3D scene token. RDT-1B: 1B-param diffusion policy pretrained across 46 datasets.

17.3 π0 flow-matching action head

VLM backbone with small flow-matching action head \(v_\theta(a_t, t, o)\):

\[\mathcal{L}_{\text{FM}} = \mathbb{E}\left\|v_\theta(a_t, t, o) - (a_1 - a_0)\right\|^2.\]

Continuous actions, fine manipulation. Standard in Physical Intelligence's stack.

17.4 Motion synthesis

MotionDiffuse, MDM (Motion Diffusion Model), PhysDiff: diffuse human motion sequences. Used for animation, gaming.

17.5 Discrete diffusion (text)

D3PM (Discrete Denoising Diffusion Probabilistic Model): forward chain on categorical data via transition matrix; reverse predicted by neural net.

Diffusion-LM, SSD-LM, GENIE: text generation via diffusion on continuous embeddings.

MaskGIT, MAGE, MAGVIT: parallel decoding via masked language modeling, related to discrete diffusion.

Show-o uses MaskGIT-style for image and AR for text in one model.

17.6 World models

Genie, Genie 2 (DeepMind): action-conditioned video diffusion = playable simulator. Cosmos World Foundation Models (NVIDIA): general world-model platform.

★ 2026 SOTA update — Unified generation + understanding

BAGEL (Emerging Properties in Unified Multimodal Pretraining): Open decoder-only Mixture-of-Transformer-Experts model (7B active / 14B total) pretrained on trillions of interleaved text-image-video-web tokens; unifies multimodal understanding and generation and shows emergent free-form image manipulation, future-frame prediction, 3D manipulation, and world navigation. A flagship 2025 open unified gen-understanding model to add beyond the sheet's Show-o mention (companion to closed GPT-Image and DeepSeek Janus-Pro).

18. RL and Alignment for Diffusion

18.1 DDPO (Denoising Diffusion Policy Optimization)

Cast diffusion as multi-step MDP (each denoising step is an action). Use PPO with reward = aesthetic / preference score on the final image. Per-step policy gradient through the sampler.

18.2 DPOK

Per-batch PPO with KL to base model. Simpler than DDPO.

18.3 Diffusion-DPO

Lift DPO to diffusion. Preferences over images \((x_w, x_l|c)\):

\[\mathcal{L}_{\text{D-DPO}} = -\mathbb{E}\,\log\sigma\!\Big(-\beta\big(\mathcal{L}_{\theta}(x^w,c) - \mathcal{L}_{\theta_{\text{ref}}}(x^w,c) - \mathcal{L}_\theta(x^l,c) + \mathcal{L}_{\theta_{\text{ref}}}(x^l,c)\big)\Big),\]

\(\mathcal{L}_\theta(x, c)\) = standard diffusion loss on \(x\). Now standard for aesthetic alignment of SDXL / SD3 / FLUX.

18.4 DRaFT, AlignProp, ReFL

Reward backpropagation through the sampler. The full diffusion ODE / SDE is differentiable (with checkpointing); compute \(\partial r(\hat{x})/\partial\theta\) end-to-end. Memory-heavy; works for short samplers (e.g., 25-step DDIM). ReFL (Reward Feedback Learning) is the canonical citation.

18.5 Reward models for image preferences

ImageReward: trained on \(\sim 137\text{k}\) human comparisons.
HPSv2/v3: human preference scores trained on web ratings.
PickScore: trained on Pick-a-Pic dataset.
VQAScore: VQA-on-generated; tests compositional fidelity.

19. Inference Acceleration and Caching

19.1 DeepCache (block caching)

Diffusion U-Nets compute similar features at adjacent timesteps. Cache deep block outputs and reuse for \(k\) steps before refreshing. \(\sim 2\text{–}4\times\) speedup.

19.2 TGATE (cross-attention gating)

Cross-attention contributes most early in sampling; freeze cross-attention output after step \(\tau^*\). \(\sim 1.5\times\) speedup with no quality loss.

19.3 PAB (Pyramid Attention Broadcast, video)

For video diffusion: cache and broadcast attention outputs across consecutive timesteps and across spatial neighbors. Video-specific \(2\times\) speedup.

19.4 Block-Cache (FLUX)

Cache attention outputs across consecutive denoising steps; recompute every \(k\) steps. Same idea as DeepCache adapted to MM-DiT.

19.5 Token Merging for SD (ToMe-SD)

Merge highly similar tokens in self-attention layers via bipartite soft matching. SD inference \(\sim 2\times\) speedup at minimal quality loss.

19.6 Quantization (Q-Diffusion, PTQ4DM, Olive-VLM, SVDQuant)

PTQ for diffusion: per-step calibration because activation distributions shift dramatically across timesteps.
Q-Diffusion, PTQ4DM: 8-bit / 4-bit U-Net.
SVDQuant: low-rank decomposition + 4-bit quantization for FLUX.

19.7 Compile and kernel optimization

torch.compile, TensorRT, Stable-Fast. Fused VAE decode. Latent caching (encode once with VAE, store latents).

19.8 LCM / DMD2 / Lightning at inference

The fastest path is fewer steps. Distilled samplers (§9) achieve 1–4 step inference at production quality.

20. Specialized Variants and Theory

20.1 Discrete / multinomial diffusion

For categorical data (text, code, masks): forward chain swaps token to a special masked / random state. D3PM, MaskGIT, MAGVIT. Inference: parallel masked decoding.

20.2 Cascaded super-resolution

Imagen, eDiff-I: low-res \(\to\) mid-res \(\to\) high-res, each conditional on the previous. Trains each stage independently; modular.

20.3 Schrödinger Bridge / Diffusion Bridges

Generalize the diffusion forward to arbitrary distributions (not necessarily Gaussian noise). Useful for image-to-image where you want to start from a structured prior. I2SB (Image-to-Image Schrödinger Bridge).

20.4 Self-conditioning

At each step, also condition on the previous step's \(\hat{\mathbf{x}}_0\) prediction. Improves sample quality at modest cost. Used in some 3D / video models.

20.5 Min-SNR weighting

Up-weight low-SNR steps (early in reverse process):

\[w(t) = \min(\mathrm{SNR}(t), \gamma)/\mathrm{SNR}(t).\]

Faster training convergence by \(2\text{–}3\times\).

20.6 EDM (Elucidating the Design Space, Karras et al. 2022)

Continuous-\(\sigma\) formulation; preconditioning of network inputs/outputs; LogNormal noise sampling; Heun + Karras-spaced steps. The cleanest theoretical framework for diffusion.

20.7 Diffusion forcing

Asynchronous noise levels per token / per frame. Enables long-horizon / autoregressive video generation by training on partially noised sequences.

21. Watermarking, Safety, Provenance

21.1 Stable Signature

Fine-tune VAE decoder so that generated images carry a \(\sim 48\)-bit watermark recoverable by a small extractor.

Survives JPEG compression, crops.

21.2 Tree-Ring Watermarks

Embed a circular pattern in the noise latent; re-detect via DDIM inversion + Fourier check. Robust to standard image edits.

21.3 Gaussian Shading

Embed message bits in the latent's Gaussian distribution; detect via sign pattern after inversion.

21.4 C2PA

Industry standard for content provenance metadata (signed). Adopted by Adobe, Microsoft, OpenAI, Google.

21.5 Deepfake / synthetic detection

Continual arms race. Detection via: forensic features (GAN/diffusion frequency artifacts), VLM-based reasoning, watermark-presence checks.

22. Evaluation Metrics

22.1 FID (Fréchet Inception Distance)

\[\mathrm{FID} = \|\mu_r - \mu_g\|^2 + \mathrm{tr}\big(\Sigma_r + \Sigma_g - 2(\Sigma_r\Sigma_g)^{1/2}\big),\]

on Inception-V3 features. Standard since 2017; thoroughly broken by modern models.

22.2 IS, KID, Precision/Recall

IS: classifier confidence vs class diversity. Largely deprecated. KID: kernel-based MMD, less biased than FID at small sample. Precision/Recall on manifold: separates fidelity from diversity.

22.3 CLIP-Score, BLIP-Score

Cosine of generated image with text in CLIP / BLIP space. Gameable; correlates weakly with human preference.

22.4 ImageReward, HPSv2/v3, PickScore

Reward models trained on human comparisons. Better correlation with humans, but reward-hacking remains.

22.5 VQAScore, GenEval, T2I-CompBench, GenAI-Bench

Compositional benchmarks: counting, attribute binding, spatial relations. VQAScore uses a frozen VLM as judge.

22.6 Image Arena

Chatbot-Arena style. Pairwise human votes; Elo / Bradley–Terry ratings. Closest to ground truth but expensive.

22.7 Video: VBench / VBench-2 / EvalCrafter

16+ axes: motion smoothness, dynamic degree, subject consistency, scene transition, object class, temporal flicker. Plus FVD.

23. Production Stack: 2026 Defaults

Use case	Default architecture	Notes
Open T2I (frontier)	FLUX.1 dev / SD 3.5 (MM-DiT + flow matching)	1–4 step distilled for production
Closed T2I (frontier)	DALL-E / Imagen / Midjourney	Closed model API
Mobile / on-device	T2I SDXL Turbo / SD 3.5 quantized / Sana FP8	W4 quantization
Image editing	FLUX Kontext / OmniEdit	end-to-end edit, no inversion
Personalization	DreamBooth-LoRA / IP-Adapter / InstantID	one LoRA per concept
Spatial control	ControlNet++ / Uni-ControlNet	canny/depth/pose/seg
Open T2V	Hunyuan Video / Wan 2.1 / Mochi 1	5–10 s @ 720p
Closed T2V (frontier)	Sora 2 / Veo 3 / Kling 2 / Runway Gen-4	long clips, audio in some
Open audio gen	Stable Audio Open / AudioLDM 2	text-to-audio
3D asset gen	Trellis / Hunyuan3D-2 / CLAY	native 3D diffusion / FM
Robotics policy	Diffusion Policy / RDT-1B / π0	FM head multi-modal action distributions
Alignment	Diffusion-DPO + ImageReward/HPS	SD3/FLUX aesthetic tuning