Diffusion Models — Step-by-Step Derivations — SDEs, ODEs & More

Updated July 2026 with 2025–2026 SOTA additions — new entries marked ★. Algorithm names link to their papers (arXiv / project page).

July 2026 · Updated Edition

Contents

1. Preliminaries and Notation
2. Score Matching: The Foundation
3. DDPM: Forward, Reverse, and the ELBO
4. DDIM: Non-Markovian Forwards and Deterministic Sampling
5. The SDE / ODE Picture (Song et al., 2021)
6. Flow Matching
7. Rectified Flow
8. Guidance: Classifier and Classifier-Free
9. ELBO ↔︎ Score-Matching Equivalence
10. Consistency Models
11. One-Page Cheat Sheet

1. Preliminaries and Notation

We write a data sample as \(\mathbf{x}\in\mathbb{R}^d\) drawn from an unknown distribution \(p_{\text{data}}(\mathbf{x})\). Throughout we use bold for vectors, \(\mathbf{s}(\mathbf{x})=\nabla_{\mathbf{x}}\log p(\mathbf{x})\) for the (Stein) score of a density \(p\), and \(\mathcal{N}(\mu,\Sigma)\) for a Gaussian density with mean \(\mu\) and covariance \(\Sigma\). The identity matrix is \(\mathbf{I}\).

We use \(\mathbb{E}_p[\cdot]\) for expectation under \(p\), \(\mathrm{KL}(p\,\|\,q)\) for the Kullback–Leibler divergence, and \(\nabla,\nabla^2,\nabla\cdot,\Delta\) for gradient, Hessian, divergence, and Laplacian respectively.

A handful of identities are used repeatedly.

Gaussian log-density gradient. For \(p(\mathbf{x})=\mathcal{N}(\mu,\sigma^2\mathbf{I})\),

\[\log p(\mathbf{x}) = -\frac{1}{2\sigma^2}\|\mathbf{x}-\mu\|^2 + \text{const}, \qquad \nabla_{\mathbf{x}}\log p(\mathbf{x}) = -\frac{\mathbf{x}-\mu}{\sigma^2}. \tag{1}\]

Reparameterization of a Gaussian. If \(\mathbf{x}\sim\mathcal{N}(\mu,\Sigma)\) then \(\mathbf{x}=\mu+L\boldsymbol{\epsilon}\) with \(\boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\) and \(LL^\top=\Sigma\).

KL between two Gaussians (same dimension, diagonal isotropic).

\[\mathrm{KL}\big(\mathcal{N}(\mu_1,\sigma_1^2\mathbf{I})\,\big\|\,\mathcal{N}(\mu_2,\sigma_2^2\mathbf{I})\big) = \frac{1}{2}\left[\frac{d\,\sigma_1^2}{\sigma_2^2} + \frac{\|\mu_1-\mu_2\|^2}{\sigma_2^2} - d + d\log\frac{\sigma_2^2}{\sigma_1^2}\right]. \tag{2}\]

Itô SDE and Fokker–Planck. For an Itô SDE \(d\mathbf{x}=f(\mathbf{x},t)\,dt+g(t)\,d\mathbf{w}\) with \(\mathbf{w}\) a standard Wiener process, the marginal density \(p_t(\mathbf{x})\) evolves under the Fokker–Planck (Kolmogorov forward) equation

\[\partial_t p_t(\mathbf{x}) = -\nabla\cdot\big(f(\mathbf{x},t)\,p_t(\mathbf{x})\big) + \tfrac{1}{2}g(t)^2\,\Delta p_t(\mathbf{x}). \tag{3}\]

2. Score Matching: The Foundation

2.1 The goal

We wish to learn a parametric score \(\mathbf{s}_\theta(\mathbf{x})\approx\nabla_{\mathbf{x}}\log p_{\text{data}}(\mathbf{x})\). The score is appealing because it does not depend on the (unknown) normalizing constant \(Z\) of \(p_{\text{data}}\): if \(p_{\text{data}}(\mathbf{x})=\tilde{p}(\mathbf{x})/Z\) then

\[\nabla\log p_{\text{data}} = \nabla\log\tilde{p}.\]

A natural loss is the Fisher divergence

\[J_{\text{ESM}}(\theta) = \frac{1}{2}\,\mathbb{E}_{p_{\text{data}}}\Big[\big\|\mathbf{s}_\theta(\mathbf{x}) - \nabla_{\mathbf{x}}\log p_{\text{data}}(\mathbf{x})\big\|^2\Big], \tag{4}\]

called explicit score matching (ESM). The catch: we do not know \(\nabla\log p_{\text{data}}\).

2.2 Hyvärinen's identity (implicit score matching)

Theorem 2.1 (Hyvärinen, 2005). Under mild regularity conditions (the densities decay so boundary terms vanish), the Fisher divergence (4) equals, up to an additive constant independent of \(\theta\),

\[J_{\text{ISM}}(\theta) = \mathbb{E}_{p_{\text{data}}}\Big[\tfrac{1}{2}\|\mathbf{s}_\theta(\mathbf{x})\|^2 + \mathrm{tr}\big(\nabla_{\mathbf{x}}\mathbf{s}_\theta(\mathbf{x})\big)\Big]. \tag{5}\]

Proof. Expand the square in (4):

\[J_{\text{ESM}}(\theta) = \tfrac{1}{2}\mathbb{E}_{p_{\text{data}}}\|\mathbf{s}_\theta\|^2 - \mathbb{E}_{p_{\text{data}}}\big[\mathbf{s}_\theta(\mathbf{x})^\top\nabla\log p_{\text{data}}(\mathbf{x})\big] + \tfrac{1}{2}\mathbb{E}_{p_{\text{data}}}\|\nabla\log p_{\text{data}}\|^2.\]

Only the cross term depends on the unknown score. Writing it out and using \(p_{\text{data}}\nabla\log p_{\text{data}}=\nabla p_{\text{data}}\),

\[-\mathbb{E}_{p_{\text{data}}}\big[\mathbf{s}_\theta^\top\nabla\log p_{\text{data}}\big] = -\int p_{\text{data}}(\mathbf{x})\,\mathbf{s}_\theta(\mathbf{x})^\top\nabla\log p_{\text{data}}(\mathbf{x})\,d\mathbf{x} = -\int \mathbf{s}_\theta(\mathbf{x})^\top\nabla p_{\text{data}}(\mathbf{x})\,d\mathbf{x}.\]

Integration by parts (componentwise) yields

\[-\int s_{\theta,i}\,\partial_i p_{\text{data}}\,d\mathbf{x} = -\underbrace{\big[s_{\theta,i}\,p_{\text{data}}\big]_{-\infty}^{\infty}}_{=0} + \int p_{\text{data}}\,\partial_i s_{\theta,i}\,d\mathbf{x} = \mathbb{E}_{p_{\text{data}}}[\partial_i s_{\theta,i}].\]

Summing \(i=1,\dots,d\) gives \(\mathbb{E}_{p_{\text{data}}}[\mathrm{tr}(\nabla\mathbf{s}_\theta)]\). The last (constant in \(\theta\)) term in \(J_{\text{ESM}}\) is dropped, giving (5). \(\square\)

The trace of the Jacobian is impractical in high dimensions (each component requires a separate backward pass). Two remedies follow.

2.3 Denoising score matching (Vincent, 2011)

Perturb the data with a small Gaussian kernel \(q_\sigma(\tilde{\mathbf{x}}\mid\mathbf{x})=\mathcal{N}(\mathbf{x},\sigma^2\mathbf{I})(\tilde{\mathbf{x}})\) and define the noised marginal \(q_\sigma(\tilde{\mathbf{x}})=\int q_\sigma(\tilde{\mathbf{x}}\mid\mathbf{x})p_{\text{data}}(\mathbf{x})\,d\mathbf{x}\).

Theorem 2.2 (Vincent, 2011). Up to a constant independent of \(\theta\),

\[\begin{aligned} J_{\text{ESM}_\sigma}(\theta) &\equiv \tfrac{1}{2}\mathbb{E}_{q_\sigma(\tilde{\mathbf{x}})}\big[\|\mathbf{s}_\theta(\tilde{\mathbf{x}}) - \nabla_{\tilde{\mathbf{x}}}\log q_\sigma(\tilde{\mathbf{x}})\|^2\big]\\ &= \tfrac{1}{2}\mathbb{E}_{p_{\text{data}}(\mathbf{x})\,q_\sigma(\tilde{\mathbf{x}}\mid\mathbf{x})}\big[\|\mathbf{s}_\theta(\tilde{\mathbf{x}}) - \nabla_{\tilde{\mathbf{x}}}\log q_\sigma(\tilde{\mathbf{x}}\mid\mathbf{x})\|^2\big] = J_{\text{DSM}}(\theta). \end{aligned} \tag{6}\]

Proof. Expand both sides and compare cross terms. The squared-norm of \(\mathbf{s}_\theta\) terms agree. The constant terms differ by a quantity independent of \(\theta\). It remains to show the cross terms are equal.

For the marginal version,

\[-\mathbb{E}_{q_\sigma(\tilde{\mathbf{x}})}\big[\mathbf{s}_\theta^\top\nabla\log q_\sigma(\tilde{\mathbf{x}})\big] = -\int \mathbf{s}_\theta(\tilde{\mathbf{x}})^\top\nabla q_\sigma(\tilde{\mathbf{x}})\,d\tilde{\mathbf{x}}.\]

Use \(\nabla q_\sigma(\tilde{\mathbf{x}})=\int p_{\text{data}}(\mathbf{x})\,\nabla_{\tilde{\mathbf{x}}}q_\sigma(\tilde{\mathbf{x}}\mid\mathbf{x})\,d\mathbf{x}\) and the identity \(\nabla q=q\nabla\log q\) for \(p_{\text{data}}(\mathbf{x})\),

\[= -\iint p_{\text{data}}(\mathbf{x})\,q_\sigma(\tilde{\mathbf{x}}\mid\mathbf{x})\,\mathbf{s}_\theta(\tilde{\mathbf{x}})^\top\nabla_{\tilde{\mathbf{x}}}\log q_\sigma(\tilde{\mathbf{x}}\mid\mathbf{x})\,d\tilde{\mathbf{x}}\,d\mathbf{x},\]

which is exactly the conditional cross term.

For the isotropic Gaussian kernel, (1) gives \(\nabla_{\tilde{\mathbf{x}}}\log q_\sigma(\tilde{\mathbf{x}}\mid\mathbf{x})=-(\tilde{\mathbf{x}}-\mathbf{x})/\sigma^2\), so writing \(\tilde{\mathbf{x}}=\mathbf{x}+\sigma\boldsymbol{\epsilon}\) with \(\boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\),

\[J_{\text{DSM}}(\theta) = \tfrac{1}{2}\,\mathbb{E}_{\mathbf{x},\boldsymbol{\epsilon}}\left[\left\|\mathbf{s}_\theta(\mathbf{x}+\sigma\boldsymbol{\epsilon}) + \frac{\boldsymbol{\epsilon}}{\sigma}\right\|^2\right]. \tag{7}\]

This single line is the workhorse of every diffusion model: learn to denoise, and the optimum equals the (marginal) score.

Remark 2.3 (Score ↔︎ noise parameterization). If we let \(\mathbf{s}_\theta(\tilde{\mathbf{x}})=-\boldsymbol{\epsilon}_\theta(\tilde{\mathbf{x}})/\sigma\), then (7) becomes the familiar mean-squared error on noise \(J=\frac{1}{2\sigma^2}\mathbb{E}\|\boldsymbol{\epsilon}_\theta(\tilde{\mathbf{x}})-\boldsymbol{\epsilon}\|^2\). DDPM and the score-SDE view are two faces of the same loss.

2.4 Sliced score matching (brief)

Sliced SM replaces the trace in (5) by a Hutchinson estimator: for \(\mathbf{v}\sim p_{\mathbf{v}}\) (e.g. Rademacher), \(\mathrm{tr}(\nabla\mathbf{s}_\theta)=\mathbb{E}_{\mathbf{v}}[\mathbf{v}^\top(\nabla\mathbf{s}_\theta)\mathbf{v}]\), giving an unbiased single-sample estimator computable with one Jacobian–vector product. It avoids the kernel bias of DSM at the cost of higher variance.

3. DDPM: Forward, Reverse, and the ELBO

3.1 The forward (noising) process

DDPM [4] fixes a Markov chain

\[q(\mathbf{x}_t\mid\mathbf{x}_{t-1}) = \mathcal{N}\big(\sqrt{1-\beta_t}\,\mathbf{x}_{t-1},\,\beta_t\mathbf{I}\big), \quad t=1,\dots,T, \tag{8}\]

with \(0<\beta_1<\cdots<\beta_T<1\). Define \(\alpha_t=1-\beta_t\) and \(\bar{\alpha}_t=\prod_{s\le t}\alpha_s\).

Closed-form marginal \(q(\mathbf{x}_t\mid\mathbf{x}_0)\). By induction, \(\mathbf{x}_t=\sqrt{\alpha_t}\,\mathbf{x}_{t-1}+\sqrt{\beta_t}\,\boldsymbol{\epsilon}_t\) with \(\boldsymbol{\epsilon}_t\sim\mathcal{N}(\mathbf{0},\mathbf{I})\) independent. Apply this \(t\) times. Because sums of independent Gaussians are Gaussian,

\[q(\mathbf{x}_t\mid\mathbf{x}_0) = \mathcal{N}\big(\sqrt{\bar{\alpha}_t}\,\mathbf{x}_0,\,(1-\bar{\alpha}_t)\mathbf{I}\big), \qquad \mathbf{x}_t = \sqrt{\bar{\alpha}_t}\,\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_t}\,\boldsymbol{\epsilon}. \tag{9}\]

Inductive proof of (9). True for \(t=1\) since \(\bar{\alpha}_1=\alpha_1=1-\beta_1\). Assume true for \(t-1\): \(\mathbf{x}_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\,\mathbf{x}_0+\sqrt{1-\bar{\alpha}_{t-1}}\,\bar{\boldsymbol{\epsilon}}\). Then

\[\mathbf{x}_t = \sqrt{\alpha_t}\sqrt{\bar{\alpha}_{t-1}}\,\mathbf{x}_0 + \sqrt{\alpha_t(1-\bar{\alpha}_{t-1})}\,\bar{\boldsymbol{\epsilon}} + \sqrt{1-\alpha_t}\,\boldsymbol{\epsilon}_t.\]

The two independent zero-mean Gaussian terms sum to a Gaussian with variance \(\alpha_t(1-\bar{\alpha}_{t-1})+(1-\alpha_t)=1-\bar{\alpha}_t\) since \(\alpha_t\bar{\alpha}_{t-1}=\bar{\alpha}_t\). \(\square\)

As \(T\to\infty\) (and \(\bar{\alpha}_T\to0\)), \(q(\mathbf{x}_T)\approx\mathcal{N}(\mathbf{0},\mathbf{I})\): data is destroyed into pure Gaussian noise.

3.2 The tractable posterior \(q(\mathbf{x}_{t-1}\mid\mathbf{x}_t,\mathbf{x}_0)\)

The Markov chain is irreversible in closed form for \(q(\mathbf{x}_{t-1}\mid\mathbf{x}_t)\), but conditioning on \(\mathbf{x}_0\) makes it tractable. By Bayes' rule

\[q(\mathbf{x}_{t-1}\mid\mathbf{x}_t,\mathbf{x}_0) = q(\mathbf{x}_t\mid\mathbf{x}_{t-1},\mathbf{x}_0)\,\frac{q(\mathbf{x}_{t-1}\mid\mathbf{x}_0)}{q(\mathbf{x}_t\mid\mathbf{x}_0)}. \tag{10}\]

The numerator is the product of two Gaussians in \(\mathbf{x}_{t-1}\):

\[\log q(\mathbf{x}_t\mid\mathbf{x}_{t-1}) = -\frac{1}{2\beta_t}\|\mathbf{x}_t-\sqrt{\alpha_t}\,\mathbf{x}_{t-1}\|^2 + C,\]

\[\log q(\mathbf{x}_{t-1}\mid\mathbf{x}_0) = -\frac{1}{2(1-\bar{\alpha}_{t-1})}\|\mathbf{x}_{t-1}-\sqrt{\bar{\alpha}_{t-1}}\,\mathbf{x}_0\|^2 + C. \tag{11}\]

Add them and complete the square in \(\mathbf{x}_{t-1}\). Collecting terms,

\[q(\mathbf{x}_{t-1}\mid\mathbf{x}_t,\mathbf{x}_0) = \mathcal{N}\big(\tilde{\mu}_t(\mathbf{x}_t,\mathbf{x}_0),\,\tilde{\beta}_t\mathbf{I}\big), \qquad \tilde{\mu}_t(\mathbf{x}_t,\mathbf{x}_0) = \frac{\sqrt{\bar{\alpha}_{t-1}}\,\beta_t}{1-\bar{\alpha}_t}\,\mathbf{x}_0 + \frac{\sqrt{\alpha_t}\,(1-\bar{\alpha}_{t-1})}{1-\bar{\alpha}_t}\,\mathbf{x}_t, \tag{12}\]

\[\tilde{\beta}_t = \frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_t}\,\beta_t. \tag{13}\]

Re-expressing the mean in terms of noise. From (9), \(\mathbf{x}_0=(\mathbf{x}_t-\sqrt{1-\bar{\alpha}_t}\,\boldsymbol{\epsilon})/\sqrt{\bar{\alpha}_t}\). Substitute into (12) and simplify:

\[\tilde{\mu}_t(\mathbf{x}_t,\mathbf{x}_0) = \frac{1}{\sqrt{\alpha_t}}\left(\mathbf{x}_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\,\boldsymbol{\epsilon}\right). \tag{14}\]

3.3 The reverse (generative) model and the ELBO

Model the reverse chain as \(p_\theta(\mathbf{x}_{t-1}\mid\mathbf{x}_t)=\mathcal{N}\big(\mu_\theta(\mathbf{x}_t,t),\,\Sigma_\theta(\mathbf{x}_t,t)\big)\), with \(p(\mathbf{x}_T)=\mathcal{N}(\mathbf{0},\mathbf{I})\). Joint density

\[p_\theta(\mathbf{x}_{0:T}) = p(\mathbf{x}_T)\prod_{t=1}^{T} p_\theta(\mathbf{x}_{t-1}\mid\mathbf{x}_t).\]

Variational lower bound. Following the standard VAE argument with latent \(\mathbf{x}_{1:T}\) and "encoder" \(q(\mathbf{x}_{1:T}\mid\mathbf{x}_0)\),

\[\log p_\theta(\mathbf{x}_0) = \log\int p_\theta(\mathbf{x}_{0:T})\,d\mathbf{x}_{1:T} \ge \mathbb{E}_{q(\mathbf{x}_{1:T}\mid\mathbf{x}_0)}\log\frac{p_\theta(\mathbf{x}_{0:T})}{q(\mathbf{x}_{1:T}\mid\mathbf{x}_0)},\]

so the negative ELBO is

\[L = -\mathbb{E}_q\log\frac{p_\theta(\mathbf{x}_{0:T})}{q(\mathbf{x}_{1:T}\mid\mathbf{x}_0)}.\]

Expanding each factor and using the Markov property \(q(\mathbf{x}_t\mid\mathbf{x}_{t-1})=q(\mathbf{x}_t\mid\mathbf{x}_{t-1},\mathbf{x}_0)\) together with the Bayes identity (10) rewrites this into the famous Sohl-Dickstein decomposition:

\[L = \underbrace{\mathrm{KL}\big(q(\mathbf{x}_T\mid\mathbf{x}_0)\,\|\,p(\mathbf{x}_T)\big)}_{L_T} + \sum_{t=2}^{T}\underbrace{\mathbb{E}_q\,\mathrm{KL}\big(q(\mathbf{x}_{t-1}\mid\mathbf{x}_t,\mathbf{x}_0)\,\|\,p_\theta(\mathbf{x}_{t-1}\mid\mathbf{x}_t)\big)}_{L_{t-1}} - \underbrace{\mathbb{E}_q\log p_\theta(\mathbf{x}_0\mid\mathbf{x}_1)}_{L_0}. \tag{15}\]

Sketch. Write \(\log p_\theta(\mathbf{x}_{0:T})-\log q(\mathbf{x}_{1:T}\mid\mathbf{x}_0)\) as \(\log p(\mathbf{x}_T)+\sum_t\log p_\theta(\mathbf{x}_{t-1}\mid\mathbf{x}_t)-\sum_t\log q(\mathbf{x}_t\mid\mathbf{x}_{t-1})\). For \(t\ge2\) replace \(q(\mathbf{x}_t\mid\mathbf{x}_{t-1})\) by \(q(\mathbf{x}_{t-1}\mid\mathbf{x}_t,\mathbf{x}_0)\,q(\mathbf{x}_t\mid\mathbf{x}_0)/q(\mathbf{x}_{t-1}\mid\mathbf{x}_0)\). Telescoping the \(q(\mathbf{x}_t\mid\mathbf{x}_0)/q(\mathbf{x}_{t-1}\mid\mathbf{x}_0)\) chain collapses to \(q(\mathbf{x}_T\mid\mathbf{x}_0)/q(\mathbf{x}_1\mid\mathbf{x}_0)\), and grouping terms recovers (15).

\(L_T\) is constant in \(\theta\) (the model's \(p(\mathbf{x}_T)\) is fixed). \(L_0\) is a discretized reconstruction term. The interesting terms are \(L_{t-1}\) for \(t\ge2\).

3.4 From ELBO to noise prediction

DDPM chooses \(\Sigma_\theta=\sigma_t^2\mathbf{I}\) (with \(\sigma_t^2=\beta_t\) or \(\tilde{\beta}_t\), both nearly optimal) so that \(L_{t-1}\) is a KL between two Gaussians with the same covariance. Using (2),

\[L_{t-1} = \mathbb{E}_q\left[\frac{1}{2\sigma_t^2}\|\tilde{\mu}_t(\mathbf{x}_t,\mathbf{x}_0) - \mu_\theta(\mathbf{x}_t,t)\|^2\right] + \text{const}. \tag{16}\]

Choose the parameterization that matches the posterior. Inspired by (14), parameterize

\[\mu_\theta(\mathbf{x}_t,t) = \frac{1}{\sqrt{\alpha_t}}\left(\mathbf{x}_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\,\boldsymbol{\epsilon}_\theta(\mathbf{x}_t,t)\right). \tag{17}\]

Subtracting (14) from (17) gives \(\tilde{\mu}_t-\mu_\theta=\frac{\beta_t}{\sqrt{\alpha_t(1-\bar{\alpha}_t)}}(\boldsymbol{\epsilon}_\theta-\boldsymbol{\epsilon})\). Plugging into (16),

\[L_{t-1} = \mathbb{E}_{\mathbf{x}_0,\boldsymbol{\epsilon}}\left[\frac{\beta_t^2}{2\sigma_t^2\,\alpha_t(1-\bar{\alpha}_t)}\,\big\|\boldsymbol{\epsilon}_\theta(\sqrt{\bar{\alpha}_t}\,\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_t}\,\boldsymbol{\epsilon},\,t) - \boldsymbol{\epsilon}\big\|^2\right]. \tag{18}\]

Ho et al. observed that dropping the prefactor (i.e. using \(\lambda_t=1\)) gives the much simpler training objective

\[L_{\text{simple}}(\theta) = \mathbb{E}_{t\sim\mathcal{U}\{1,\dots,T\},\,\mathbf{x}_0,\boldsymbol{\epsilon}}\big[\|\boldsymbol{\epsilon} - \boldsymbol{\epsilon}_\theta(\mathbf{x}_t,t)\|^2\big], \qquad \mathbf{x}_t = \sqrt{\bar{\alpha}_t}\,\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_t}\,\boldsymbol{\epsilon}. \tag{19}\]

This is exactly \(\sigma\)-scaled denoising score matching (7) at noise scale \(\sigma_t^2=1-\bar{\alpha}_t\) if we identify \(\mathbf{s}_\theta(\mathbf{x}_t,t)=-\boldsymbol{\epsilon}_\theta(\mathbf{x}_t,t)/\sqrt{1-\bar{\alpha}_t}\).

3.5 Sampling (ancestral)

Given a trained \(\boldsymbol{\epsilon}_\theta\), sample \(\mathbf{x}_T\sim\mathcal{N}(\mathbf{0},\mathbf{I})\) and for \(t=T,\dots,1\) draw \(\mathbf{x}_{t-1}\sim p_\theta(\mathbf{x}_{t-1}\mid\mathbf{x}_t)\):

\[\mathbf{x}_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left(\mathbf{x}_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\,\boldsymbol{\epsilon}_\theta(\mathbf{x}_t,t)\right) + \sigma_t\,\mathbf{z}, \qquad \mathbf{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I}),\ \ \mathbf{z}\equiv\mathbf{0}\ \text{at}\ t=1. \tag{20}\]

4. DDIM: Non-Markovian Forwards and Deterministic Sampling

DDIM [5] starts from a key observation: the DDPM training loss (19) only depends on the marginals \(q(\mathbf{x}_t\mid\mathbf{x}_0)\), not on the joint Markov structure. So any forward process that shares those marginals admits the same trained model but may allow a different—possibly deterministic—reverse rule.

4.1 A family of non-Markovian forwards

For each schedule \(\{\sigma_t\}_{t=1}^{T}\ge0\), define the joint

\[q_\sigma(\mathbf{x}_{1:T}\mid\mathbf{x}_0) = q_\sigma(\mathbf{x}_T\mid\mathbf{x}_0)\prod_{t=2}^{T} q_\sigma(\mathbf{x}_{t-1}\mid\mathbf{x}_t,\mathbf{x}_0), \tag{21}\]

where the marginals \(q_\sigma(\mathbf{x}_t\mid\mathbf{x}_0)=\mathcal{N}(\sqrt{\bar{\alpha}_t}\,\mathbf{x}_0,\,(1-\bar{\alpha}_t)\mathbf{I})\) are forced to match DDPM's, and

\[q_\sigma(\mathbf{x}_{t-1}\mid\mathbf{x}_t,\mathbf{x}_0) = \mathcal{N}\left(\sqrt{\bar{\alpha}_{t-1}}\,\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_{t-1}-\sigma_t^2}\cdot\frac{\mathbf{x}_t-\sqrt{\bar{\alpha}_t}\,\mathbf{x}_0}{\sqrt{1-\bar{\alpha}_t}},\,\sigma_t^2\mathbf{I}\right). \tag{22}\]

The mean is the unique linear combination of \(\mathbf{x}_0\) and the "direction" \((\mathbf{x}_t-\sqrt{\bar{\alpha}_t}\,\mathbf{x}_0)/\sqrt{1-\bar{\alpha}_t}\) that matches the marginals, with covariance \(\sigma_t^2\).

Verification of marginals. Marginalize \(\mathbf{x}_t\) out of \(q_\sigma(\mathbf{x}_{t-1}\mid\mathbf{x}_t,\mathbf{x}_0)\) given \(q_\sigma(\mathbf{x}_t\mid\mathbf{x}_0)\). The conditional mean is linear in \(\mathbf{x}_t\) with coefficient \(c_t=\sqrt{(1-\bar{\alpha}_{t-1}-\sigma_t^2)/(1-\bar{\alpha}_t)}\). Hence

\[\mathbb{E}[\mathbf{x}_{t-1}\mid\mathbf{x}_0] = \sqrt{\bar{\alpha}_{t-1}}\,\mathbf{x}_0 - c_t\sqrt{\bar{\alpha}_t}\,\mathbf{x}_0 + c_t\,\mathbb{E}[\mathbf{x}_t\mid\mathbf{x}_0] = \sqrt{\bar{\alpha}_{t-1}}\,\mathbf{x}_0,\]

and \(\mathrm{Var}[\mathbf{x}_{t-1}\mid\mathbf{x}_0]=c_t^2(1-\bar{\alpha}_t)+\sigma_t^2=1-\bar{\alpha}_{t-1}\). \(\square\)

4.2 Recovering DDPM and the deterministic limit

Choose \(\sigma_t^2=\tilde{\beta}_t=\beta_t(1-\bar{\alpha}_{t-1})/(1-\bar{\alpha}_t)\). Then (22) algebraically equals (12)–(13); we recover DDPM.

Choose \(\sigma_t\equiv0\). The reverse step becomes deterministic:

\[\mathbf{x}_{t-1} = \sqrt{\bar{\alpha}_{t-1}}\,\hat{\mathbf{x}}_0(\mathbf{x}_t) + \sqrt{1-\bar{\alpha}_{t-1}}\cdot\frac{\mathbf{x}_t-\sqrt{\bar{\alpha}_t}\,\hat{\mathbf{x}}_0(\mathbf{x}_t)}{\sqrt{1-\bar{\alpha}_t}}, \tag{23}\]

where, since the trained network predicts the noise, \(\hat{\mathbf{x}}_0(\mathbf{x}_t)=\big(\mathbf{x}_t-\sqrt{1-\bar{\alpha}_t}\,\boldsymbol{\epsilon}_\theta(\mathbf{x}_t,t)\big)/\sqrt{\bar{\alpha}_t}\).

The neat form (23) is the DDIM update. Interpolating \(\sigma_t\) between \(0\) and \(\tilde{\beta}_t\) trades determinism (faster and higher quality at small step counts) for stochastic mixing.

4.3 DDIM as an ODE solver

Define \(\bar{\alpha}_t=1/(1+\lambda_t^2)\), let \(\bar{\mathbf{x}}_t=\mathbf{x}_t/\sqrt{\bar{\alpha}_t}\). Substituting into (23),

\[\bar{\mathbf{x}}_{t-1} - \bar{\mathbf{x}}_t = (\lambda_{t-1}-\lambda_t)\,\boldsymbol{\epsilon}_\theta(\mathbf{x}_t,t). \tag{24}\]

In the continuous limit this is the Euler step for

\[\frac{d\bar{\mathbf{x}}}{d\lambda} = \boldsymbol{\epsilon}_\theta\big(\sqrt{\bar{\alpha}(\lambda)}\,\bar{\mathbf{x}},\,t(\lambda)\big).\]

DDIM is therefore a first-order ODE integrator in the \(\lambda\)-parameterization of the probability-flow ODE we derive next.

5. The SDE / ODE Picture (Song et al., 2021)

5.1 Forward SDE

Consider the Itô SDE

\[d\mathbf{x} = f(\mathbf{x},t)\,dt + g(t)\,d\mathbf{w}, \qquad t\in[0,T], \tag{25}\]

with \(\mathbf{x}_0\sim p_{\text{data}}\). Three classical families:

• VE: \(f=0\), \(g(t)=\sqrt{d[\sigma^2(t)]/dt}\), so \(\mathbf{x}_t=\mathbf{x}_0+\sigma(t)\boldsymbol{\epsilon}\).
• VP: \(f=-\tfrac{1}{2}\beta(t)\mathbf{x}\), \(g(t)=\sqrt{\beta(t)}\).
• sub-VP: \(f=-\tfrac{1}{2}\beta(t)\mathbf{x}\), \(g(t)=\sqrt{\beta(t)\big(1-e^{-2\int_0^t\beta(s)\,ds}\big)}\).

DDPM as discretized VP. For VP, the linear OU SDE has solution

\[\mathbf{x}_t = e^{-\frac{1}{2}\int_0^t\beta(s)\,ds}\,\mathbf{x}_0 + \sqrt{1-e^{-\int_0^t\beta(s)\,ds}}\,\boldsymbol{\epsilon}.\]

Identify \(e^{-\int_0^t\beta}=\bar{\alpha}_t\) and you recover (9).

5.2 Anderson's reverse-time SDE

Theorem 5.1 (Anderson, 1982). The reverse-time process of (25), run from \(t=T\) back to \(t=0\), satisfies

\[d\mathbf{x} = \big[f(\mathbf{x},t) - g(t)^2\,\nabla_{\mathbf{x}}\log p_t(\mathbf{x})\big]\,dt + g(t)\,d\bar{\mathbf{w}}, \tag{26}\]

where \(dt<0\) and \(\bar{\mathbf{w}}\) is a reverse-time Wiener process.

Heuristic derivation via Fokker–Planck. The forward FP equation is

\[\partial_t p_t = -\nabla\cdot(f p_t) + \tfrac{1}{2}g^2\Delta p_t.\]

Use \(\Delta p_t=\nabla\cdot(p_t\nabla\log p_t)\):

\[\partial_t p_t = -\nabla\cdot(f p_t) + \tfrac{1}{2}g^2\nabla\cdot(p_t\nabla\log p_t).\]

We may split the diffusion term into a drift contribution plus a diffusion contribution in two ways. Writing

\[\tfrac{1}{2}g^2\Delta p_t = g^2\nabla\cdot(p_t\nabla\log p_t) - \tfrac{1}{2}g^2\Delta p_t,\]

giving

\[\partial_t p_t = -\nabla\cdot\big[(f - g^2\nabla\log p_t)\,p_t\big] - \tfrac{1}{2}g^2\Delta p_t.\]

Reversing time turns the negative diffusion term back into a positive one, leaving the modified drift \(f-g^2\nabla\log p_t\). A rigorous argument requires time-reversal of Itô processes; see Anderson (1982).

This is the central result of the score-SDE framework: denoising is solving the reverse-time SDE with the data score in place of the oracle.

5.3 The probability-flow ODE

Theorem 5.2 (Song et al., 2021). The deterministic ODE

\[\frac{d\mathbf{x}}{dt} = f(\mathbf{x},t) - \tfrac{1}{2}g(t)^2\,\nabla_{\mathbf{x}}\log p_t(\mathbf{x}) \tag{27}\]

has the same marginal \(p_t\) as the forward SDE (25).

Proof. Define \(\tilde{f}=f-\tfrac{1}{2}g^2\nabla\log p_t\) and apply the FP equation with diffusion \(\tilde{g}=0\):

\[\partial_t p_t = -\nabla\cdot(\tilde{f}p_t) = -\nabla\cdot(f p_t) + \tfrac{1}{2}g^2\nabla\cdot(p_t\nabla\log p_t) = -\nabla\cdot(f p_t) + \tfrac{1}{2}g^2\Delta p_t,\]

which is exactly the FP equation of (25). Same FP, same marginal. \(\square\)

Thus there is a deterministic flow whose pushforward of \(p_T\approx\mathcal{N}(\mathbf{0},\mathbf{I})\) equals \(p_0=p_{\text{data}}\). This ODE is the workhorse of fast samplers (DDIM, DPM-Solver, …) and of exact likelihood computation via the change-of-variables formula.

5.4 Training the score in the SDE setting

The SDE counterpart of (19) is the continuous denoising score-matching loss

\[\mathcal{L}(\theta) = \mathbb{E}_{t\sim\mathcal{U}[0,T]}\,\mathbb{E}_{\mathbf{x}_0}\,\mathbb{E}_{q_{t|0}(\mathbf{x}_t\mid\mathbf{x}_0)}\left[\lambda(t)\,\big\|\mathbf{s}_\theta(\mathbf{x}_t,t) - \nabla_{\mathbf{x}_t}\log q_{t|0}(\mathbf{x}_t\mid\mathbf{x}_0)\big\|^2\right]. \tag{28}\]

For Gaussian transitions \(q_{t|0}=\mathcal{N}(m_t\mathbf{x}_0,\,s_t^2\mathbf{I})\),

\[\nabla_{\mathbf{x}_t}\log q_{t|0}(\mathbf{x}_t\mid\mathbf{x}_0) = -\frac{\mathbf{x}_t-m_t\mathbf{x}_0}{s_t^2} = -\frac{\boldsymbol{\epsilon}}{s_t} \quad\text{when}\quad \mathbf{x}_t = m_t\mathbf{x}_0 + s_t\boldsymbol{\epsilon}.\]

With the choice \(\lambda(t)=s_t^2\) the loss matches the variational ELBO up to a constant (Kingma et al., 2021). Different \(\lambda\) give different but equally consistent estimators.

5.5 Putting it together

6. Flow Matching

Flow matching [9, 10] generalizes diffusion training. Instead of denoising at every noise scale, we directly learn a vector field whose flow transports a prior \(p_1\) (typically \(\mathcal{N}(\mathbf{0},\mathbf{I})\)) to the data \(p_0\).

6.1 Continuous normalizing flows in one slide

A time-dependent vector field \(u_t:\mathbb{R}^d\to\mathbb{R}^d\) defines a flow \(\phi_t:\mathbb{R}^d\to\mathbb{R}^d\) by

\[\frac{d}{dt}\phi_t(\mathbf{x}) = u_t(\phi_t(\mathbf{x})), \qquad \phi_0(\mathbf{x}) = \mathbf{x}. \tag{29}\]

The pushforward of \(p_0\) under \(\phi_t\) has density \(p_t\), which satisfies the continuity equation

\[\partial_t p_t + \nabla\cdot(p_t u_t) = 0. \tag{30}\]

6.2 The flow-matching objective

Suppose we are given a target probability path \(\{p_t\}_{t\in[0,1]}\) with \(p_0=p_{\text{data}}\) and \(p_1=\text{prior}\), and its generating vector field \(u_t\). Flow matching minimizes

\[\mathcal{L}_{\text{FM}}(\theta) = \mathbb{E}_{t\sim\mathcal{U}[0,1],\,\mathbf{x}\sim p_t}\big[\|v_\theta(\mathbf{x},t) - u_t(\mathbf{x})\|^2\big]. \tag{31}\]

Problem: \(u_t\) is unknown in closed form.

6.3 Conditional flow matching (the key trick)

Theorem 6.1 (Conditional flow matching, Lipman et al. 2023). Let \(p_t(\mathbf{x}\mid\mathbf{x}_1)\) be a family of conditional probability paths with conditional vector fields \(u_t(\mathbf{x}\mid\mathbf{x}_1)\) (each satisfying (30) marginally). Define the marginal path \(p_t(\mathbf{x})=\int p_t(\mathbf{x}\mid\mathbf{x}_1)p_{\text{data}}(\mathbf{x}_1)\,d\mathbf{x}_1\) and the marginal vector field

\[u_t(\mathbf{x}) = \mathbb{E}_{p(\mathbf{x}_1\mid\mathbf{x}_t=\mathbf{x})}\big[u_t(\mathbf{x}\mid\mathbf{x}_1)\big]. \tag{32}\]

Then the FM loss (31) equals, up to an additive constant in \(\theta\),

\[\mathcal{L}_{\text{CFM}}(\theta) = \mathbb{E}_{t,\,\mathbf{x}_1\sim p_{\text{data}},\,\mathbf{x}\sim p_t(\cdot\mid\mathbf{x}_1)}\big[\|v_\theta(\mathbf{x},t) - u_t(\mathbf{x}\mid\mathbf{x}_1)\|^2\big]. \tag{33}\]

Proof. Expand both squared losses; the \(\|v_\theta\|^2\) terms agree because the marginal density used in FM is the same as the joint marginal in CFM. The remaining cross terms agree:

\[\mathbb{E}_{p_t(\mathbf{x})}\big[v_\theta^\top u_t(\mathbf{x})\big] = \int p_t(\mathbf{x})\,v_\theta(\mathbf{x},t)^\top\mathbb{E}_{p(\mathbf{x}_1\mid\mathbf{x})}[u_t(\mathbf{x}\mid\mathbf{x}_1)]\,d\mathbf{x} = \iint p_t(\mathbf{x},\mathbf{x}_1)\,v_\theta(\mathbf{x},t)^\top u_t(\mathbf{x}\mid\mathbf{x}_1)\,d\mathbf{x}_1\,d\mathbf{x},\]

which is exactly the CFM cross term. \(\square\)

Why this matters. We never compute the (intractable) marginal \(u_t\) we train against; we train against the conditional \(u_t(\mathbf{x}\mid\mathbf{x}_1)\) whose closed form we get to design. The optimum is the same regression target \(u_t(\mathbf{x})\).

6.4 Gaussian probability paths

\[p_t(\mathbf{x}\mid\mathbf{x}_1) = \mathcal{N}\big(\mu_t(\mathbf{x}_1),\,\sigma_t(\mathbf{x}_1)^2\mathbf{I}\big).\]

Pick a conditional path linear in \(t\). Parameterize a sample as \(\mathbf{x}=\mu_t(\mathbf{x}_1)+\sigma_t(\mathbf{x}_1)\boldsymbol{\epsilon}\) with \(\boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\). The conditional vector field that generates this path is obtained by differentiating the sample path with \(\boldsymbol{\epsilon}\) fixed:

\[u_t(\mathbf{x}\mid\mathbf{x}_1) = \dot{\sigma}_t(\mathbf{x}_1)\,\frac{\mathbf{x}-\mu_t(\mathbf{x}_1)}{\sigma_t(\mathbf{x}_1)} + \dot{\mu}_t(\mathbf{x}_1). \tag{34}\]

Why (34) generates the path. With \(\mathbf{x}_t=\mu_t+\sigma_t\boldsymbol{\epsilon}\) (suppressing \(\mathbf{x}_1\)), \(\dot{\mathbf{x}}_t=\dot{\mu}_t+\dot{\sigma}_t\boldsymbol{\epsilon}=\dot{\mu}_t+\dot{\sigma}_t(\mathbf{x}_t-\mu_t)/\sigma_t\), which is exactly \(u_t(\mathbf{x}_t\mid\mathbf{x}_1)\). Hence the flow with this vector field follows the same sample trajectory; the marginal density along the trajectory is precisely \(p_t(\mathbf{x}\mid\mathbf{x}_1)\) by construction.

Diffusion as a special case. Choose \(\mu_t(\mathbf{x}_1)=\alpha_t\mathbf{x}_1\), \(\sigma_t(\mathbf{x}_1)=\sigma_t\) with \(\alpha_0=1\), \(\sigma_0=0\) and \(\alpha_1=0\), \(\sigma_1=1\). Then (34) matches the probability-flow ODE (27) of a VP-style diffusion (after a change of time).

7. Rectified Flow

7.1 Linear interpolant

Rectified flow [10] chooses the simplest possible conditional path: a straight line between independent samples \(\mathbf{x}_0\sim p_{\text{data}}\) and \(\mathbf{x}_1\sim p_1\) (the prior):

\[\mathbf{x}_t = (1-t)\mathbf{x}_0 + t\,\mathbf{x}_1, \qquad t\in[0,1]. \tag{35}\]

Its conditional vector field is constant in \(t\): \(u_t(\mathbf{x}_t\mid\mathbf{x}_0,\mathbf{x}_1)=\dot{\mathbf{x}}_t=\mathbf{x}_1-\mathbf{x}_0\). Plugging into (33) (now with both endpoints as conditioning):

\[\mathcal{L}_{\text{RF}}(\theta) = \mathbb{E}_{t\sim\mathcal{U}[0,1],\,(\mathbf{x}_0,\mathbf{x}_1)\sim p_{\text{data}}\otimes p_1}\big[\|v_\theta(\mathbf{x}_t,t) - (\mathbf{x}_1-\mathbf{x}_0)\|^2\big], \qquad \mathbf{x}_t = (1-t)\mathbf{x}_0 + t\mathbf{x}_1. \tag{36}\]

At convergence \(v_\theta(\mathbf{x},t)=\mathbb{E}[\mathbf{x}_1-\mathbf{x}_0\mid\mathbf{x}_t=\mathbf{x}]\) — the average velocity of the joint pairs passing through \(\mathbf{x}\) at time \(t\).

7.2 Why "rectified"? The reflow procedure

The marginal flow \(v_\theta\) generated by independent \((\mathbf{x}_0,\mathbf{x}_1)\) pairs is in general not straight — trajectories that intersect get averaged, producing curves. Construct a new coupling by solving the trained ODE backwards:

\[\mathbf{x}_1\sim p_1, \qquad \mathbf{x}_0 = \mathbf{x}_1 - \int_0^1 v_\theta(\mathbf{x}_t,t)\,dt. \tag{37}\]

Train another rectified-flow model on these matched pairs. Liu et al. prove that this reflow step strictly reduces the transport cost and that trajectories straighten: after a few iterations, the ODE can be integrated with only one or two Euler steps.

7.3 Relation to diffusion

Compare (36) to DDPM's (19). Diffusion uses \(\mathbf{x}_t=\alpha_t\mathbf{x}_0+\sigma_t\boldsymbol{\epsilon}\) and learns \(\boldsymbol{\epsilon}\); rectified flow uses \(\mathbf{x}_t=(1-t)\mathbf{x}_0+t\mathbf{x}_1\) and learns \(\mathbf{x}_1-\mathbf{x}_0\). Both are \(\ell_2\) regressions whose minimizer is a conditional expectation; both correspond to a probability-flow ODE. Rectified flow's straight-line interpolant is what makes few-step sampling so effective.

8. Guidance: Classifier and Classifier-Free

For conditional generation \(p(\mathbf{x}\mid y)\), we want the conditional score \(\nabla_{\mathbf{x}}\log p_t(\mathbf{x}\mid y)\).

8.1 Classifier guidance

Bayes' rule gives

\[\nabla_{\mathbf{x}}\log p_t(\mathbf{x}\mid y) = \nabla_{\mathbf{x}}\log p_t(\mathbf{x}) + \nabla_{\mathbf{x}}\log p_t(y\mid\mathbf{x}). \tag{38}\]

Train an unconditional diffusion (gives the first term) and a separate time-conditional classifier \(p_t(y\mid\mathbf{x})\), then sample with the modified score. To strengthen the guidance, scale:

\[\nabla_{\mathbf{x}}\log\tilde{p}_t^{(w)}(\mathbf{x}\mid y) = \nabla_{\mathbf{x}}\log p_t(\mathbf{x}) + w\,\nabla_{\mathbf{x}}\log p_t(y\mid\mathbf{x}). \tag{39}\]

The implied distribution is \(\tilde{p}\propto p_t(\mathbf{x})\,p_t(y\mid\mathbf{x})^w\).

8.2 Classifier-free guidance (CFG)

A trained conditional score is \(\nabla\log p_t(\mathbf{x}\mid y)\). Use (38) to express the classifier gradient implicitly: \(\nabla\log p_t(y\mid\mathbf{x})=\nabla\log p_t(\mathbf{x}\mid y)-\nabla\log p_t(\mathbf{x})\). Substitute into (39):

\[\nabla\log\tilde{p}_t^{(w)}(\mathbf{x}\mid y) = (1-w)\,\nabla\log p_t(\mathbf{x}) + w\,\nabla\log p_t(\mathbf{x}\mid y). \tag{40}\]

Equivalently, in noise-prediction form, \(\tilde{\boldsymbol{\epsilon}}_\theta(\mathbf{x},t,y)=(1-w)\,\boldsymbol{\epsilon}_\theta(\mathbf{x},t,\varnothing)+w\,\boldsymbol{\epsilon}_\theta(\mathbf{x},t,y)\). Training: drop the conditioning \(y\to\varnothing\) with some probability \(p_{\text{drop}}\) during DDPM training, so one network learns both the conditional and unconditional scores. At sampling time, mix them as above with \(w>1\) (typical: 1.5–8).

9. ELBO ↔︎ Score-Matching Equivalence

Kingma et al. (2021) made the bridge from VAE-style ELBO to denoising score matching explicit.

In the SDE picture,

\[\mathrm{KL}\big(p_{\text{data}}\,\big\|\,p_\theta^{(0)}\big) \le \frac{1}{2}\int_0^T \mathbb{E}_{p_t(\mathbf{x})}\Big[g(t)^2\,\big\|\mathbf{s}_\theta(\mathbf{x},t) - \nabla\log p_t(\mathbf{x})\big\|^2\Big]\,dt + C, \tag{41}\]

which is exactly the continuous DSM loss (28) with weighting \(\lambda(t)=g(t)^2\). Different weightings give different variational bounds—all yield consistent estimators of the score, but \(\lambda=g^2\) corresponds to maximum-likelihood training.

Sketch. Differentiate \(\mathrm{KL}(p_t\,\|\,p_\theta^{(t)})\) and use both FP equations. The derivative reduces to a Fisher divergence; integrating from \(t=0\) to \(t=T\) and bounding the boundary term gives (41).

10. Consistency Models

Consistency models [13] aim for one-step sampling along the probability-flow ODE.

10.1 Definition

Given the PF-ODE (27) with solution trajectory \(\{\mathbf{x}_t\}_{t\in[\epsilon,T]}\), define the consistency function

\[f^\star(\mathbf{x}_t,t) := \mathbf{x}_\epsilon \quad\text{(the trajectory endpoint at } t=\epsilon).\]

Two defining properties:

1. Boundary: \(f^\star(\mathbf{x}_\epsilon,\epsilon)=\mathbf{x}_\epsilon\).
2. Self-consistency: \(f^\star(\mathbf{x}_t,t)=f^\star(\mathbf{x}_{t'},t')\) for any pair \((t,t')\) on the same trajectory.

The consistency model \(f_\theta\) is parameterized to satisfy the boundary condition by construction, e.g.

\[f_\theta(\mathbf{x},t) = c_{\text{skip}}(t)\,\mathbf{x} + c_{\text{out}}(t)\,F_\theta(\mathbf{x},t) \quad\text{with}\quad c_{\text{skip}}(\epsilon)=1,\ c_{\text{out}}(\epsilon)=0. \tag{42}\]

10.2 Consistency-distillation training

Discretize times \(\epsilon=t_1<\cdots<t_N=T\). Given a pretrained score \(\mathbf{s}_\phi\) (the teacher), use one solver step \(\Phi\) of the PF-ODE to go from \(t_{n+1}\) to \(t_n\): \(\hat{\mathbf{x}}_{t_n}=\Phi(\mathbf{x}_{t_{n+1}},\,t_{n+1}\to t_n;\,\mathbf{s}_\phi)\). Enforce self-consistency by minimizing

\[\mathcal{L}_{\text{CD}}(\theta) = \mathbb{E}_{n,\mathbf{x},\boldsymbol{\epsilon}}\big[d\big(f_\theta(\mathbf{x}_{t_{n+1}},t_{n+1}),\,f_{\theta^-}(\hat{\mathbf{x}}_{t_n},t_n)\big)\big], \tag{43}\]

where \(\theta^-\) is an EMA of \(\theta\) (the "target network") and \(d\) is any metric (e.g. \(\ell_2\) or LPIPS). Sampling with a trained \(f_\theta\) takes one evaluation: \(\mathbf{x}_0=f_\theta(\mathbf{x}_T,T)\) with \(\mathbf{x}_T\sim\mathcal{N}(\mathbf{0},\mathbf{I})\).

10.3 Consistency training (no teacher)

Replace \(\hat{\mathbf{x}}_{t_n}\) in (43) by the closed-form noised data \(\mathbf{x}_0+t_n\boldsymbol{\epsilon}\) (for VE) and use the same loss. The bound \(\mathcal{L}_{\text{CT}}\ge\mathcal{L}_{\text{CD}}\) holds when the noise scales are chosen carefully (the original paper shows consistency between the two estimators in the continuous limit).

11. One-Page Cheat Sheet

The Rosetta Stone. With \(\mathbf{x}_t=m_t\mathbf{x}_0+s_t\boldsymbol{\epsilon}\) and \(\boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\):

\[\underbrace{\nabla_{\mathbf{x}_t}\log q_t(\mathbf{x}_t)}_{\text{score}} \approx -\frac{\boldsymbol{\epsilon}_\theta(\mathbf{x}_t,t)}{s_t} \iff \hat{\mathbf{x}}_0 = \frac{\mathbf{x}_t - s_t\boldsymbol{\epsilon}_\theta}{m_t} \iff v_\theta(\mathbf{x}_t,t) = \dot{m}_t\hat{\mathbf{x}}_0 + \dot{s}_t\boldsymbol{\epsilon}_\theta.\]

Loss families (all are \(\ell_2\) regressions).

Samplers.

Equivalences worth remembering.

• DDPM training = denoising score matching at noise scale \(s_t=\sqrt{1-\bar{\alpha}_t}\).
• DDPM forward chain = Euler–Maruyama discretization of the VP-SDE.
• DDIM (\(\sigma=0\)) = first-order solver of the probability-flow ODE.
• Flow matching with \(\mu_t=\alpha_t\mathbf{x}_1\), \(\sigma_t\) linear = training a diffusion model up to weighting.
• Rectified flow = flow matching with the linear interpolant \(\mathbf{x}_t=(1-t)\mathbf{x}_0+t\mathbf{x}_1\).
• Classifier-free guidance = tempering \(\tilde{p}\propto p(\mathbf{x})\,p(y\mid\mathbf{x})^w\) implemented by mixing conditional/unconditional scores.

References

[1] P. Vincent. A Connection Between Score Matching and Denoising Autoencoders. Neural Computation, 2011.

[2] A. Hyvärinen. Estimation of Non-Normalized Statistical Models by Score Matching. JMLR, 2005.

[3] J. Sohl-Dickstein et al. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. ICML, 2015.

[4] J. Ho, A. Jain, P. Abbeel. Denoising Diffusion Probabilistic Models. NeurIPS, 2020.

[5] J. Song, C. Meng, S. Ermon. Denoising Diffusion Implicit Models. ICLR, 2021.

[6] Y. Song et al. Score-Based Generative Modeling through Stochastic Differential Equations. ICLR, 2021.

[7] B.D.O. Anderson. Reverse-Time Diffusion Equation Models. Stochastic Processes and Applications, 1982.

[8] D. Kingma et al. Variational Diffusion Models. NeurIPS, 2021.

[9] Y. Lipman et al. Flow Matching for Generative Modeling. ICLR, 2023.

[10] X. Liu, C. Gong, Q. Liu. Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. ICLR, 2023.

[11] P. Dhariwal, A. Nichol. Diffusion Models Beat GANs on Image Synthesis. NeurIPS, 2021.

[12] J. Ho, T. Salimans. Classifier-Free Diffusion Guidance. NeurIPS Wkshp 2021 / 2022.

[13] Y. Song et al. Consistency Models. ICML, 2023.

[14] T. Karras et al. Elucidating the Design Space of Diffusion-Based Generative Models. NeurIPS, 2022.

★ 2026 SOTA update — ICLR 2024

• Consistency Trajectory Models (CTM): Generalizes consistency and score models: a single network learns both the score and unrestricted 'long jumps' along the probability-flow ODE trajectory, unifying the ODE-solver and one-step views in the derivation's SDE/ODE section. Introduces the flow-map perspective (any-time-to-any-time).

★ 2026 SOTA update — arXiv (OpenAI)

• Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models (sCM): Continuous-time reformulation of consistency training that removes discretization error and EMA teacher, with a unified parameterization of diffusion and consistency models. Directly extends the document's Consistency-Models section; reaches near-diffusion FID in 2 steps at 1.5B scale.

★ 2026 SOTA update — arXiv / ICLR 2025

• One Step Diffusion via Shortcut Models: Conditions the velocity network on step size as well as noise level, so a single model + single training phase produces samples in one or many ODE steps. A clean self-consistency extension of the flow-matching / rectified-flow ODE picture without reflow or a teacher.

★ 2026 SOTA update — arXiv

• Inductive Moment Matching (IMM): Single-stage few-step generator trained by moment matching rather than distillation; guarantees distribution-level convergence and is more stable than consistency training. Offers an alternative theoretical route to the few-step ODE target studied in the derivations.

★ 2026 SOTA update — NeurIPS 2025 (Oral)

• Mean Flows for One-step Generative Modeling (MeanFlow): Introduces the average-velocity field and a MeanFlow identity relating average to instantaneous velocity, giving a from-scratch one-step model (1-NFE FID 3.43 on ImageNet 256). A direct theoretical generalization of the instantaneous-velocity flow-matching objective in Sections 6-7.

★ 2026 SOTA update — arXiv (NVIDIA)

• Align Your Flow: Scaling Continuous-Time Flow Map Distillation: Defines flow maps that connect any two noise levels in one step and stay effective across all step counts, with two new continuous-time objectives generalizing both flow-matching and consistency losses. Consolidates the ODE-solver / consistency-model spectrum from the cheat sheet into one framework.