Video Generation — Models, Technologies & Tricks

Updated July 2026 with 2025–2026 SOTA additions — new entries marked ★. Algorithm names link to their papers (arXiv / project page).

April 2026 · Version 1.0

Contents

1. Foundations
2. The Architecture Pattern
3. Tokenizers for Video
4. Open Frontier Models (Lineup)
5. Sora and Sora 2
6. Veo and Google's Stack
7. Hunyuan Video and Wan
8. Image-to-Video and Conditioning
9. Image-Driven Animation
10. Audio-Joint Generation
11. Long-Video Generation
12. Camera Control and Motion Conditioning
13. World-Model Framing
14. Editing
15. Distillation for Real-Time / Low-Latency
16. Conditioning for Control
17. Training Data and Compute
18. Evaluation
19. Failure Modes and Common Issues
20. Production Deployment
21. Frontier 2025–2026
22. Production Stack 2026

• Appendix A: Twenty-Five Things to Know
• Appendix B: Decision Tree — "Which Video Gen?"
• Appendix C: Year-by-Year Video Generation Milestones

1. Foundations

1.1 The video generation problem

Generate a video clip \(V \in \mathbb{R}^{T \times H \times W \times 3}\) conditioned on:

• Text prompt (T2V).
• Reference image (I2V).
• Reference video + edit (V2V).
• Action sequence (world model).
• Audio + identity (talking head).

1.2 Three eras

1. 2022–23: per-frame stitching via image diffusion + temporal post-hoc (AnimateDiff, ModelScope).
2. 2023–24: native video diffusion via 3D U-Net or DiT; short clips (SVD, CogVideoX).
3. 2024–26: foundation video models with spatiotemporal MM-DiT + causal 3D VAE; long, high-res (Sora 2, Veo 3, Hunyuan, Wan).

1.3 Why video gen is hard

• Token count: 1080p 5s clip = millions of latent tokens.
• Temporal consistency: identity, geometry, lighting across frames.
• Physical commonsense: gravity, collisions, object permanence.
• Long-horizon: multi-minute coherence remains open.
• Compute: training Sora-class costs \(\sim\) tens of millions of dollars.

1.4 Why video gen exploded in 2024–26

• Diffusion + DiT proved scalable for image generation; extended cleanly to video.
• Causal 3D VAEs unlocked tractable training token counts.
• Massive web video corpora became available (filtered, captioned).
• H100/B200 hardware made the compute feasible.
• Strategic interest from OpenAI / Google / Meta drove racing.

1.5 The 2026 production stack at a glance

Key

Causal 3D VAE encodes video to spatiotemporal latents. Spatiotemporal MM-DiT (multi-modal Diffusion Transformer) generates latents conditioned on text + image. Often flow matching loss instead of DDPM. Joint image+video training. Distilled to 4–8 step inference for production.

2. The Architecture Pattern

2.1 Causal 3D VAE

Encoder: 3D convolutions with causal temporal padding (current frame depends only on past). Compresses video \(V \in \mathbb{R}^{T \times H \times W \times 3}\) to latent \(Z \in \mathbb{R}^{T' \times H' \times W' \times C}\) with typical:

\[T' = T/4, \quad H' = H/8, \quad W' = W/8, \quad C \in \{4, 16\}.\]

Decoder mirrors encoder; output back to RGB.

2.2 Spatiotemporal MM-DiT (Sora, SD3, FLUX video)

Two parallel streams (text, video) with separate \(W_Q, W_K, W_V, W_O\) and MLP, interacting through one joint self-attention over concatenated tokens:

\[[Q_\text{txt}; Q_\text{vid}], \quad [K_\text{txt}; K_\text{vid}], \quad [V_\text{txt}; V_\text{vid}].\]

3D-RoPE on video tokens (time + height + width).

2.3 Diffusion vs autoregressive

Diffusion (Sora, Veo, Hunyuan, Wan, Cosmos Predict):

• Parallel generation across all frames per denoising step.
• High visual quality.
• Hard to extend indefinitely; chunked AR for long clips.

Autoregressive (Cosmos AR, Emu3 video, GameNGen):

• One frame / chunk at a time.
• Naturally indefinite-horizon.
• Compatible with discrete-token tokenizers (LFQ, FSQ).
• Slightly lower quality at fixed compute.

2.4 Conditioning mechanisms

• Cross-attention (legacy): image tokens attend to text tokens (SD video).
• Joint MM-DiT (modern): text + video in one self-attn (SD3, FLUX, Sora).
• adaLN-Zero: scale + shift LayerNorm with timestep + condition.
• Image conditioning: encode reference image, prepend to sequence.
• Camera conditioning: Plücker coordinates per pixel; or text-encoded motion.

2.5 Joint image+video training

A still image is the \(T = 1\) case. Same model trains on both with packing. Often weighted:

\[\mathcal{L} = \lambda_I \, \mathcal{L}_\text{img} + \lambda_V \, \mathcal{L}_\text{video}.\]

2.6 Variable resolution / aspect / fps

Modern stacks support variable \((T, H, W, \text{fps})\) via:

• Native 3D-RoPE on patch positions.
• Token packing.
• Padding masks at attention.

Sora explicitly trained on diverse resolutions / aspect ratios.

Key

★ 2026 SOTA update — Autoregressive video backbones

• MAGI-1: chunk-wise autoregressive DiT denoising per-chunk monotonically-increasing noise; causal, streaming, constant peak memory regardless of length (Sand AI, 24B).
• Self-Forcing: closes AR train-test gap by rolling out with KV-cache during training + holistic distribution matching; real-time streaming 480p ~16fps on one H100.

3. Tokenizers for Video

3.1 Why the tokenizer matters

The tokenizer determines compression rate (training cost), reconstruction quality (output ceiling), and compatibility with downstream model. Bad tokenizer dooms the whole pipeline.

3.2 Causal 3D VAE (continuous)

Standard \(4 \times 8 \times 8\) compression \((T, H, W)\). Continuous latents (typically 16 channels). Used in Sora, Open-Sora, CogVideoX, Mochi, Wan, Cosmos Predict (Diffusion).

3.3 MAGVIT-v2 (LFQ)

Lookup-Free Quantization: project latent to dim \(L\), sign-quantize \(\to\) vocab \(2^L\). No codebook collapse. Joint image + video. Used in Genie, Emu3, Cosmos Predict (AR).

3.4 FSQ (Finite Scalar Quantization)

Per-dim rounding to a small set; vocab \(\prod_i (2K_i + 1)\). Simpler than LFQ; competitive.

3.5 Cosmos Tokenizer

NVIDIA's joint image + video tokenizer; continuous (CV) and discrete (DV) variants; up to 8K resolution.

3.6 First-frame asymmetry

Causal VAEs handle the first frame asymmetrically (no past to depend on). Common tricks:

• Pad first frame as its own past.
• Use a separate first-frame encoder.
• Treat first frame as an image (\(T=1\)) under joint im/video training.

3.7 Token-budget math

\[\text{At } 24\text{ fps},\ H = W = 512,\ 4 \times 8 \times 8 \text{ compression:}\]

\[\text{tokens/sec} = 24 \cdot 512^2 / (4 \cdot 8 \cdot 8) = 24576.\]

A 5-second clip \(\sim\) 120k tokens (manageable). A 1-minute clip \(\sim\) 1.5M (heavy). Drives need for ring attention / sequence parallelism.

3.8 Higher compression efforts

• TiTok-style 1D tokenization for ultra-compact.
• Larger temporal compression (\(T/8\), \(T/16\)) at quality cost.
• Hybrid: dense in keyframes, sparse in between.

Key

★ 2026 SOTA update — Ultra-compressed video VAE

• LTX-Video: holistic VAE+transformer with 1:192 (\(32 \times 32 \times 8\)) compression, patchify moved into the VAE; enables full spatiotemporal attention and real-time 720p generation.

4. Open Frontier Models (Lineup)

4.1 Open T2V model lineup (2026)

4.2 Closed frontier (April 2026)

4.3 Comparison matrix (key axes)

• Resolution: most produce 720p–1080p; few at 4K.
• Length: open \(\sim\) 5s; closed Sora 2 / Veo 3 push 60s+.
• Audio: only Veo 3, MovieGen produce native audio.
• Camera control: Luma, Sora, Veo support explicit camera paths.
• Identity preservation: Kling 2 strong; varies elsewhere.
• Motion realism: Sora 2, Kling 2 lead; some open close.
• Distillation / few-step: Hyper-FLUX-Video, LTX-Video few-step.

Key

★ 2026 SOTA update — 2025 open-frontier model papers

• Wan: Alibaba's open 1.3B/14B DiT suite (the 2.1 technical report) with new VAE, scaling-law study; basis for Wan 2.2 MoE (high-/low-noise experts).
• Seaweed-7B: 7B trained in just 665k H100-hrs yet rivals larger models; strong I2V, human, and audio-video downstream (ByteDance).
• Open-Sora 2.0: commercial-level 11B trained for only $200k via Video DC-AE deep-compression autoencoder + staged training.

5. Sora and Sora 2

5.1 Sora (Feb 2024)

• Spacetime patches: video tokenized as 3D patches.
• DiT backbone (likely MM-DiT variant; details closed).
• Trained on diverse video data with re-captioning.
• Up to 1-minute clips at 1080p.
• Demonstrated emergent world-modeling (some physics, motion blur, even some object permanence).
• Closed; no public API initially.

5.2 Sora 2 (Sep 2025)

• Significantly improved physical consistency.
• Audio generation joint with video.
• Longer clips, better identity preservation.
• Available via Sora app and API.
• OpenAI explicitly marketed as "world simulator."

5.3 Spacetime patches

Each patch covers \((t_p, h_p, w_p)\) region of video latent. Standard: \(1 \times 2 \times 2\) patches over latent (already \(4 \times 8 \times 8\) compressed by VAE). Total token count: \(T/4 \times H/16 \times W/16\).

5.4 Re-captioning

Sora's training corpus had original captions re-generated by a strong VLM for higher-quality, more detailed text descriptions. Critical for prompt-following.

5.5 Sora's world model claims

OpenAI's positioning: Sora as a foundation model for understanding physical world. Caveats: still many physics failures (objects vanish, collide impossibly). Not yet a true simulator; a generative video model with emergent physical priors.

6. Veo and Google's Stack

6.1 Veo 1 / 2 / 3 (Google DeepMind)

• Veo 1 (May 2024): 1080p, up to 1 minute, camera control.
• Veo 2 (Dec 2024): improved motion, physics.
• Veo 3 (May 2025): native audio generation; further fidelity gains.
• Imagen-derived backbone; integrated with Gemini.

6.2 Camera control conditioning

Veo accepts explicit camera trajectories. Implemented via Plücker coordinates per pixel as conditioning input to the diffusion transformer.

6.3 Native audio (Veo 3)

Joint diffusion of video + audio latents. Single model produces synchronized output. Shared architecture; crossmodal attention between video and audio token streams.

6.4 Imagen Video lineage

Veo derives from Imagen Video research. Cascaded super-resolution; T5 text encoder; spatial + temporal U-Net (V1). Migrated to DiT in V2/V3.

7. Hunyuan Video and Wan

7.1 Hunyuan Video (Tencent, Dec 2024)

• 13B parameters; open weights.
• DiT backbone with double-stream + single-stream blocks.
• Causal 3D VAE.
• Trained on \(\sim\) 10M curated videos.
• Strong on photoreal motion, character consistency.
• I2V and T2V variants.
• Frontier-open quality, comparable to closed leaders for many tasks.

7.2 Wan 2.1 / 2.2 (Alibaba, 2025)

• 14B (Wan 2.1) / multiple sizes.
• Open weights with permissive license.
• Strong on Chinese-specific cultural content + general.
• Extensive ControlNet variants.
• Wan 2.2: improved motion + audio support.

7.3 Mochi 1 (Genmo, Oct 2024)

• 10B AsymmDiT (asymmetric: most params in visual stream).
• Open Apache 2.0.
• 5.4s / 480p natively; upscaling to 1080p.
• Strong motion quality.

7.4 LTX-Video (Lightricks, Dec 2024)

• 2B params.
• Real-time generation on A100.
• Open; aimed at interactive use.

7.5 Step-Video (StepFun, 2025)

• 30B params; largest open T2V at release.
• Open weights.
• Strong on instruction following.

7.6 Allegro (RhymeAI, 2024)

2.8B; 6s / 720p. Open.

8. Image-to-Video and Conditioning

8.1 I2V (image-to-video) basics

Given a reference image, generate video starting from / containing it.

• Encode reference image to latent.
• Either: condition first-frame latent and denoise rest (first-frame I2V).
• Or: condition globally via cross-attention / concatenation.

8.2 First-frame I2V

Most common pattern. Provide image as the literal first frame; diffusion fills in temporal continuation.

• SVD, AnimateDiff, CogVideoX, Hunyuan I2V.
• User has tight control over starting state.

8.3 Last-frame and key-frame I2V

• Last-frame: end at this image (e.g., transition).
• Multi-keyframe: interpolate between specified key frames.
• Used in Pika, Kling, Runway for storyboard-style control.

8.4 Identity-preserving I2V

• Reference image describes character; generate them moving.
• Identity drift is a major artifact in long clips.
• ControlNet-style or IP-Adapter-style conditioning helps.

8.5 V2V (video-to-video)

• Style transfer: apply new style to existing video.
• Re-animation: re-pose existing character.
• Temporal stylization (e.g., Runway Gen-3 video-to-video).

8.6 Sketch-to-video, depth-to-video

Condition on per-frame sketch or depth video. Useful for storyboard \(\to\) video and animation pipelines.

9. Image-Driven Animation

9.1 The category

A static reference image driven by motion (audio, pose video, driving video) to produce animation. Distinct from full T2V generation.

9.2 Pose-driven character animation

• AnimateAnyone (Alibaba 2024): pose video drives static character.
• MimicMotion (Tencent): pose-controlled with motion transfer.
• Champ: parametric pose conditioning.
• MagicAnimate: dense-pose conditioning.
• Standard for short-form content creation, e-commerce.

9.3 Audio-driven talking head

• EMO (Alibaba 2024): audio \(\to\) photoreal portrait.
• Audio2Photoreal (Meta): audio \(\to\) full body.
• V-Express, Hallo, AniPortrait, HeyGen, D-ID.
• Live Portrait: image + driving video \(\to\) animated portrait.

9.4 Reference-only conditioning (ReferenceNet)

A trainable copy of the U-Net runs on the reference image; KVs of main U-Net's self-attention concatenated with reference KVs:

\[K = \text{Concat}(K_\text{self}, K_\text{ref}), \quad V = \text{Concat}(V_\text{self}, V_\text{ref}).\]

Standard mechanism for identity preservation in animation pipelines.

9.5 Hybrid VLM-conditioned animation

Recent: use VLM to interpret long instructions, decompose into motion sequences, drive animation. Bridges T2V and I2V.

10. Audio-Joint Generation

10.1 Why joint audio?

Native audio is critical for usability:

• No need for separate audio model + sync.
• Sound matches visuals (footsteps, splashes, dialogue lip-sync).
• Strongly demanded by film / advertising.

10.2 Veo 3 audio

Joint diffusion of video + audio latents. Single model; shared transformer backbone with cross-modal attention. Synchronized output, lip sync handled.

10.3 MovieGen Audio (Meta)

Separate audio model trained jointly with video. Music + foley + dialogue. Combined with MovieGen Video.

10.4 V2A models

• MMAudio: video-to-audio dub.
• ReCorD: reasoning-conditioned audio.
• V2A-Mapper: video conditioning \(\to\) audio LDM.
• Used as add-on to silent video gen.

10.5 Sora 2 audio

Sora 2 (Sep 2025) introduced native audio. Architecture details closed; competitive with Veo 3.

10.6 Wan 2.2 audio

Wan 2.2 added audio support. Open-frontier audio + video.

10.7 Lip-sync challenges

Highly noticeable failure mode. Solutions:

• Joint generation (Veo 3, Sora 2 default).
• Per-frame lip alignment networks (e.g., Wav2Lip post-process).
• Phoneme-aware conditioning.

Key

★ 2026 SOTA update — Native joint audio-video

• Ovi: twin matched DiT towers (video + from-scratch audio) fused by blockwise bidirectional cross-attention with scaled-RoPE; one-pass synchronized speech, SFX and video.

11. Long-Video Generation

11.1 The horizon problem

General video models cap at \(\sim\) 1–2 minutes coherent. Beyond, identity / scene drift.

11.2 Strategies

• Chunked autoregression: generate 5–10s chunks; condition next on last frames.
• Anchor-frame conditioning: provide keyframes throughout.
• Long-context training: train on long clips with ring attention.
• Hierarchical: high-level plan \(\to\) per-scene gen \(\to\) stitch.
• World-model framing: action-conditioned indefinite-horizon.

11.3 Sora 2 long clips

Reportedly 60s+ via chunked attention with carefully managed memory. Details closed.

11.4 Genie 2 (DeepMind)

Up to 1 minute interactive 3D worlds from a single image. Maintains scene persistence via recurrent state.

11.5 Streaming / live generation

• Lower-latency models for live use.
• One-frame-at-a-time AR.
• Caching of intermediate features.

11.6 Practical quality vs length

Open trade-off. Most applications today use 5–15 s clips chained into longer narratives via cuts (handled in editor, not single-pass gen).

Key

★ 2026 SOTA update — Long / consistent video 2025

• SkyReels-V2: infinite-length film model via Diffusion Forcing (non-decreasing per-frame noise) + MLLM captioning + motion RL.
• FramePack: next-frame-prediction with importance-weighted context packing (fixed context for thousands of frames) plus anti-drift sampling.
• LongLive: frame-level AR with KV-recache on prompt switch, streaming long-tuning and attention sink; interactive minute-scale, 20.7fps on one H100.
• Self-Forcing++: extends distilled AR students to minute-scale without long-video teachers, curbing latent-space error compounding.

12. Camera Control and Motion Conditioning

12.1 Camera-control signals

• Plain text ("zoom in slowly").
• 6-DoF camera trajectory per frame.
• Plücker coordinates per pixel as conditioning map.
• Pre-rendered 3D scene + camera path.

12.2 CameraCtrl, MotionCtrl, ViewCrafter

Open methods for camera-controlled generation. Train on data with known camera trajectories; condition on Plücker maps.

12.3 Veo / Sora / Luma camera control

Frontier closed models accept camera prompts (rotate, dolly, pan) via text or explicit trajectory.

12.4 3D-aware conditioning

• Use 3DGS scene + render camera path; feed renders as conditioning.
• Hybrid: high-quality 3D representation drives view; diffusion adds detail.

12.5 Trajectory-following

Drag-to-direct interfaces (Luma's keyframes, Pika 2's gestures). Translates user gestures into camera + motion conditioning.

13. World-Model Framing

13.1 When video gen becomes a world model

• Conditioned on past frames + actions.
• Indefinite horizon.
• Used for downstream control (RL training, AV closed-loop, robotics imagination).

13.2 GAIA-1 / GAIA-2 (Wayve)

Driving-specific. Diffusion conditioned on past frames + steering / throttle. Closed-loop AV simulation.

13.3 Cosmos World Foundation Models (NVIDIA)

• Cosmos Predict (Diffusion + Autoregressive).
• Action conditioning.
• Open weights (some); commercial-grade.
• Used for robotics + AV pretraining + sim.

13.4 Genie 2 (DeepMind)

Latent action discovery + 3D playable from single image.

13.5 Sora as world model

OpenAI's framing; emergent physical priors. Not a true simulator yet; trends toward becoming one.

13.6 Boundary blur with video gen

• Modern T2V models double as world models with action conditioning fine-tune.
• Cosmos is video gen + world model platform.
• Wan 2.2 has world-model variants.

Key

★ 2026 SOTA update — Real-time interactive world models

• Matrix-Game 2.0: open real-time streaming interactive world model; few-step causal AR diffusion, frame-level mouse/keyboard action injection, minute-level 25fps.
• Hunyuan-GameCraft: interactive game video unifying keyboard/mouse into a shared camera space, hybrid history-conditioned AR extension + distillation for real-time play.

14. Editing

14.1 Categories

• Inpainting: edit a region across frames.
• Outpainting: expand canvas spatially / temporally.
• Style transfer: re-style entire video.
• Object replacement: swap object identities.
• Re-timing: speed up / slow down.
• Re-lighting: change lighting (covered in Relighting cheat sheet).

14.2 Open and closed editing tools

• Runway Gen-3 video-to-video: style transfer, motion edits.
• Pika Modify Region: inpainting.
• Sora Storyboard: long-form composition.
• Hunyuan Video Edit: open object editing.
• Wan Edit: open variants.

14.3 Temporal consistency challenge

Per-frame edits flicker without temporal regularization. Solutions:

• Edit at latent level (not per-frame).
• Optical-flow-based constraints.
• ControlNet-style conditioning.
• Diffusion-DPO with temporal preference data.

14.4 Reference-driven editing

Provide reference image of desired edit; diffusion conditions on it. Flux Kontext-Video pattern; emerging.

15. Distillation for Real-Time / Low-Latency

15.1 Why distill video?

Video gen is the most compute-heavy generative task. Distillation makes it deployable.

15.2 Approaches

• Latent Consistency Models (LCM) for video.
• Hyper-Video / Hyper-FLUX-Video: blended adversarial + consistency.
• Phased Consistency Models (PCM) for video.
• DMD2-Video: distribution matching for one-step video.
• LTX-Video: native fast model, real-time on A100.

15.3 Step-distillation results

Open frontier video models (Hunyuan, Wan) typically take 20–50 NFEs per clip. Distilled variants 4–8 NFEs at modest quality loss.

15.4 Block-cache (FLUX-style applied to video)

Cache attention outputs across denoising steps; recompute every \(k\) steps. Accelerates without retraining.

15.5 TGATE for video

Cross-attention freezing after early steps; speeds up text-conditioned video gen.

15.6 PAB (Pyramid Attention Broadcast)

Cache + broadcast attention spatially + temporally. Video-specific 2× speedup.

15.7 Real-time video gen frontier

LTX-Video, GameNGen, Oasis demonstrate 10–30 fps real-time generation on a single GPU. Trade-off: lower quality than Sora-class.

Key

★ 2026 SOTA update — Few-step & real-time distillation

• CausVid: converts slow bidirectional teacher into a fast causal student via asymmetric DMD; KV-cached streaming ~9.4fps, 1.3s latency (CVPR 2025).
• Seaweed-APT: adversarial post-training against real data for one-step generation; real-time 2s 720p24 video (and 1024px images) in a single forward step.

16. Conditioning for Control

16.1 ControlNet for video

Apply ControlNet-style branch to video DiT with control video (depth maps over time, pose sequences).

• CogVideoX-Control.
• Wan ControlNet variants.
• Hunyuan ControlNet variants.

16.2 IP-Adapter for video

Image conditioning via parallel cross-attn on reference image; for identity preservation across video.

16.3 Diffusion-DPO for video

Lift Diffusion-DPO to video: preference pairs over clip pairs. Used to align Hunyuan / Wan to aesthetic / motion preferences.

16.4 LoRA for video

Per-style / per-character LoRAs trained on small datasets. Standard for production customization. Common formats released for open T2V.

16.5 Reference video conditioning

Provide a reference video for style + motion; generate new content. "Like this clip but X."

17. Training Data and Compute

17.1 Data pipeline

1. Source: YouTube, Vimeo, licensed content, in-house.
2. Filter: aesthetic, motion magnitude, OCR for text.
3. Dedupe: SimHash / CLIP embedding NN.
4. Re-caption: VLM (GPT-4V, Qwen-VL).
5. NSFW / safety filtering.
6. Format conversion: standardize fps, aspect.

17.2 Captioning

Original web captions are usually poor. Modern pipelines:

• Re-caption with strong VLM.
• Multi-view captions (factual + creative + motion-aware).
• Sora used GPT-4-class captioner.

17.3 Compute scale

• Sora-class training: \(\sim\) tens of millions of dollars; thousands of H100s for months.
• Open frontier (Hunyuan, Wan): \(\sim\) millions of dollars; hundreds of H100s.
• Mid-scale (Mochi, Open-Sora): \(\sim\) hundreds of thousands; tens of H100s.

17.4 Stage-wise training

1. Pretrain on images (transfer of visual priors).
2. Joint image+video at moderate resolution.
3. High-resolution finetuning.
4. Long-context (long clip) finetuning.
5. Domain-specific (audio, camera control, motion) finetuning.
6. Safety / alignment finetuning.

17.5 Open data availability

• Panda-70M: 70M video clips with captions.
• HD-VG-130M: 130M HD video clips.
• InternVid: 7M long-form videos.
• WebVid: legacy, smaller.

18. Evaluation

18.1 Per-frame metrics

• PSNR, SSIM, LPIPS (limited use for generated content).

18.2 Video-level metrics

• FVD (Fréchet Video Distance): I3D feature distance; standard.
• KVD: kernel variant.
• IS-V: video Inception Score.

18.3 VBench / VBench-2

Comprehensive benchmark with 16+ axes:

• Subject consistency.
• Background consistency.
• Motion smoothness.
• Dynamic degree.
• Temporal flicker.
• Human action.
• Multi-object handling.
• Object class coverage.
• Spatial relationships.
• Color, scene, style.
• Imaging quality.
• Aesthetic quality.
• Overall consistency.

Aggregated scores; standard for video gen comparison.

18.4 T2V-CompBench

Compositional benchmark: counting, attribute binding, spatial relations, motion, action over time.

18.5 EvalCrafter

Evaluation framework with 17 metrics; built on VBench + others.

18.6 VideoScore

Learned reward model for video quality. Trained on human preferences. Used in alignment + filtering.

18.7 Human evaluation

• Side-by-side preferences.
• Aesthetic, prompt fidelity, motion realism.
• Cost: $1–5 per pairwise comparison.
• Standard for frontier model comparison.

18.8 Image Arena / Video Arena

Crowd-sourced pairwise preferences with Elo / Bradley-Terry rankings. The closest to ground truth.

19. Failure Modes and Common Issues

19.1 Identity drift

Subject (face, object) drifts visually across frames. Mitigations: ReferenceNet, stronger identity conditioning, shorter clips.

19.2 Physics violations

• Objects appear / disappear.
• Limbs morph through objects.
• Liquids defy gravity.
• Shadows inconsistent with lighting.

19.3 Temporal flicker

High-frequency frame-to-frame changes that shouldn't exist. Mitigations: temporal regularization losses, EMA, longer chunks.

19.4 Hands and faces

Notoriously hard. Hands have complex articulation; faces have high human attention. Often fails before other content.

19.5 Long-clip incoherence

Beyond \(\sim\) 10s, scene drift becomes visible. Use chunked AR + anchor frames.

19.6 Multi-agent interaction

Two characters interacting (handshake, conversation) frequently fails (clipping, mismatched motions). Solutions: better data, structured representations.

19.7 Text in video

Generated text in scenes (signs, handwriting) often gibberish. Standard failure.

19.8 Action faithfulness (for world models)

Generated rollouts may not actually respond to specified actions. Needs explicit action-conditioning + verification.

20. Production Deployment

20.1 Deployment topologies

• Cloud API (Sora API, Veo API, Runway, Kling, Pika): pay-per-use.
• Self-hosted open (Hunyuan, Wan, Mochi): control + privacy.
• Hybrid: open base + commercial post-processing.

20.2 Latency engineering

• Step distillation (4–8 NFEs).
• Quantization (FP8, INT8).
• Batching multiple requests.
• Caching shared prefixes (text encoder).

20.3 Cost models

• Open self-hosted: GPU-hour costs.
• API: typically $0.10–$1 per second of video.
• Sora 2 / Veo 3: premium pricing for frontier.

20.4 Safety and provenance

• Watermarking (SynthID for Veo, C2PA across providers).
• NSFW / harmful content filters.
• Identity manipulation detection.
• Real-vs-synthetic detection.

20.5 Scaling tips

• Cache text embeddings.
• Cache VAE encode of reference images.
• Asynchronous generation pipeline.
• Pre-fetch user request batching.

20.6 Tools and ecosystem

• ComfyUI: node-based workflow; primary open platform.
• Forge: web UI variant.
• Diffusers (HuggingFace): standard Python library for video gen.
• LoRA marketplaces: Civitai for community LoRAs.

21. Frontier 2025–2026

21.1 Trends

• Native audio (Veo 3, Sora 2, Wan 2.2, MovieGen) becomes standard.
• Long-clip (\(\sim\) 60s) coherent generation viable.
• World-model framing (action conditioning) blurring video gen / world model boundary.
• Real-time generation (LTX-Video, distilled SDXL-Video) for interactive use.
• Multi-modal control (text + image + camera + audio + sketch) widespread.
• Open frontier closing gap with closed (Wan, Hunyuan competitive).

21.2 Hot research directions

• Multi-minute coherent generation.
• Physical commonsense (hybrid neural + analytical).
• Editability mid-generation.
• Causal / interventional video gen.
• Multi-agent dynamics.
• Real-time on consumer hardware.
• Fine-grained motion control.

21.3 Open problems

• Identity preservation in long clips.
• Hand and face generation.
• Multi-object physics.
• Text rendering in scenes.
• Causal correctness.
• Evaluation rigor (most metrics game-able).

21.4 Production reality (April 2026)

• Frontier closed (Sora 2, Veo 3): premium creative + advertising.
• Frontier open (Wan 2.2, Hunyuan): self-hosted creative.
• Real-time (LTX, distilled): interactive products.
• Animation (AnimateAnyone, EMO, Live Portrait): specific verticals.
• World models (Cosmos, GAIA): robotics / AV.

22. Production Stack 2026

Appendix A: Twenty-Five Things to Know

1. Causal 3D VAE: \(T/4 \times H/8 \times W/8\) standard compression.
2. Spatiotemporal MM-DiT: text + video joint self-attention.
3. Joint image + video training: \(T = 1\) for images.
4. 3D-RoPE on \((t, h, w)\) video tokens.
5. Diffusion vs autoregressive for video: parallel quality vs indefinite horizon.
6. Sora (Feb 2024): spacetime patches, "world simulator."
7. Sora 2 (Sep 2025): + audio, longer clips, physics improvement.
8. Veo 3 (May 2025): native audio + improved fidelity.
9. Hunyuan Video (Tencent, 13B, open frontier).
10. Wan 2.1 / 2.2 (Alibaba, 14B+, open).
11. Mochi 1 (Genmo, 10B AsymmDiT, open Apache).
12. LTX-Video (Lightricks, real-time on A100).
13. Step-Video (StepFun, 30B, largest open).
14. AnimateAnyone / MimicMotion: pose-driven character animation.
15. EMO / Live Portrait / Audio2Photoreal: talking-head animation.
16. ReferenceNet for identity preservation.
17. Re-captioning of training data is critical.
18. VBench / VBench-2: 16+ axis evaluation standard.
19. FVD: Fréchet Video Distance via I3D / InternVideo.
20. ControlNet for video (CogVideoX-Control, Wan-Control).
21. LoRA per-style / per-character standard.
22. Step distillation: 4–8 NFEs (LCM-Video, Hyper-Video, DMD2-Video).
23. Block-cache / TGATE / PAB for inference speed.
24. World-model framing: video gen + action conditioning.
25. Open frontier (Wan, Hunyuan) closing gap with closed.

Appendix B: Decision Tree — "Which Video Gen?"

1. Need premium creative quality, willing to pay? \(\to\) Sora 2 / Veo 3 / Kling 2 (closed APIs).
2. Self-hosted open frontier? \(\to\) Hunyuan Video or Wan 2.1/2.2.
3. Real-time interactive? \(\to\) LTX-Video or distilled Mochi.
4. Need native audio? \(\to\) Veo 3 (closed) or Wan 2.2 / MovieGen (open).
5. Image-to-video? \(\to\) SVD / Hunyuan I2V / Kling I2V.
6. Pose-driven character? \(\to\) AnimateAnyone / MimicMotion / Champ.
7. Audio-driven portrait? \(\to\) EMO / Live Portrait.
8. Camera control? \(\to\) Veo / Sora / Luma (closed) or CameraCtrl (open).
9. Long-form coherent (\(> 30\text{s}\))? \(\to\) Sora 2 / Veo 3 or chunked AR with anchor frames.
10. Action-conditioned (world model for robotics / AV)? \(\to\) Cosmos Predict or GAIA-2.
11. Custom style / character? \(\to\) LoRA on Hunyuan or Wan via ComfyUI.
12. Real-time playable? \(\to\) Genie 2 / Oasis / GameNGen (specialized).

Appendix C: Year-by-Year Video Generation Milestones

• 2022: Make-A-Video, Imagen Video, Phenaki — early T2V research.
• 2023: ModelScope T2V (open baseline); AnimateDiff (SD + temporal modules); SVD / SVD-XT (Stability I2V); ZeroScope; Show-1.
• Feb 2024: Sora (OpenAI) — transformative quality jump; "world simulator."
• May 2024: Veo 1 (Google).
• 2024 (mid): Kling 1 (Kuaishou); Pika 1.0; Runway Gen-3; Luma Dream Machine; CogVideoX 5B (Tsinghua, open); Open-Sora; AnimateAnyone (Alibaba); EMO; MimicMotion.
• 2024 (late): Mochi 1 (Genmo, open); Hunyuan Video (Tencent, 13B open frontier); LTX-Video (real-time); Allegro; Step-Video; MovieGen (Meta, joint video+audio); Cosmos World Foundation Models (NVIDIA); Genie 2 (DeepMind, playable 3D).
• 2025 (early): Veo 2 with camera control; Wan 2.1 (Alibaba, open frontier); Hunyuan Video Avatar variants.
• May 2025: Veo 3 (Google) — native audio.
• Sep 2025: Sora 2 (OpenAI) — audio + longer + physics improvement.
• 2025 (late): Wan 2.2 (audio support); Hunyuan Video improvements; Cosmos Predict 2; LTX-Video 2.
• 2026: native multimodal video + audio standard; world-model framing mainstream; long-form (> 1 min) viable; real-time on consumer hardware via distillation; open frontier (Wan, Hunyuan) competitive with closed.