A — Things to Memorize (Master Merge)

A single merged study sheet consolidating the "things to memorize" appendix from every cheat sheet in this knowledge base (36 files). Each source's list is preserved under its topic; paper links are kept intact.

Compiled August 2026. Organized into six thematic parts; a synthesized cross-cutting summary is up top. Source section titles vary ("Things to Know", "Equations to Memorize", "Numbers Worth Memorizing", "…You Must Own Cold") but all serve the same role.

Contents

• Cross-Cutting Principles
• Part I — LLMs: Architecture, Tokenization & Efficiency
  • Attention
  • Transformer / xFormer Catalogue
  • KV Cache
  • Mixture-of-Experts (MoE)
  • Tokenization & Context (v1)
  • Tokenization & Context (v2)
  • Quantization
  • Pruning
  • Distillation
  • Parameter-Efficient Fine-Tuning (PEFT)
  • Scaling Laws
• Part II — LLMs: Training, Reasoning, RL & Agents
  • Foundation Models
  • Reasoning Technologies
  • RL Training Strategies & Recipes
  • Policy Optimization
  • Reward Functions
  • Agentic Intelligence
  • Prompt · Context · Harness · Graph Engineering & Self-Improving AI
  • Test-Time & Training-Free Optimization
• Part III — Data & Evaluation
  • Data Collection & Curation
  • Metrics & Evaluations
• Part IV — Generative Models: Diffusion, Video, 3D & World
  • Diffusion Models
  • Diffusion — Derivations
  • Video Generation
  • 3D & Multi-View Generation
  • World Models
• Part V — Neural Rendering & 3D Reconstruction
  • NeRF
  • Gaussian Splatting
  • Neural Rendering
  • Structure from Motion
  • Delighting & Relighting
  • Photorealistic Avatars
• Part VI — Core Vision, Robotics & Autonomy
  • Computer Vision — Principal Deep Dive
  • Computer Vision — Principal Math
  • Vision-Language-Action (VLA) Models
  • Autonomous Driving

Cross-Cutting Principles (read these first)

Fifteen ideas that recur across every sheet — the compression of the compression.

1. Scaling is lawful, and there are now two axes. Pretraining follows Chinchilla-style compute-optimal token/parameter balance; since 2024, test-time compute (search, longer thinking) is a co-equal scaling axis that can beat parameter scaling at matched FLOPs.
2. Attention is the bottleneck, and the stack that fixed it is settled. \(O(n^2)\) memory → FlashAttention (exact, IO-aware) + GQA/MLA (KV-cache) + RoPE is the default modern backbone.
3. Context is a finite budget, not a free window. Retrieve just-in-time, compact, and isolate sub-agents; bigger windows ≠ solved (Context Rot, Lost-in-the-Middle).
4. The harness often beats the checkpoint. Same weights, different loop/tools/skills → large capability delta; Agent Skills are the composable capability layer.
5. Improvement is verifier-bounded. Self-consistency, best-of-\(N\), RLVR, self-play, and recursive self-improvement only go as far as their checker/signal is correct — this is the single most repeated caveat in the corpus.
6. Diffusion converged on flow matching. $arepsilon$-prediction ↔︎ score; classifier-free guidance is the workhorse; distill to 1–4 steps for deployment.
7. 3DGS overtook NeRF for most new reconstruction (real-time, explicit), but they share the volume-rendering core; feed-forward pose/geometry (VGGT/π³) is the 2025–26 shift.
8. Compress by composing quantization + pruning + distillation. Outliers are the enemy — rotation/SmoothQuant-style fixes; FP8/FP4 and 2:4 sparsity are production-real.
9. Alignment math is a small family. The RLHF optimum is a KL-tilted reference; DPO removes the reward model; GRPO removes the value net; RL from verifiable rewards powers reasoning models.
10. Self-improvement has three failure modes. Reward hacking (Goodhart), model collapse (recursive self-generated data), and the intrinsic self-correction gap (LLMs can't reliably self-correct reasoning without external feedback).
11. Reasoning models internalize long CoT. Budget the thinking; don't hand-scaffold it. Process rewards beat outcome rewards for verifier-guided search.
12. Reach for training-free levers first. Decoding contrasts, activation steering, model merging (task arithmetic), and speculative decoding change behavior/speed with no training run.
13. Adapt at test time when the distribution shifts. Entropy-minimization TTA (guard against class collapse), self-supervised inner-loop TTT, per-task LoRA; TTT-layers make the hidden state itself a learner.
14. Everything is converging on multimodal world models as the substrate — native multimodal + long-context video, with VLA policies for robotics and autonomous driving.
15. Evaluation is the hard part. Contamination, saturation, and LLM-as-judge caveats mean benchmarks lag true capability — trust verifiable/held-out signals over leaderboard deltas.

Part I — LLMs: Architecture, Tokenization & Efficiency

Attention

Merged from "Twenty Things to Memorize" — Attention_SOTA_Updated.md.

1. Scaled dot-product equation and the \(\sqrt{d_k}\) argument.
2. Stable softmax (subtract row-max).
3. Online softmax recurrence (FlashAttention).
4. RoPE rotation matrix and its relative-position invariance proof.
5. ALiBi linear bias formula.
6. KV-cache bytes-per-token formula: \(2 \cdot L\, H\, d_h\) bytes.
7. GQA vs MQA vs MLA trade-offs.
8. Paged KV cache: virtual memory analogy.
9. Sliding window + sink token rationale.
10. Linear attention kernel-trick reformulation.
11. FlashAttention I/O complexity argument.
12. Ring attention's P-step rotation.
13. Swin shifted-window mechanism.
14. Deformable attention sampling formula.
15. U-Net cross-attention for SD; MM-DiT joint attention for SD3.
16. ToMe-SD bipartite merging.
17. Self-attention guidance (SAG) and PAG formulas.
18. Prompt-to-Prompt cross-attn manipulation patterns.
19. Speculative decoding accept/reject probability.
20. adaLN-zero conditioning for DiT blocks.

Transformer / xFormer Catalogue

Merged from "Pictures Worth Memorizing" — XFormer_Catalogue_SOTA_Updated.md.

For each, you should be able to draw the architecture cleanly on a whiteboard:

1. Original Transformer (encoder-decoder).
2. BERT block (pre-norm vs post-norm).
3. GPT block (causal mask).
4. T5 (encoder + decoder with cross-attn).
5. ViT (patch + CLS + Transformer).
6. Swin (hierarchical + shifted window).
7. DETR (encoder + object-query decoder + Hungarian matching).
8. Mask2Former (universal mask predictor).
9. CLIP (dual encoder + InfoNCE).
10. BLIP-2 (frozen image \(\to\) Q-Former \(\to\) frozen LLM).
11. LLaVA (CLIP \(\to\) MLP \(\to\) LLM).
12. Flamingo (gated cross-attn into frozen LLM).
13. Chameleon (interleaved tokens, single AR Transformer).
14. DiT (adaLN-Zero on noised tokens).
15. MM-DiT (two streams + joint attention).
16. Mixtral / Mixture-of-Experts block (router + top-k experts).
17. Mamba (selective SSM + parallel scan).
18. Perceiver (latent array \(\to\) cross-attn \(\to\) refine).
19. RT-2 / OpenVLA (VLM + action token vocab).
20. \(\pi_0\) (VLM + flow-matching action head).

KV Cache

Merged from "Twenty-Five Things to Know" — KV_Cache_SOTA_Updated.md.

1. KV bytes/token formula: \(2 \cdot L \cdot H_{kv} \cdot d_h \cdot\) bytes dtype.
2. Llama-3-70B GQA-8 in FP16: 320 KB/token.
3. DeepSeek-V3 MLA: \(\sim 70\) KB/token (\(14\times\) smaller than equivalent MHA).
4. Prefill is FLOP-bound; decode is bandwidth-bound.
5. GQA-8 is the 2024 standard; MLA is the 2025–26 frontier.
6. MLA absorbs up-projections into \(W_Q\), \(W_O\) at inference.
7. Decoupled RoPE: split each head into RoPE + non-RoPE components.
8. KV quantization: per-channel for \(K\), per-token for \(V\) (KIVI).
9. INT4 KV typically loses 1–2% quality; INT2 loses 5–10%.
10. FP8 (E5M2) for KV; FP8 (E4M3) for forward.
11. PagedAttention block size: 16 tokens default.
12. Automatic prefix caching: hash blocks; reuse on match.
13. SGLang RadixAttention: prefix tree of cached blocks.
14. Sliding window + sinks (StreamingLLM) for unbounded streaming.
15. Sinks (\(k \sim 4\)) prevent softmax-redistribution collapse.
16. H2O eviction: keep heavy hitters + recent.
17. SnapKV: prefill-time importance pooling.
18. Pyramid KV: deeper layers retain fewer tokens.
19. FastV: drop vision tokens after layer \(K\) in VLMs.
20. Quest: query-aware KV block retrieval.
21. Continuous batching: per-token, not per-request.
22. DistServe / Mooncake: disaggregated prefill / decode.
23. Anthropic prompt cache: 90% discount on cached prefix.
24. Tree attention (Medusa/EAGLE) needs CoW paged KV.
25. Mamba / RWKV / RetNet replace KV with constant-size state.

Mixture-of-Experts (MoE)

Merged from "Twenty-Five Things to Know" — MoE_SOTA_Updated.md.

1. Total vs active params: always cite both.
2. Top-k routing: k = 1 Switch, k = 2 standard, k = 8 DeepSeek fine-grained.
3. Switch's load-balance loss: \(\alpha N \sum f_i p_i\).
4. Router z-loss for stability.
5. Capacity factor \(\sim 1.25\) training, \(\sim 2\) inference.
6. DeepSeek's auxiliary-loss-free trick: per-expert bias \(b_i\).
7. Fine-grained experts + shared experts (DeepSeekMoE).
8. Sparse upcycling: dense \(\to\) MoE cheaply.
9. Branch-Train-MiX: independent training \(\to\) assemble.
10. Soft MoE for vision: no hard routing, no token drop.
11. Expert Choice routing: experts pick tokens.
12. Hash MoE: fixed routing baseline.
13. EP all-to-all: dispatch + combine per layer.
14. Composing EP × TP × PP × DP.
15. MegaBlocks block-sparse GEMM kernels.
16. Tutel adaptive parallelism.
17. Continuous batching with per-token expert dispatch.
18. ktransformers for consumer-hardware MoE.
19. DeepSeek-V3: 671B / 37B / FP8 / MTP / MLA.
20. Mixtral 8×7B as the open MoE that broke through.
21. Llama 4 brought MoE into the Llama line.
22. MoE wins at fixed active params; dense wins at fixed memory.
23. Routing collapse fix: aux loss + z-loss + capacity.
24. Mixture-of-Depths: sparse over depth, not width.
25. Shared expert pattern: 1–2 always-on for common knowledge.

Tokenization & Context (v1)

Merged from "Twenty-Five Things to Memorize" — Tokenization_Context_SOTA_Updated.md.

1. BPE merge algorithm; greedy frequency-based merges.
2. WordPiece score formula: \(\mathrm{count}(ab)/(\mathrm{count}(a)\,\mathrm{count}(b))\).
3. Unigram LM training via EM for SentencePiece.
4. Byte-level BPE: no UNK ever.
5. ChatML and Llama 3 chat template structure.
6. Number tokenization: digit-split via regex (Llama 3, GPT-4o).
7. Multilingual tokenizer fairness problem.
8. ViT patch embedding: \(W_p\) as a stride-\(p\) conv.
9. Class token vs average pooling for ViT classification.
10. Pixel unshuffle for \(r^2\) token compression in VLMs.
11. LLaVA-NeXT AnyRes tiling pattern.
12. Native dynamic resolution + 2D-RoPE (Qwen2-VL).
13. Register tokens for absorbing high-norm artifacts.
14. VQ-VAE training loss with commitment term.
15. LFQ: \(q = \mathrm{sgn}(z)\), no codebook, vocab \(2^L\).
16. FSQ: per-dim rounding to a small set, no entropy reg.
17. Causal 3D VAE for video: \((T/4)\,(H/8)\,(W/8)\) compression.
18. Position Interpolation: \(p \to p \cdot L_{\text{train}}/L_{\text{eval}}\).
19. NTK-aware RoPE base scaling.
20. YaRN's piecewise-by-frequency rescaling + temp adjust.
21. Sliding window + sink tokens (StreamingLLM).
22. Sequence packing with block-diagonal attention mask.
23. Loss masking in SFT: -100 on user / system positions.
24. Prefix caching / Anthropic prompt cache cost model.
25. RAG vs long-context trade-off framework.

Tokenization & Context (v2)

Merged from "Twenty-Five Things to Memorize" — Tokenization_Context_SOTA_Updated_v2.md.

1. BPE merge algorithm; greedy frequency-based merges.
2. WordPiece score formula: \(\text{count}(ab)/(\text{count}(a)\,\text{count}(b))\).
3. Unigram LM training via EM for SentencePiece.
4. Byte-level BPE: no UNK ever.
5. ChatML and Llama 3 chat template structure.
6. Number tokenization: digit-split via regex (Llama 3, GPT-4o).
7. Multilingual tokenizer fairness problem.
8. ViT patch embedding: \(W_p\) as a stride-\(p\) conv.
9. Class token vs average pooling for ViT classification.
10. Pixel unshuffle for \(r^2\) token compression in VLMs.
11. LLaVA-NeXT AnyRes tiling pattern.
12. Native dynamic resolution + 2D-RoPE (Qwen2-VL).
13. Register tokens for absorbing high-norm artifacts.
14. VQ-VAE training loss with commitment term.
15. LFQ: \(q = \text{sgn}(z)\), no codebook, vocab \(2^L\).
16. FSQ: per-dim rounding to a small set, no entropy reg.
17. Causal 3D VAE for video: \((T/4)\cdot(H/8)\cdot(W/8)\) compression.
18. Position Interpolation: \(p \to p \cdot L_{\text{train}}/L_{\text{eval}}\).
19. NTK-aware RoPE base scaling.
20. YaRN's piecewise-by-frequency rescaling + temp adjust.
21. Sliding window + sink tokens (StreamingLLM).
22. Sequence packing with block-diagonal attention mask.
23. Loss masking in SFT: \(-100\) on user / system positions.
24. Prefix caching / Anthropic prompt cache cost model.
25. RAG vs long-context trade-off framework.

Quantization

Merged from "Twenty-Five Things to Know" — Quantization_SOTA_Updated.md.

1. Affine quant: \(\hat{x} = s \cdot \operatorname{round}((x - z)/s) + z.\)
2. Symmetric vs asymmetric (\(z = 0\) vs not).
3. Granularity: per-tensor / per-channel / per-token / per-block.
4. BF16 has same range as FP32 (8 exp bits).
5. FP8: E4M3 forward, E5M2 backward.
6. FP4 (E2M1) on Blackwell at 2× FP8 throughput.
7. Microscaling (MXFP4, MXFP6): per-block FP standard.
8. GPTQ: Hessian-based per-column quantization with update.
9. AWQ: salient-channel scaling formula.
10. SmoothQuant: migrate scale from activations to weights.
11. QuaRot: orthogonal rotation spreads outliers.
12. LLM.int8(): outlier channels in FP16, rest INT8.
13. KIVI: per-channel K, per-token V.
14. NF4: 4-bit format for normal-distributed weights.
15. GGUF K-quants: per-layer adaptive precision.
16. QLoRA: NF4 base + BF16 LoRA adapters.
17. Marlin / Machete: fast W4 GEMM kernels.
18. DeepSeek-V3 FP8 recipe: per-block + double-quant + custom kernels.
19. BitNet b1.58: ternary weights; matches BF16 at > 3B with retraining.
20. PTQ vs QAT: post-training cheap vs aware-training quality.
21. STE: gradient passes through round.
22. SVDQuant: low-rank residual + INT4 for FLUX.
23. Per-expert calibration for MoE quantization.
24. ktransformers: GGUF Q4 + offload for 671B on workstation.
25. Quantization × distillation × sparsity = compounding gains.

Pruning

Merged from "Twenty-Five Things to Know" — Pruning_SOTA_Updated.md.

1. Three pruning targets: parameters / Gaussians / tokens.
2. Magnitude pruning: prune \(|w| < \tau\); surprisingly competitive baseline.
3. Lottery Ticket Hypothesis: sparse subnets in dense networks.
4. IMP: prune → reset → retrain.
5. Movement pruning: \(\text{sign}(w \cdot \nabla L) \cdot |w|\).
6. Pruning-at-init (SNIP / GraSP / SynFlow).
7. Random pruning is a strong baseline.
8. 2:4 N:M sparsity: hardware-supported on H100 / Blackwell, 2× matmul.
9. SparseGPT: layer-wise OBS with Hessian.
10. Wanda: \(|w| \cdot \|x\|\); cheap activation-aware.
11. ShortGPT: layer pruning via Block Importance.
12. SliceGPT: orthogonal slicing of hidden dims.
13. LLM-Pruner: structured channel pruning.
14. Pruning + LoRA recovery: standard recipe.
15. 3DGS density control: clone / split / prune / opacity reset.
16. LightGaussian: importance-based pruning + INT8 + SH distill.
17. CompGS: codebook compression for 3DGS.
18. ToMe (Bolya & Hoffman): bipartite soft-matching merge.
19. ToMeSD: ToMe in SD U-Net.
20. DynamicViT: learned per-token importance.
21. FastV: drop vision tokens after layer 2–4 in VLM.
22. H2O: heavy hitters + recent for KV eviction.
23. SnapKV: prefill-time importance pooling.
24. Pyramid KV: deeper layers retain fewer tokens.
25. StreamingLLM: sinks + sliding window for unbounded streaming.

Distillation

Merged from "Twenty-Five Things to Know" — Distillation_SOTA_Updated.md.

1. Hinton's KD loss with \(\tau^2\) scaling.
2. Forward KL: mass-covering; reverse KL: mode-seeking.
3. Sequence KD (teacher argmax) often as good as token-level.
4. GKD: on-policy distillation; samples from student.
5. FitNets: feature-level distillation with projection.
6. Attention transfer: match attention maps.
7. RKD: relational (pairwise distances + angles).
8. DeiT distillation token: special token for teacher signal.
9. DINO/DINOv2: self-distillation as SSL with EMA + sharpening + centering.
10. Mean Teacher: EMA-of-student as semi-supervised target.
11. EMA in diffusion: required for sample fidelity.
12. DistilBERT loss: MLM + KL + cosine.
13. TinyBERT two-stage: general distill then task distill.
14. MiniLM: distill last-layer attention only.
15. Phi family: distillation via synthetic textbook data.
16. Open-instruction pattern: filter teacher data + SFT smaller model.
17. R1-Distill series: long-CoT traces → smaller student.
18. Distill cold-start enables small-model GRPO.
19. Progressive distillation: halve steps each round.
20. Consistency Models: predict \(x_0\) from any \(t\).
21. DMD2: one-step diffusion via score matching + GAN.
22. LCM / Hyper-SD / DMD2 / Lightning are the standard SDXL/FLUX few-step distills.
23. Sparse upcycling: dense → MoE via expert replication.
24. Constitutional distillation: critique + revise + train.
25. Cascades + distillation: 70–90% cost reduction.

Parameter-Efficient Fine-Tuning (PEFT)

Merged from "Twenty-Five Things to Know" — PEFT_SOTA_Updated.md.

1. LoRA: \(W' = W + \alpha \cdot BA/r\); \(A\) Gaussian, \(B\) zero.
2. Standard rank: 8–64 for LLM SFT; 16–128 for diffusion.
3. LoRA matches full FT at ≥7B for many tasks at 1/100–1/1000 params.
4. QLoRA: NF4 base + BF16 LoRA + paged optimizer + double quant.
5. Unsloth: 2–5× QLoRA training speedup.
6. DoRA: magnitude + direction decomposition; closer to full FT than LoRA.
7. rsLoRA: \(\alpha/\sqrt{r}\) scaling; better at high rank.
8. ReLoRA: iteratively merge + re-init for higher effective rank.
9. OFT / BOFT: orthogonal rotation; preserves spectrum.
10. VeRA: shared A, B across layers; even fewer params.
11. LoHa, LoKr: Hadamard / Kronecker decompositions.
12. IA3: per-feature scaling vectors; tiny param count.
13. BitFit: train only biases.
14. Adapters (Houlsby / Pfeiffer): bottleneck modules.
15. Soft prompts: learnable embedding tokens prepended.
16. Prefix tuning: per-layer learnable KV prefixes.
17. LoRA target modules: attention (Q,K,V,O) + MLP (up, down).
18. For DPO: LoRA + ref = base (no memory doubling).
19. For RL (GRPO): cold-start SFT-LoRA → RL-LoRA.
20. Multi-LoRA serving: vLLM / S-LoRA / LoRAX.
21. LoRA stacking: \(W + \sum_i \alpha_i B_i A_i\).
22. LoCon / LoHa / LoKr / DoRA in LyCORIS for diffusion.
23. LoRA + FSDP / DeepSpeed ZeRO for multi-GPU PEFT.
24. Layer-importance / AdaLoRA for adaptive rank allocation.
25. Don't use PEFT for: massive shift, vocab change, very long continual.

Scaling Laws

Merged from "Twenty-Five Things to Know" — Scaling_Laws_SOTA_Updated.md.

1. Scaling laws are empirical power laws: \(L = A X^{-\alpha} + L_\infty\).
2. Kaplan (2020): scale params faster than data; \(\alpha_N = 0.076\).
3. Kaplan was wrong; Chinchilla (2022) corrected.
4. Chinchilla rule: 20 tokens per parameter at compute-optimal.
5. \(C \approx 6ND\) for Transformer training.
6. Chinchilla loss: \(L = E + A/N^{\alpha} + B/D^{\beta}\).
7. \(\alpha \approx 0.34\), \(\beta \approx 0.28\), \(E \approx 1.69\).
8. Llama 3 8B: 15T tokens / 8B = 1875; over-trained.
9. Inference-aware scaling shifts smaller \(N\), more \(D\).
10. Snell et al.: 14× smaller model with optimal test-time compute matches larger.
11. Best-of-\(N\): \(\mathbb{E}[\max] \approx \mu + \sigma \sqrt{2 \ln N}\).
12. MoE: \(L\) scales with \(N_{\mathrm{active}}\) at fixed data.
13. Optimal MoE sparsity: \(N_{\mathrm{total}}/N_{\mathrm{active}} \in [10, 30]\).
14. DeepSeek-V3: 671B total / 37B active / 14T tokens.
15. ViT-22B: parallel attn+MLP, QK-LN, no biases.
16. DiT scaling: power-law in FID with model + compute.
17. Distillation: ~0.1×–1× pretraining compute of student.
18. Born-Again students often beat teacher.
19. FineWeb-Edu evidence: data quality ~5–10× more efficient.
20. Repeated data: ~0.6× effective per epoch; 4 epochs cap.
21. µP: width-aware parameterization for HP transfer.
22. Schaeffer: emergent abilities may be metric artifact.
23. Hardware utilization: typically 40–55% of peak.
24. Inference cost often > 10× training cost over model lifetime.
25. Multi-axis scaling: pretrain + post-train + inference all matter.

Part II — LLMs: Training, Reasoning, RL & Agents

Foundation Models

Merged from "Names to Recognize on Sight" — Foundation_Models_SOTA_Updated.md.

If an interviewer says these, you should know exactly what they refer to:

• CLIP / SigLIP / DINOv3: vision encoders.
• SAM 2: promptable segmentation.
• Grounding DINO: open-vocab detection.
• DUSt3R / MASt3R / VGGT: feed-forward 3D.
• 3DGS / 3D Gaussian Splatting: scene rep.
• MAE / V-JEPA: SSL recipes.
• LLaVA / InternVL / Qwen-VL: open VLMs.
• GPT-4o / Gemini 2.5 / Claude Opus: closed VLMs.
• Chameleon / Show-o / Emu3: native multimodal.
• SDXL / SD3 / FLUX: image diffusion.
• MM-DiT: image diffusion architecture.
• Sora / Veo 3 / Kling 2 / Hunyuan Video / Wan 2.1: video gen.
• MovieGen: joint video + audio.
• LRM / GS-LRM / MeshLRM: feed-forward 3D.
• Trellis / Hunyuan3D-2: native 3D generation.
• π0 / OpenVLA / GR00T / Helix: VLAs.
• GAIA / Cosmos / Genie 2: world models.
• Whisper: ASR.
• Stable Audio / MusicGen / Suno: audio gen.
• Codec Avatars / EMO / Live Portrait: avatars.
• R1 / VLM-R1 / Vision-R1: reasoning models.
• ControlNet / IP-Adapter: conditioning.

Reasoning Technologies

Merged from "Twenty-Five Things to Memorize" — Reasoning_Technologies_SOTA_Updated.md.

1. Zero-shot CoT trigger: "Let's think step by step."
2. Few-shot CoT exemplar pattern.
3. Self-consistency: \(K\) samples + majority vote.
4. Tree-of-Thought: branch + evaluate + search.
5. Best-of-\(N\) scaling: \(\mathbb{E}[\max] \approx \mu + \sigma\sqrt{2\ln N}\).
6. PRM training: per-step binary cross-entropy.
7. Math-Shepherd auto-labeling rule.
8. Implicit PRM via DPO closed-form.
9. GRPO advantage: group-relative z-score.
10. GRPO clipped per-token objective.
11. Verifiable-reward composition (format + accuracy + length + lang).
12. R1 4-stage training pipeline.
13. R1-Zero pure-RL emergence.
14. Snell et al. inference-time scaling: \(14\times\) smaller w/ TT compute.
15. ReAct loop pattern.
16. Reflexion episodic memory.
17. PAL: code as reasoning trace.
18. AlphaGeometry / AlphaProof = LLM + symbolic + search.
19. rStar two-LM MCTS pattern.
20. Multi-Agent Debate convergence.
21. Visible vs hidden CoT trade-offs.
22. Faithfulness gap warning.
23. Self-consistency as uncertainty estimator.
24. Pareto-optimal compute allocation by difficulty.
25. Distill long-CoT traces to bootstrap small models.

RL Training Strategies & Recipes

Merged from "Twenty Things a Senior RL Engineer Should Just Know" — RL_Training_Strategies_Recipes_CheatSheet_with_Links.md.

1. Adam \(\epsilon = 10^{-5}\) for RL.
2. Reward normalization by std, not mean.
3. Observation normalization is necessary for MuJoCo.
4. GAE with \(\lambda = 0.95\).
5. KL anchor is non-negotiable in RLHF.
6. Length normalize RM data.
7. Use ensembles of RMs.
8. Filter trivial-pass prompts in GRPO.
9. Dr. GRPO removes length bias.
10. DPO needs only 1 epoch.
11. DPO LR is much smaller than SFT LR.
12. For diffusion DPO, \(\beta \sim 1000\), not 0.1.
13. Sample-and-read your policy outputs every checkpoint.
14. Privileged-to-vision distillation closes most sim-to-real gaps.
15. Eureka can design rewards better than humans.
16. Vectorized envs first; distributed only when GPU saturated.
17. LoRA + frozen base saves \(2\times\) memory in DPO/GRPO.
18. Pre-compute reference log-probs once per epoch.
19. AlpacaEval 2 + Arena-Hard correlate well with human prefs.
20. The held-out human eval is the only ground truth.

Policy Optimization

Merged from "Twenty-Five Equations to Memorize" — Policy_Optimization_SOTA_Updated.md.

1. Policy gradient theorem in advantage form.
2. GAE recurrence.
3. REINFORCE update with baseline.
4. TRPO trust-region constraint.
5. PPO clipped surrogate.
6. DPG theorem.
7. DDPG critic loss with target networks.
8. TD3 clipped double-Q + target smoothing.
9. SAC max-entropy objective and soft Bellman.
10. SAC's auto-tuned \(\alpha\) loss.
11. DQN Bellman target with target network.
12. Distributional Bellman update.
13. Decision Transformer conditioning on returns-to-go.
14. Bradley-Terry RM loss.
15. KL-regularized RLHF objective and closed-form \(\pi^*\).
16. DPO loss derivation cancelling \(\log Z(x)\).
17. IPO squared-margin loss.
18. KTO prospect-utility loss.
19. ORPO odds-ratio combined SFT+pref.
20. SimPO length-normalized loss.
21. GRPO group-relative advantage.
22. GRPO per-token clipped objective.
23. RLOO leave-one-out advantage.
24. Diffusion-DPO surrogate.
25. Asymmetric A-C distillation \(\mathcal{L}_{\mathrm{distill}}\).

Reward Functions

Merged from "Twenty-Five Key Equations and Patterns" — Reward_Functions_SOTA_Updated.md.

1. Discounted return \(G_t = \sum \gamma^k r_{t+k+1}\).
2. Potential-based shaping invariance: \(F = \gamma\Phi(s') - \Phi(s)\).
3. HER relabeling: replace goal with achieved.
4. Bradley-Terry preference probability.
5. BT reward-model loss.
6. Plackett-Luce listwise probability.
7. PRM step-level binary loss.
8. Math-Shepherd auto-labeling rule.
9. Implicit PRM via DPO closed form.
10. Verifiable math reward template.
11. Verifiable code test-pass fraction.
12. Format reward via regex.
13. Detection IoU reward.
14. Dice / Mask IoU.
15. CIDEr / CLIP-Score / VQAScore composition.
16. Locomotion composite reward template.
17. AMP / DeepMimic Gaussian-error mimic reward.
18. Eureka loop steps.
19. Constrained MDP with Lagrangian update.
20. ICM curiosity error in feature space.
21. RND novelty error.
22. NovelD differential novelty.
23. KL-anchored RLHF objective.
24. Reward overoptimization curve \(a\sqrt{\text{KL}} - b\, \text{KL}\).
25. Length-normalized RM regression.

Agentic Intelligence

Merged from "Twenty-Five Things to Know" — Agentic_Intelligence_SOTA_Updated.md.

1. ReAct loop: Thought \(\to\) Action \(\to\) Observation.
2. Plan-and-Execute beats pure ReAct on long horizons.
3. Reflexion: episodic reflection in memory.
4. Tool description quality > tool implementation quality (model uses what it understands).
5. JSON schema: prefer enums, shallow nesting, required fields.
6. Always sandbox code execution; never assume safety.
7. Treat tool outputs as untrusted (prompt injection defense).
8. MCP standardizes tool discovery + invocation.
9. Set-of-Mark prompting improves GUI click accuracy dramatically.
10. Anthropic prompt caching: up to 90% cost savings on prefix.
11. Continuous batching: per-token, not per-request, kernel batching.
12. Re-rank with cross-encoder after vector retrieval.
13. HyDE often beats raw question embedding for retrieval.
14. GraphRAG wins for multi-hop on structured knowledge.
15. Multi-agent only when single agent demonstrably worse.
16. LangGraph for stateful production agents.
17. DSPy for declarative, optimization-driven flows.
18. Voyager skill library + auto-curriculum for open-ended learning.
19. Always log per-step traces; replay-able trajectories.
20. Doom loops: detect N-step repetition; back off / replan.
21. Model routing: cheap-first cascade saves \(\sim 70\%\) cost.
22. OSWorld / WebArena / SWE-bench for cross-domain agent eval.
23. Human-in-loop for irreversible / high-stakes actions.
24. CLAUDE.md / AGENTS.md for persistent project context.
25. Don't use an agent when one well-prompted call suffices.

Prompt · Context · Harness · Graph Engineering & Self-Improving AI

Merged from "Thirty-One Things to Memorize" — Prompt_Context_Harness_Graph_Engineering_SOTA_Updated.md.

1. The stack: prompt ⊂ context ⊂ harness, with graph cutting across and self-improvement wrapping all.
2. Same model, different harness → large capability delta.
3. Chain-of-Thought and Zero-shot CoT ("let's think step by step").
4. Self-Consistency: majority vote over sampled chains; highest-ROI CoT add-on.
5. Chain → Tree (ToT) → Graph (GoT) of thoughts.
6. ReAct = reason + act + observe; the canonical agent loop.
7. Don't over-prompt reasoning models — budget reasoning, don't scaffold it.
8. Test-time compute: PRMs, compute-optimal scaling, DeepSeek-R1.
9. Programmatic prompting: DSPy (signatures/compilers), TextGrad, ★ GEPA (reflection beats RL at fewer rollouts).
10. ICL theory: induction heads, Bayesian inference, implicit GD; label correctness matters less than format/distribution.
11. Instruction hierarchy + spotlighting; indirect prompt injection is the top agent risk.
12. Context engineering = managing the whole dynamic window as a finite budget.
13. Lost in the Middle and ★ Context Rot: bigger window ≠ solved.
14. ★ NoLiMa: literal needle-in-haystack overstates long-context reasoning.
15. RAG: original, HyDE, hybrid + ColBERT rerank, ★ Contextual Retrieval.
16. Adaptive RAG: Self-RAG, CRAG, FLARE.
17. Agent memory: MemGPT, Mem0, ★ Zep/Graphiti temporal KG.
18. Compaction, note-taking, JIT retrieval, sub-agent isolation for long horizons.
19. Prompt compression: LLMLingua/-2; prompt caching for stable prefixes.
20. Harness patterns (Anthropic): chaining, routing, parallelization, orchestrator-workers, evaluator-optimizer.
21. MCP standardizes tools; CodeAct uses code as the action space.
22. SWE-agent: the Agent-Computer Interface can matter more than the model.
23. GraphRAG: Microsoft GraphRAG, LightRAG, HippoRAG, PathRAG.
24. Optimizable agent graphs: ★ GPTSwarm, ★ AFlow; orchestration via LangGraph.
25. Inference-time self-correction: Self-Refine, Reflexion, CRITIC.
26. Caveat: LLMs can't self-correct reasoning without external feedback.
27. Self-training: STaR, ReST-EM; self-reward: Self-Rewarding LMs, Constitutional AI.
28. Self-play from zero: ★ Absolute Zero, ★ R-Zero; ★ TTRL.
29. RSI: Gödel Machine (theory) → STOP, ADAS, ★ Darwin Gödel Machine, ★ AlphaEvolve, ★ SEAL.
30. Every self-improvement loop is bounded by a verifier and a sandbox; beware model collapse and reward hacking.
31. ★ Agent Skills: packaged procedural knowledge (SKILL.md) via progressive disclosure; the description is the routing key; self-authored skill libraries (Voyager) are weight-free self-improvement.

Test-Time & Training-Free Optimization

Merged from "Twenty-Five Things to Memorize" — Test_Time_and_Training_Free_Optimization_SOTA_Updated.md.

1. Two families: training-free inference vs test-time optimization/training.
2. Three axes: what's optimized, what signal, parallel vs sequential compute.
3. Decoding contrasts: CD, DoLa, CAD, CFG-LM all extrapolate along a log-prob difference.
4. MBR: decode the consensus (max expected utility), not the mode.
5. Steering: \(h\leftarrow h+\alpha v\) — the training-free dual of fine-tuning (RepE, CAA, ITI).
6. Task vector \(\tau=\theta_{\text{ft}}-\theta_{\text{pre}}\); merge by \(\theta_0+\sum_i\lambda_i\tau_i\) (Task Arithmetic).
7. TIES/DARE suppress merge interference (trim, elect-sign, drop-and-rescale).
8. Speculative decoding is lossless: accept w.p. \(\min(1,p/q)\), resample residual \((p-q)_+\).
9. Proof: \(\min(p,q)+(p-q)_+=p\) → emitted token \(\sim p\) exactly.
10. Expected tokens per block \(=\tfrac{1-\alpha^{\gamma+1}}{1-\alpha}\) (EAGLE/Medusa).
11. CFG: classifier-free guidance = implicit classifier \(\nabla\log p(y\mid x)=\nabla\log p(x\mid y)-\nabla\log p(x)\).
12. Self-consistency error \(\le\exp(-2N(p-\tfrac12)^2)\); floored by correlated error.
13. Best-of-\(N\) success \(=1-(1-p)^N\) with a perfect verifier; else verifier-bounded.
14. PRM > ORM > majority vote, gap widens with \(N\) (Let's Verify).
15. Math-Shepherd: step value = fraction of rollouts reaching the right answer.
16. Compute-optimal test-time scaling is difficulty-dependent: sequential for easy, parallel for hard (Snell).
17. Coverage power law \(-\log c(k)\approx a k^{-b}\) (Large Language Monkeys); selection saturates without a verifier.
18. ★ s1 budget forcing: append "Wait" to extend thinking.
19. ★ TTRL: RL at test time with a majority-vote pseudo-reward.
20. ★ Diffusion test-time scaling = search over noises with a verifier (Ma et al.).
21. TENT: minimize prediction entropy over BN affine params; risks class collapse.
22. Collapse fix = diversity/mutual-info (SHOT), reliable selection + SAM (SAR), anchoring (CoTTA).
23. TTT helps iff \(\langle\nabla\ell_m,\nabla\ell_s\rangle>0\) (SSL–main gradient alignment).
24. ★ TTT layers: hidden state is a model; \(W_t=W_{t-1}-\eta\nabla\ell(W_{t-1};x_t)\) per token (linear attention is the no-gradient special case).
25. ★ TPO: align at inference via textual gradients, zero weight updates — the training-free DPO.

Part III — Data & Evaluation

Data Collection & Curation

Merged from "Twenty-Five Things to Know" — Data_Collection_Curation_SOTA_Updated.md.

1. Data quality dominates architecture for frontier models.
2. Common Crawl is the foundation of nearly every LLM corpus.
3. C4 / The Pile / RefinedWeb / Dolma / DCLM / FineWeb lineage.
4. FineWeb-Edu: classifier-filtered > raw at small scale.
5. DCLM-baseline: small classifier filter beats hand rules.
6. Gopher heuristics for rule-based filtering.
7. MinHash + LSH for near-duplicate dedup.
8. SemDeDup for semantic dedup via embeddings.
9. Re-captioning is critical for image / video training (Sora, FLUX, DALL-E 3).
10. Phi-1: textbook synthetic data → small model wins.
11. R1-Distill: 800k long-CoT traces transfer reasoning.
12. Self-Instruct foundation for synthetic instructions.
13. Constitutional AI generates safety data via critique-revise.
14. Best-of-N rejection sampling (RAFT, RFT) cheap alternative to PPO.
15. HH-RLHF, UltraFeedback, Nectar are the canonical open preference datasets.
16. PRM800K, Math-Shepherd, OmegaPRM for process reward data.
17. NuminaMath: 860k math reasoning problems.
18. LAION-5B, COYO-700M, DataComp-1B, DFN-2B for image-text.
19. OBELICS for interleaved multimodal.
20. Open-X-Embodiment 1.4M trajectories for robot foundation.
21. Tesla's data engine: deploy → failure detect → label → retrain.
22. DoReMi for domain-mixture optimization.
23. dolma / datatrove / NeMo Curator for production curation.
24. WebDataset / Mosaic Streaming for training data loading.
25. Per-license, per-jurisdiction compliance is now mandatory.

Metrics & Evaluations

Merged from "Twenty-Five Things to Know" — Metrics_Evaluations_SOTA_Updated.md.

1. Eval is the moat at frontier; design carefully.
2. Goodhart's law: every metric gets gamed.
3. Wilson confidence interval for proportions.
4. Pearson vs Spearman: linear vs rank.
5. PSNR / SSIM / LPIPS hierarchy.
6. FID formula: \(\|\mu_r - \mu_g\|^2 + \mathrm{tr}(\Sigma_r + \Sigma_g - 2(\Sigma_r\Sigma_g)^{1/2})\).
7. COCO mAP averages over IoU 0.50–0.95.
8. BLEU / ROUGE / METEOR mostly weak for modern LLM open-ended.
9. LLM-as-judge biases: position, length, self-preference.
10. Order-swap + multi-judge + CoT-then-rate mitigations.
11. Chatbot Arena Elo is closest to ground truth for LLMs.
12. Bradley-Terry preference probability.
13. pass@k formula: \(1 - \prod(1 - p_i)\).
14. Best-of-N expected: \(\mathbb{E}[\max] \approx \mu + \sigma\sqrt{2 \ln N}\).
15. Reward overoptimization scaling: \(a\sqrt{\mathrm{KL}} - b\,\mathrm{KL}\).
16. KL anchor + RM ensemble for hacking defense.
17. LiveCodeBench / current AIME / HLE for contamination-resistance.
18. N-gram overlap detection for contamination check.
19. HELM, lm-eval-harness, Open LLM Leaderboard v2 are open standards.
20. RULER for long-context (better than NIAH alone).
21. VBench-2 16+ axes for video gen.
22. VQAScore / GenEval / T2I-CompBench for compositional T2I.
23. Per-task breakdown + per-segment slicing always.
24. Calibration: ECE, reliability diagrams.
25. Closed-loop downstream metric > human > judge > automatic.

Part IV — Generative Models: Diffusion, Video, 3D & World

Diffusion Models

Merged from "Twenty-Five Equations to Memorize" — Diffusion_Models_SOTA_Updated.md.

1. Forward closed form \(\mathbf{x}_t = \sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1-\bar\alpha_t}\,\boldsymbol\epsilon\).
2. Reverse posterior mean \(\tilde\mu_t\).
3. Simplified DDPM loss \(\|\boldsymbol\epsilon - \boldsymbol\epsilon_\theta\|^2\).
4. Score-noise equivalence \(\boldsymbol\epsilon_\theta = -\sqrt{1-\bar\alpha_t}\,s_\theta\).
5. Variance-preserving SDE.
6. Probability flow ODE.
7. DDIM update with \(\eta\to 0\).
8. Cosine schedule formula.
9. Karras EDM preconditioning (\(c_{\text{skip}}, c_{\text{out}}, c_{\text{in}}, c_{\text{noise}}\)).
10. Min-SNR weight.
11. CFG: \(\tilde{\boldsymbol\epsilon} = \boldsymbol\epsilon^\varnothing + w(\boldsymbol\epsilon^c - \boldsymbol\epsilon^\varnothing)\).
12. Negative-prompt CFG.
13. adaLN-Zero block (\(\alpha(c)\) on residual init zero).
14. Latent diffusion loss in latent space.
15. MM-DiT joint attention over \([Q_{\text{txt}}; Q_{\text{img}}]\).
16. Conditional FM affine path: \(u_t = \mathbf{x}_1 - \mathbf{x}_0\).
17. Rectified flow re-pairing concept.
18. Consistency model loss.
19. DMD score-distillation gradient.
20. ControlNet zero-conv connection rationale.
21. DreamBooth prior-preservation loss.
22. LoRA decomposition \(W + BA\).
23. SDS gradient \(w(t)(\boldsymbol\epsilon_\phi - \boldsymbol\epsilon)\,\partial\mathbf{x}/\partial\theta\).
24. Diffusion-DPO loss with reference policy.
25. FID formula.

Diffusion — Derivations

Merged from "One-Page Cheat Sheet" — Diffusion_Derivations_SOTA_Updated.md.

The Rosetta Stone. With \(\mathbf{x}_t=m_t\mathbf{x}_0+s_t\boldsymbol{\epsilon}\) and \(\boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\):

\[\underbrace{\nabla_{\mathbf{x}_t}\log q_t(\mathbf{x}_t)}_{\text{score}} \approx -\frac{\boldsymbol{\epsilon}_\theta(\mathbf{x}_t,t)}{s_t} \iff \hat{\mathbf{x}}_0 = \frac{\mathbf{x}_t - s_t\boldsymbol{\epsilon}_\theta}{m_t} \iff v_\theta(\mathbf{x}_t,t) = \dot{m}_t\hat{\mathbf{x}}_0 + \dot{s}_t\boldsymbol{\epsilon}_\theta.\]

Loss families (all are \(\ell_2\) regressions).

Samplers.

Equivalences worth remembering.

• DDPM training = denoising score matching at noise scale \(s_t=\sqrt{1-\bar{\alpha}_t}\).
• DDPM forward chain = Euler–Maruyama discretization of the VP-SDE.
• DDIM (\(\sigma=0\)) = first-order solver of the probability-flow ODE.
• Flow matching with \(\mu_t=\alpha_t\mathbf{x}_1\), \(\sigma_t\) linear = training a diffusion model up to weighting.
• Rectified flow = flow matching with the linear interpolant \(\mathbf{x}_t=(1-t)\mathbf{x}_0+t\mathbf{x}_1\).
• Classifier-free guidance = tempering \(\tilde{p}\propto p(\mathbf{x})\,p(y\mid\mathbf{x})^w\) implemented by mixing conditional/unconditional scores.

Video Generation

Merged from "Twenty-Five Things to Know" — Video_Generation_SOTA_Updated.md.

1. Causal 3D VAE: \(T/4 \times H/8 \times W/8\) standard compression.
2. Spatiotemporal MM-DiT: text + video joint self-attention.
3. Joint image + video training: \(T = 1\) for images.
4. 3D-RoPE on \((t, h, w)\) video tokens.
5. Diffusion vs autoregressive for video: parallel quality vs indefinite horizon.
6. Sora (Feb 2024): spacetime patches, "world simulator."
7. Sora 2 (Sep 2025): + audio, longer clips, physics improvement.
8. Veo 3 (May 2025): native audio + improved fidelity.
9. Hunyuan Video (Tencent, 13B, open frontier).
10. Wan 2.1 / 2.2 (Alibaba, 14B+, open).
11. Mochi 1 (Genmo, 10B AsymmDiT, open Apache).
12. LTX-Video (Lightricks, real-time on A100).
13. Step-Video (StepFun, 30B, largest open).
14. AnimateAnyone / MimicMotion: pose-driven character animation.
15. EMO / Live Portrait / Audio2Photoreal: talking-head animation.
16. ReferenceNet for identity preservation.
17. Re-captioning of training data is critical.
18. VBench / VBench-2: 16+ axis evaluation standard.
19. FVD: Fréchet Video Distance via I3D / InternVideo.
20. ControlNet for video (CogVideoX-Control, Wan-Control).
21. LoRA per-style / per-character standard.
22. Step distillation: 4–8 NFEs (LCM-Video, Hyper-Video, DMD2-Video).
23. Block-cache / TGATE / PAB for inference speed.
24. World-model framing: video gen + action conditioning.
25. Open frontier (Wan, Hunyuan) closing gap with closed.

3D & Multi-View Generation

Merged from "Twenty-Five Things to Know" — 3D_MultiView_Generation_SOTA_Updated.md.

1. Four paradigms: SDS / multi-view diffusion / feed-forward / native 3D.
2. DreamFusion (2022): SDS gradient \(w(t)(\epsilon_\phi - \epsilon)\,\partial x/\partial\theta\).
3. Janus problem: front-face replication; major SDS issue.
4. ProlificDreamer (VSD): replaces noise target with learned distribution.
5. Magic3D: two-stage NeRF \(\to\) mesh.
6. MVDream / ImageDream: multi-view consistent diffusion.
7. Zero123 / Zero123++: novel-view synthesis from single image.
8. SV3D: video-diffusion approach to multi-view.
9. LRM (Adobe): single image \(\to\) triplane via Transformer.
10. InstantMesh / MeshLRM / GS-LRM / Long-LRM / TripoSR / CRM / SF3D / SPAR3D: LRM family.
11. Native 3D diffusion (2025 wave): Trellis / Hunyuan3D-2 / CLAY / Direct3D.
12. Trellis: structured sparse-voxel latent + flow matching.
13. Hunyuan3D-2: native 3D + texture; open frontier.
14. Mesh autoregressive: MeshGPT / MeshXL / MeshAnything / EdgeRunner / BPT.
15. DUSt3R: pointmap from 2 images (camera 1 frame).
16. MASt3R / MASt3R-SfM: + matching head + global SfM.
17. VGGT (2025): dominant feed-forward 3D.
18. π3: pose-free scaled VGGT successor.
19. Plücker coordinates per pixel for camera conditioning.
20. Cross-view attention + reference attention for multi-view consistency.
21. DreamGaussian: multi-view + 3DGS optimization in ~1 min.
22. LGM / Splatter Image: feed-forward Gaussians.
23. Texture: SyncMVD / Paint3D / Hunyuan3D-2 joint.
24. Objaverse / Objaverse-XL: 800k–10M+ 3D dataset.
25. T3Bench / GPTEval3D / ULIP for 3D-text alignment evaluation.

World Models

Merged from "Twenty-Five Things to Know" — World_Models_SOTA_Updated.md.

1. World model = generative model of environment dynamics conditioned on actions.
2. Two threads: latent dynamics for control vs foundation video world models.
3. Dyna-Q: original mix of model-based + model-free.
4. PILCO: GP dynamics; sample-efficient on low-dim.
5. PETS: NN ensembles + CEM planning.
6. Dreamer V3: RSSM + symlog + two-hot; same hyperparameters across 150+ tasks.
7. RSSM: deterministic GRU + stochastic latent.
8. Symlog: \(\mathrm{sgn}(x)\log(|x| + 1)\) for cross-task robustness.
9. TD-MPC2: latent dynamics + MPC; strong continuous control.
10. IRIS: tokenizer + Transformer dynamics; Atari at low data.
11. DIAMOND: diffusion world model (Atari, CS:GO).
12. Sora: spacetime patches + DiT; OpenAI's "world simulator".
13. Veo 3: + audio; closed.
14. Cosmos: NVIDIA platform (Tokenizer / Predict / Transfer / Reason / Curator).
15. GAIA-1/2: Wayve's AV world model.
16. Genie 1/2: latent action discovery; playable worlds.
17. Oasis: open real-time Minecraft world model.
18. GameNGen: DOOM via diffusion at 20 fps.
19. Causal 3D VAE for video tokenization.
20. LFQ (MAGVIT-v2) for discrete video tokens.
21. Latent action models discover actions from unlabeled video.
22. Camera control via Plücker coordinates / extrinsics conditioning.
23. Long-horizon: chunked AR + anchor frames.
24. Physical commonsense is the open challenge; Cosmos Reason as evaluator.
25. Closed-loop downstream success is the only bulletproof eval.

Part V — Neural Rendering & 3D Reconstruction

NeRF

Merged from "25 things every principal must know about NeRF" — NeRF_SOTA_Updated.md.

1. NeRF is a continuous 5D function \(f_\theta(\mathbf{x}, \mathbf{d}) \to (\mathbf{c}, \sigma)\) with classical volume rendering.
2. Discretized rendering: \(C = \sum_i T_i (1 - e^{-\sigma_i \delta_i})\, \mathbf{c}_i\).
3. Vanilla NeRF uses sinusoidal positional encoding (\(L = 10\) for \(\mathbf{x}\), \(L = 4\) for \(\mathbf{d}\)).
4. Hierarchical sampling (coarse + fine MLPs) was the original importance-sampling trick.
5. Mip-NeRF replaces points with cones; Integrated PE anti-aliases naturally.
6. Mip-NeRF 360 contraction maps unbounded scenes onto a finite ball; proposal MLP speeds up sampling.
7. Zip-NeRF combines Mip-NeRF 360 with hash grids via multi-sample IPE; current quality SOTA.
8. Instant-NGP uses a multiresolution hash grid and a tiny MLP; train in seconds.
9. Plenoxels / DVGO show MLPs aren't strictly required for NeRF-quality NVS.
10. TensoRF, k-Planes, tri-planes are tensor-factorized alternatives.
11. NeuS / VolSDF / NeuralAngelo replace density with SDF for sharp surfaces; use eikonal loss.
12. Ref-NeRF re-parameterizes view direction for accurate specularities.
13. NeRO splits diffuse + specular with reflection direction; great for glossy.
14. KiloNeRF / SNeRG / PlenOctrees / MERF bake fast renderers.
15. BakedSDF converts to mesh + neural shader for mobile.
16. D-NeRF / Nerfies / HyperNeRF / NSFF / K-Planes handle dynamic scenes.
17. NeRF-W and Block-NeRF use per-image appearance embeddings for in-the-wild scenes.
18. Block-NeRF / Mega-NeRF / BungeeNeRF scale to city / aerial.
19. DreamFusion uses Score Distillation Sampling (SDS) on a frozen 2D diffusion model.
20. EG3D uses tri-planes + StyleGAN super-resolver; foundational 3D-aware GAN.
21. LRM / Zero-1-to-3 / Wonder3D / Trellis are feed-forward 3D generators.
22. LERF distills CLIP features into a 3D field for open-vocabulary 3D queries.
23. Instruct-NeRF2NeRF iteratively re-renders training views with InstructPix2Pix for editing.
24. Production NeRF stack: Nerfstudio (Nerfacto + Zip-NeRF + NeuS / NeuralAngelo + LERF) + Instant-NGP for speed.
25. In 2026, GS replaced NeRF for real-time NVS; NeRF still owns relighting, mesh extraction, generative 3D, and hybrid pipelines.

Gaussian Splatting

Merged from "25 things every principal must know about Gaussian Splatting" — Gaussian_Splatting_SOTA_Updated.md.

1. 3DGS replaces NeRF's MLP+ray-march with explicit anisotropic Gaussians + tile rasterization.
2. Each primitive stores \((\boldsymbol{\mu}, q, s, \alpha, \mathrm{SH})\); covariance is reconstructed \(\Sigma = R\,\mathrm{diag}(s)^2\,R^\top\).
3. EWA splatting linearizes the perspective projection at \(\boldsymbol{\mu}_c\) to get \(\Sigma_{2D}\).
4. Volume rendering equation is approximated as alpha compositing of 2D Gaussian footprints.
5. Adaptive density control (clone, split, prune, opacity reset) is the key training trick.
6. Vanilla 3DGS uses 1–3 M Gaussians for room-scale; 200–800 MB on disk.
7. Mip-Splatting fixes scale-aliasing via 3D smoothing + 2D Mip filter.
8. 2DGS replaces 3D ellipsoids with oriented disks; far better surfaces and meshes.
9. Scaffold-GS / Octree-GS use voxel anchors + tiny MLP → compression and far-view extrapolation.
10. Hierarchical 3DGS (Inria) gives LoD from city-scale to centimeter close-ups.
11. Densification gradient threshold (\(\sim\!2\text{e-}4\)) and opacity reset every 3000 iters are load-bearing.
12. Popping artifacts come from per-tile center-depth sort; StopThePop, GOF, exact pixel sort fix it.
13. Compression: prune importance + SH distill + VQ + zstd → smaller, <0.5 dB drop.
14. LightGaussian, SOG, EAGLES, RDO-Gaussian are the standard compression toolbox.
15. 4DGS approaches: per-frame; canonical+deformation field; native 4D primitives (Spacetime Gaussians).
16. Gaussian SLAM: SplaTAM, MonoGS, LoopSplat, Gaussian-SLAM all replace NeRF-SLAM.
17. Generative 3D: feed-forward (LRM, Splatter Image, Trellis, Hunyuan3D-2) and SDS (DreamGaussian) both target GS now.
18. Relightable GS: per-Gaussian BRDF + visibility (Relightable 3DG, GS-IR, GShader).
19. PhysGaussian unifies rendering and simulation primitives via MPM.
20. Avatars: FLAME / SMPL-X driven; GaussianAvatars, AnimatableGaussians, Codec Avatars 3D.
21. AV simulation: StreetGaussians, OmniRe, NeuRAD — per-actor + per-scene splats.
22. Production format: .spz / .splat / quantized .ply with bbox + LoD header.
23. gsplat (Nerfstudio) is the reference implementation; Inria's diff-gaussian-rasterization is the original.
24. Benchmark numbers without compression / FPS / memory are misleading; always report all four.
25. The frontier is foundation 3DGS + 4D world models + real-time on-device rendering.

Neural Rendering

Merged from "Twenty-Five Things to Know" — Neural_Rendering_SOTA_Updated.md.

1. Volume rendering equation (Mildenhall NeRF): \(C(\mathbf{r}) = \int_{t_n}^{t_f} T(t)\,\sigma(\mathbf{r}(t))\,c(\mathbf{r}(t), \mathbf{d})\,\mathrm{d}t\).
2. Discretization: \(\hat{C} = \sum T_i(1 - e^{-\sigma_i \delta_i})\,c_i\).
3. Positional encoding: \(\gamma(p) = [\sin(2^k \pi p),\ \cos(2^k \pi p)]\).
4. Hierarchical sampling (coarse + fine).
5. Instant-NGP hash grid: trains in seconds.
6. Mip-NeRF integrated PE for anti-aliasing.
7. Mip-NeRF 360 scene contraction.
8. 3DGS representation: \((\mu, R, S, \alpha, \text{SH})\) per Gaussian.
9. 3DGS projection: \(\Sigma' = J W \Sigma W^\top J^\top\).
10. 3DGS rasterization: alpha-composite front-to-back.
11. 3DGS loss: \((1 - \lambda)\,L_1 + \lambda\,\text{D-[SSIM](https://ece.uwaterloo.ca/~z70wang/publications/ssim.html)}\).
12. Adaptive density control: clone / split / prune / opacity reset.
13. 3DGS variants: Mip-Splatting / 2D-GS / Scaffold-GS / 4D-GS / SuGaR / GS-IR / LightGaussian.
14. DUSt3R pointmap formulation.
15. VGGT: dominant feed-forward 3D in 2025.
16. MoGe-2: metric monocular geometry.
17. DreamFusion SDS gradient.
18. ProlificDreamer VSD reduces Janus.
19. Trellis / Hunyuan3D-2 native 3D diffusion (2025 wave).
20. MeshGPT autoregressive mesh generation.
21. EMO / Live Portrait audio-driven portrait.
22. GS-SLAM family: MonoGS, GS-SLAM, SplaTAM.
23. Cosmos: NVIDIA neural simulator.
24. Nerfstudio + gsplat are the standard research frameworks.
25. Eval: PSNR / SSIM / LPIPS for image; Chamfer for geometry.

Structure from Motion

Merged from "Twenty-Five Things to Know" — Structure_from_Motion_SOTA_Updated.md.

1. Pinhole camera: \(\lambda x = K[R|t]X\).
2. Brown-Conrady distortion (radial + tangential).
3. Zhang's calibration: planar checkerboard from multiple views.
4. Epipolar constraint: \(x_2^{\top} E x_1 = 0\).
5. Essential matrix: \(E = [t]_\times R\), 5 DoF.
6. Fundamental matrix: \(F = K_2^{-\top} E K_1^{-1}\), 7 DoF, rank-2.
7. Eight-point algorithm with Hartley normalization.
8. Five-point algorithm (Nistér): minimal calibrated case.
9. DLT triangulation via SVD.
10. PnP: P3P (4 candidates) or EPnP \(O(n)\).
11. RANSAC iterations: \(N = \log(1 - p)/\log(1 - w^s)\).
12. MAGSAC++: marginalize over noise scale.
13. Reprojection error + Huber kernel + LM = bundle adjustment.
14. Schur complement reduces BA to camera-only system.
15. Ceres / g2o / GTSAM standard solvers.
16. COLMAP: incremental SfM + PatchMatch MVS.
17. HLoc: SuperPoint + SuperGlue / LightGlue + COLMAP.
18. LightGlue is the production-default learned matcher (2024+).
19. DUSt3R: predict pixel-aligned 3D pointmaps from 2 images.
20. MASt3R-SfM: replaces COLMAP at \(\sim 100\times\) speedup.
21. VGGT (Meta 2025): feed-forward N-image \(\to\) depth + cameras.
22. Visual SLAM = online SfM with real-time constraints.
23. ORB-SLAM3 / VINS-Fusion / OKVIS-2 are mature SLAM stacks.
24. Loop closure: place recognition + geom verify + pose graph opt.
25. NetVLAD / MixVPR / AnyLoc are place recognition standards.

Delighting & Relighting

Merged from "Twenty-Five Things to Know" — Delighting_Relighting_SOTA_Updated.md.

1. Rendering equation (Kajiya 1986).
2. BRDF properties: non-negative, reciprocity, energy conservation.
3. Microfacet form: \(f_r = FGD / (4\cos\theta_i\cos\theta_o)\).
4. GGX is the industry-standard \(D\).
5. Schlick Fresnel: \(F = F_0 + (1 - F_0)(1 - \cos\theta)^5\).
6. Disney Principled BRDF is the artist standard.
7. PBR maps: albedo, normal, roughness, metallic, +AO.
8. Intrinsic image: \(I = R \cdot S\) (reflectance × shading).
9. Retinex: sharp log \(I\) edges = reflectance; smooth = shading.
10. IIW + CGIntrinsics are the standard datasets.
11. Photometric stereo (Woodham): \(\mathbf{I} = L \cdot (\rho\, \mathbf{n})\).
12. Cross-polarized capture removes specular for clean albedo.
13. Light stage = hundreds of LEDs + reflectance field.
14. Reflectance field: relight is matrix-vector multiply in lighting basis.
15. Spherical Harmonics for low-frequency env lighting (9 coefficients standard).
16. Spherical Gaussians for higher-frequency / specular.
17. Pre-filtered cubemap + split-sum for real-time IBL (UE4-style).
18. Sun et al. (2019): single-image deep portrait relighting baseline.
19. Total Relighting: foreground matting + relighting end-to-end.
20. SwitchLight / IC-Light: 2024 diffusion-based relighting wave.
21. Relightable 3D Gaussians / GS-IR: 3DGS + relighting.
22. ARKit / ARCore env probes for mobile AR lighting.
23. LED volumes (Stagecraft): real-time IBL on virtual production.
24. Photogrammetry \(\to\) delight \(\to\) PBR maps (Quixel pattern).
25. Diffusion gives plausible; physics gives correct (hybrid emerging).

Photorealistic Avatars

Merged from "Twenty-Five Things to Know" — Photorealistic_Avatars_SOTA_Updated.md.

1. Five components: geometry / appearance / rig / driving / renderer.
2. FLAME (face): 300+100+pose params.
3. SMPL-X (body): 119 dims, 6890 vertices.
4. Light stages capture full reflectance field.
5. Codec Avatars: VAE + view-conditioned neural texture.
6. Apple Persona: few-second enrollment + on-device inference.
7. GaussianAvatars: 3DGS bound to FLAME mesh.
8. FlashAvatar: 300+ fps 3DGS face.
9. EMO: audio + reference image \(\to\) photoreal portrait video.
10. Audio2Photoreal: audio + Codec Avatars-style full body.
11. Live Portrait: image + driving video \(\to\) animated.
12. AnimateAnyone: pose video drives static character via ReferenceNet.
13. ReferenceNet: trainable U-Net copy + KV concatenation.
14. InstantID / PhotoMaker / PuLID: face encoder + structure encoder for ID.
15. ArcFace identity embedding for ID preservation loss.
16. FACS (Action Units): facial expression decomposition.
17. Marschner BCSDF for strand-based hair.
18. LSE-D / LSE-C / SyncNet for lip-sync evaluation.
19. NeRSemble dataset: multi-view facial.
20. 4D-DRESS: clothed body 4D.
21. Body Gaussians (GauHuman, HumanGaussian) for full body.
22. LightGaussian for mobile / web deployment.
23. Apple Vision Pro / Codec Avatars 3.0 are the production-grade telepresence apps.
24. Trellis / Hunyuan3D-2 for stylized avatar generation.
25. Watermarking (SynthID, C2PA) for synthetic-content provenance.

Part VI — Core Vision, Robotics & Autonomy

Computer Vision — Principal Deep Dive

Merged from "Appendix: Ten Derivations You Must Own Cold" — CV_Principal_DeepDive_SOTA_Updated.md.

1. ELBO from Jensen \(\to\) VAE objective.
2. ELBO \(\to\) DDPM simplified loss (the variance-cancelling step).
3. DDPM noise prediction \(\Leftrightarrow\) score matching: \(\epsilon_\theta = -\sigma_t s_\theta\).
4. DPO from KL-constrained RLHF: closed-form policy \(\to\) Bradley–Terry on log-ratios.
5. Policy gradient theorem from \(J(\theta) = \sum_s d^\pi(s)\sum_a\pi(a\mid s)Q^\pi(s, a)\).
6. PPO clipped surrogate \(\to\) trust-region intuition.
7. GRPO advantage and why removing the value head reduces variance / cost.
8. Flow matching from continuous normalizing flows: continuity equation and the conditional FM objective.
9. Eckart–Young in two lines (SVD + orthogonal decomposition).
10. NeRF discretization from the volume-rendering equation (alpha-compositing).

If you can do all ten without notes, you can survive any research-deep round at the principal level.

Computer Vision — Principal Math

Merged from "Numbers Worth Memorizing" — CV_Principal_Math_SOTA_Updated.md.

• ImageNet-1k: 1.28M train, 50k val, 1000 classes.
• COCO: 118k train, 5k val, 80 classes.
• LAION-5B: 5.85B image-text pairs (large public web scrape).
• LVIS: 1203 long-tail classes. Open Images V7: 9M images, 600 boxes classes.
• nuScenes: 1000 scenes, 6 cameras + LiDAR + radar; Waymo Open: \(\sim 2030\) scenes.
• Open-X-Embodiment: \(\sim 1.4\) M robot trajectories across 22 embodiments.
• Humanoid robots common DoF: H1 \(\sim 19\), G1 \(\sim 23\), Atlas \(\sim 28\).
• One H100 SXM: 989 TFLOPs BF16, 1979 TFLOPs FP8, 80 GB HBM3 (3 TB/s); B200: \(\sim 4500\) TFLOPs FP8, 192 GB HBM3e.
• Stable Diffusion 1.5: 860M U-Net + 123M text encoder + 84M VAE; SDXL: 2.6B U-Net.
• LLaVA-1.5-13B: ViT-L/14-336 (303M) + Vicuna-13B + 2-layer MLP projector.
• Diffusion default schedule: 1000 train steps; cosine; train EMA decay 0.9999.
• Adam defaults: \(\beta_1 = 0.9\), \(\beta_2 = 0.999\), \(\epsilon = 10^{-8}\); AdamW for LMs uses \(\beta_2 = 0.95\).

Vision-Language-Action (VLA) Models

Merged from "Twenty-Five Things to Know" — VLA_Models_SOTA_Updated.md.

1. VLA = vision + language + action in single model.
2. Recipe: pretrained VLM + action head + robot data.
3. RT-1 (2022): discrete actions, 35M params, baseline.
4. RT-2 (2023): VLM + action vocab; co-train with web data.
5. RT-X / Open-X-Embodiment: 1.4M trajectories, 22 embodiments.
6. OpenVLA: open RT-2 reference; Llama 2 + DINOv2 + SigLIP.
7. Octo: smaller diffusion-policy open VLA.
8. RDT-1B / RDT-2B: 1B/2B-param diffusion VLA.
9. π0: PaliGemma + flow-matching action head.
10. \(\pi_0\) FM loss: \(\left\| v_\theta - (a_1 - a_0) \right\|^2\).
11. \(\pi_{0.5}\): hierarchical S2/S1 for long-horizon.
12. GR00T: NVIDIA humanoid platform; Cosmos sim integration.
13. Helix: Figure's S2/S1 split for on-board inference.
14. PaLM-E: predates RT-2; embodied multimodal LM.
15. Discrete action tokens: 256 bins per action dim.
16. Diffusion Policy: multi-modal action chunks.
17. Flow matching: π0's choice; few-step inference.
18. ACT: action chunking transformer + CVAE.
19. Temporal ensembling: average overlapping chunks.
20. Spatial VLMs: SpatialVLM / SpatialBot / RoboPoint.
21. Sim-to-real: DR + asymmetric A-C + privileged-to-vision distill.
22. Cross-embodiment via OXE; positive transfer.
23. LoRA per task / per embodiment, shared base.
24. RL fine-tune via Q-chunking / residual / Diffusion DPO / GRPO.
25. Edge: Jetson Thor + distillation + INT8.

Autonomous Driving

Merged from "Twenty-Five Things to Know" — Autonomous_Driving_SOTA_Updated.md.

1. Levels of autonomy: L0 (none) to L5 (full).
2. Sensor stack: camera + LiDAR + radar + IMU + GNSS.
3. BEV (Bird's-Eye-View): unified top-down representation.
4. LSS / BEVFormer / PETR for camera-only 3D detection.
5. PointPillars / CenterPoint / SAFDNet for LiDAR.
6. BEVFusion / TransFusion for sensor fusion.
7. Occupancy networks (FB-OCC / SparseOcc) for general obstacles.
8. MapTR / MapTRv2 for online HD mapping.
9. Tracking-by-detection (ByteTrack) + Transformer trackers (TransTrack).
10. Motion prediction: Wayformer / MTR++ / QCNet.
11. Min-of-\(K\) loss for multi-modal trajectory prediction.
12. UniAD (CVPR 2023) pioneered end-to-end joint training.
13. VAD / Hydra-MDP / SparseDrive / DiffusionDrive: end-to-end variants.
14. Tesla FSD v12+ is end-to-end neural, mostly imitation-learned.
15. Wayve LINGO-2 / DriveVLM / Senna / EMMA: VLM-based driving.
16. GAIA-1/2 / Cosmos / DriveDreamer: AV world models.
17. StreetGaussians / EmerNeRF / OmniRe: neural AV simulators.
18. RSS (Mobileye): formal safety framework.
19. Tesla data-engine pattern: rare-event mining + auto-label.
20. NVIDIA Thor 1000+ TOPS / Tesla HW4 / Mobileye EyeQ for on-vehicle compute.
21. Waymo: sensor-rich modular; Tesla: vision-only end-to-end.
22. Tesla mapless vs Waymo HD-map: opposing bets.
23. Closed-loop sim (GAIA + Cosmos + Waymax): replaces some real testing.
24. Foundation-model backbones (DINOv3 / SigLIP) increasingly in AV.
25. Long-tail edge cases dominate L4 deployment effort.