Foundation Models — Vision, Graphics, Multimodality

Updated July 2026 with 2025–2026 SOTA additions — new entries marked ★. Algorithm names link to their papers (arXiv / project page).

April 2026 · Version 1.0

Contents

1. The Map: How Foundation Models Cluster
2. Vision Encoders: Self-Supervised
3. Vision Encoders: Promptable / Universal
4. Depth / Geometry Foundation Models
5. Vision-Language Foundation Models (VLMs / MLLMs)
6. Image Generation: Diffusion & Flow Matching
7. Video Generation
8. Audio and Music Foundation Models
9. NeRF, 3DGS, and 3D Scene Foundation
10. Feed-Forward 3D (Single / Few-View Reconstruction)
11. Avatars and Humans
12. World Models
13. Vision-Language-Action (VLA) Models
14. Embedding / Retrieval Foundation Models
15. Specialized / Domain-Specific Foundation Models
16. Action / Motion Foundation Models
17. Specialized Vision Models Worth Knowing
18. Cross-Cutting Architectural Patterns
19. The 2026 Production Stack

• Appendix A: Names to Recognize on Sight
• Appendix B: Family Tree (One-Page Mental Map)
• Appendix C: Comparison Tables

1. The Map: How Foundation Models Cluster

1.1 Six pillars

1. Vision encoders: input → features (CLIP, DINO, MAE, SAM).
2. Vision-language models (VLMs): image + text → text (LLaVA, InternVL, Qwen-VL, Chameleon, GPT-4o).
3. Generative models: text → image/video/audio (SD, FLUX, Sora, Stable Audio).
4. 3D foundation: image(s) → 3D (DUSt3R, VGGT, 3DGS scenes, LRM, Trellis).
5. World models: action + state → next state (GAIA, Cosmos, Genie).
6. Vision-Language-Action (VLA): image + instruction → action (π0, OpenVLA, GR00T).

1.2 The 2026 trend lines

• Native multimodal replacing adapter-style VLMs (Chameleon, GPT-4o, Gemini, Show-o).
• Flow matching replacing diffusion for new generative releases (SD3, FLUX, MovieGen, Cosmos).
• Feed-forward 3D (VGGT, MASt3R) replacing classical SfM.
• 3D Gaussian Splatting replacing NeRF as default scene rep.
• Reasoning VLMs (R1-style multimodal) emerging as a category.
• Robotic foundation models (π0, GR00T, Helix) becoming the dominant approach.

1.3 Open vs closed gap (2026)

The open frontier (Llama 4, Qwen 3, DeepSeek-V3/R1, FLUX, Hunyuan Video, Wan 2.1, π0) closed dramatically in 2024–25. For most tasks except bleeding-edge frontier reasoning and proprietary video gen, open models match or exceed closed.

2. Vision Encoders: Self-Supervised

2.1 The CLIP family (dual-encoder contrastive)

2.1.1 CLIP (OpenAI 2021)

Image and text encoders trained with symmetric InfoNCE on \(\sim\) 400M pairs. ViT-B/32, B/16, L/14, RN50/x4/x16/x64.

Established zero-shot transfer.

2.1.2 ALIGN, OpenCLIP, MetaCLIP, DFN, EVA-CLIP

• ALIGN (Google): noisier 1.8B-pair web data; quality ≈ CLIP.
• OpenCLIP: open re-implementation; many checkpoints; community standard.
• MetaCLIP / MetaCLIP 2 (Meta): metadata-balanced data curation.
• DFN (Apple): tiny filter network curates training data; matches OpenCLIP at 5× less data.
• EVA-CLIP: large-scale CLIP with EVA-style mask reconstruction warm-start.

2.1.3 SigLIP / SigLIP 2

Per-pair sigmoid loss (vs CLIP's softmax over batch).

Sample-efficient, scales cleanly to small or huge batch.

SigLIP 2 adds captioning + self-distillation auxiliaries for stronger dense features.

2.2 The DINO family (self-distillation)

• DINO (2021): student-teacher self-distillation with sharpening + centering.
• DINOv2 (2023): + iBOT mask prediction + KoLeo regularizer + LayerScale + LVD-142M curated data; strong dense features.
• DINOv3 (2025): scale to 7B params with gram-matching loss to preserve dense quality at scale.

2.3 The MAE / BEiT family (masked image modeling)

• MAE (2021): mask 75% of patches; reconstruct in pixel space.
• BEiT, BEiT v2: predict discrete VQ tokens.
• SimMIM: simpler raw-pixel MIM.
• MaskFeat: predict HOG features.
• EVA, EVA-02: predict frozen CLIP features (semantic targets).
• FCMAE (ConvNeXt V2): MIM with sparse convs for ConvNeXt.
• BEiT-3: unified VL encoder with modality experts; SOTA on many VL benchmarks 2022.

2.4 JEPA family (latent prediction)

• I-JEPA, V-JEPA: predict embeddings of target patches via EMA target encoder.
• V-JEPA 2: scale to billion-frame video pretraining.

2.5 Comparison table

★ 2026 SOTA update — 2025 vision encoder frontier

• Perception Encoder: Meta's SOTA image/video encoder; best embeddings live in intermediate layers, extracted via language + spatial alignment; PE-core beats SigLIP2/InternVideo2.
• AIMv2: Apple's multimodal autoregressive pretraining (regress patches then decode text); dense per-patch/token supervision; scales to 3B, beats CLIP/SigLIP on MLLM tasks.
• Web-SSL: language-free DINO/ViT SSL scaled to 7B on 2B MetaCLIP images; matches/exceeds CLIP on VQA/OCR/charts without any captions.

3. Vision Encoders: Promptable / Universal

3.1 Segment Anything family

• SAM (Meta 2023): promptable segmentation (point/box/mask/text) trained on SA-1B (1.1B masks). Ambiguity-aware multi-mask outputs.
• SAM 2 (2024): extends to video with memory bank for streaming; handles point/box prompts on any frame.
• HQ-SAM, Semantic-SAM, EfficientSAM, MobileSAM, FastSAM: quality / efficiency variants.
• SAM 3 (rumored 2026): more language-aware / compositional.

3.2 Open-vocabulary detection

• GLIP: phrase grounding + detection.
• Grounding DINO 1.5/1.6/Pro (IDEA): cross-modal feature enhancer + text-conditioned query selection. Strong open-set detection.
• DINO-X (IDEA 2024): unified all-task perception model (detection + grounding + segmentation + counting).
• T-Rex2 (IDEA): visual + text prompting; visual reference detection.
• OWL-ViT, OWLv2 (Google): CLIP-based open-vocab detection.
• YOLO-World, YOLO-World v2: open-vocab YOLO; real-time.

3.3 Universal perception

• Florence, Florence-2, Florence-3 (Microsoft): foundation models for vision; one model handles caption / detection / segmentation / OCR via task tokens.
• APE (Aligning and Prompting Everything): promptable everything.
• OneFormer: unified semantic + instance + panoptic via single set of queries.
• Sa2VA: SAM 2 + LLaVA fusion for grounded video chat.

★ 2026 SOTA update — SAM 3: promptable concept segmentation

• SAM 3: Meta 2025 successor to SAM 2; Promptable Concept Segmentation from a noun phrase or image exemplar returns masks + IDs for ALL matching instances (not one-per-prompt); shared detector + memory tracker with presence head; doubles prior PCS accuracy.

4. Depth / Geometry Foundation Models

4.1 Monocular depth

• MiDaS, DPT: classical baselines (2019–2022).
• ZoeDepth: relative + metric.
• Depth Anything v1/v2 (TikTok / HKU 2024): 100M-image scale; strong zero-shot.
• Marigold: diffusion-based monocular depth; high quality, slower.
• GeoWizard, Lotus: diffusion depth / normal jointly.
• StableNormal: surface normals.
• MoGe / MoGe-2 (HKUST): metric-scale monocular geometry estimator with affine ambiguity removed.

4.2 Multi-view / pose-free 3D from images

The big 2024–2025 wave: skip classical SfM/MVS and predict 3D directly via Transformer.

• DUSt3R (Naver 2024): 2-image → pixel-aligned pointmaps; camera + depth + matches all decoded from pointmaps.
• MASt3R / MASt3R-SfM: + dense matching head; scales to dozens of views via global SfM.
• Spann3R: incremental, processes one new view at a time.
• Splatt3R, NoPoSplat, Fast3R: fast variants.
• VGGT (Visual Geometry Grounded Transformer, Meta 2025): large transformer; depth + cameras + per-pixel 3D from N unposed images in one forward pass. Dominant 2025–26.

Key

The trend: classical SfM/MVS is being absorbed into feed-forward learned 3D. By 2027, COLMAP will be a fallback / refinement step, not a default.

★ 2026 SOTA update — Pose-free 3D: VGGT successors

• Depth Anything 3: single plain-DINO transformer, depth-ray target, adaptive cross-view attention; SOTA on new visual-geometry benchmark, +44% camera pose / +25% geometry over VGGT.
• Pi3: fully permutation-equivariant geometry net; no reference view; predicts affine-invariant poses + scale-invariant pointmaps; robust to input ordering, SOTA pose/depth/pointmap.
• MapAnything: universal feed-forward metric 3D; ingests images + optional intrinsics/poses/depth, regresses factored depth+rays+poses+scale; one model does SfM, MVS, mono-depth.

5. Vision-Language Foundation Models (VLMs / MLLMs)

5.1 Anatomy of a VLM

Three injection styles for vision into LLM:

• Linear / MLP projection (LLaVA family): every patch becomes an LLM token. Token-hungry; flexible.
• Q-Former (BLIP-2): \(n_q \sim 32\) learnable queries cross-attend over image features. Fixed-size.
• Cross-attention (Flamingo): insert gated cross-attn into LLM.

5.2 Open VLM frontier (2026)

Major open VLM lines

LLaVA family. LLaVA → LLaVA-1.5 → LLaVA-NeXT (AnyRes tiling) → LLaVA-OneVision (image / multi-image / video unified).

Qwen-VL family. Qwen-VL → Qwen2-VL (native dynamic resolution + 2D-RoPE + hour-long video) → Qwen2.5-VL (2025 frontier-open) → Qwen3-VL (reasoning).

InternVL family. InternVL → InternVL 2 → InternVL 2.5 → InternVL3 (78B+, frontier-open). InternViT-6B as huge vision encoder.

Other open. CogVLM, MiniCPM-V (mobile-friendly), Molmo + PixMo (AI2's open data), GLM-4V, Yi-VL, mPLUG-Owl, IDEFICS, KOSMOS-1/2/3, Pixtral (Mistral's VLM).

5.3 Closed frontier VLMs

• GPT-4V → GPT-4o → GPT-5 (OpenAI): GPT-4o introduced native multimodal, GPT-5 is the 2025 frontier.
• Gemini 1.5 / 2.0 / 2.5 (Google): native multimodal, very long context (multi-hour video in 2.5).
• Claude 3.5 Sonnet / 4 / Opus 4.6 (Anthropic): strong vision + agentic computer-use.
• Grok-Vision (xAI).

5.4 Native early-fusion (Chameleon family)

A single Transformer trained on interleaved text + image tokens (and audio / video).

• Chameleon (Meta 2024): VQ image tokens; pure next-token prediction.
• Show-o: text AR + image discrete diffusion in one backbone.
• Transfusion: text AR + image continuous diffusion in one backbone.
• Janus / Janus-Pro (DeepSeek): decoupled vision encoders for understanding vs generation, shared LLM.
• Emu3 (BAAI): pure next-token over text + image + video.

5.5 Reasoning VLMs (multimodal o1 / R1)

• LLaVA-CoT / LLaVA-o1: structured stages with MCTS at inference.
• Mulberry, Insight-V: collective MCTS with step-level critique.
• Vision-R1, VLM-R1, R1-V, MM-EUREKA, LMM-R1, Video-R1: GRPO + verifiable visual rewards.
• Qwen2.5-VL-Thinking, Kimi-VL-Thinking: long visual CoT.

5.6 Comparison table (frontier-open VLMs, 2026)

★ 2026 SOTA update — Open VLM frontier (late 2025)

• Qwen3-VL: Qwen3-VL dense (2B-32B) + MoE (30B-A3B / 235B-A22B); native 256K interleaved context, stronger text + long-video, thinking variants; frontier-open successor to Qwen2.5-VL.
• InternVL3.5: Cascade-RL (offline then online) + efficiency stack; +16% reasoning, 4x faster vs InternVL3; 241B-A28B MoE narrows gap to GPT-5; adds GUI + embodied agency.

★ 2026 SOTA update — Unified understanding + generation

• BAGEL: ByteDance 7B-active (14B) Mixture-of-Transformer-Experts, dual VAE+ViT encoders; understands, generates and edits images in one model; emergent free-form manipulation / world navigation.
• Emu3.5: BAAI next-token multimodal world learner over 10T+ tokens of video frames+transcripts; RL post-training; Discrete Diffusion Adaptation (DiDA) gives \(\sim\) 20x faster image decode; rivals Nano-Banana on edit.

6. Image Generation: Diffusion & Flow Matching

6.1 The 2021–2024 lineage

Era I: pixel / early latent diffusion

GLIDE (OpenAI 2021): pixel-space, classifier-free guidance born here.

DALL-E 2 / unCLIP (2022): CLIP latent prior + decoder.

Imagen (Google 2022): pixel cascade with T5-XXL text encoder.

Stable Diffusion 1.4 / 1.5 (2022): latent diffusion, open-sourced; catalyzed the ecosystem.

SD 2.0 / 2.1: OpenCLIP, v-prediction.

6.2 Era II: SDXL scale

• SDXL (2023): 2.6B U-Net + refiner, multi-aspect-ratio training.
• SDXL Turbo / Lightning: distilled few-step.
• DALL-E 3 (2023): GPT-4 prompt rewriter + improved diffusion.

6.3 Era III: DiT and flow matching

• PixArt-α/Σ/δ: open DiT + T5; δ adds LCM distillation.
• SD3 / 3.5 (2024): MM-DiT + rectified flow + CLIP+T5; 2B / 8B variants.
• FLUX.1 (Black Forest Labs 2024): MM-DiT + RF; dev / schnell / pro / Kontext editor. Dominant open frontier.
• Stable Cascade: 3-stage cascade for efficient training.
• Sana (NVIDIA): linear-attention DiT + 32× VAE for 4K speed.
• Lumina-T2I: LLaMA backbone reused for text + image.
• Hunyuan-DiT, CogView3/4: Asian-led DiT lines.

6.4 Era IV: closed frontier

• DALL-E / GPT Image (OpenAI), Imagen 2/3/4 (Google), Midjourney v5/v6/v7, Ideogram 1/2/3 (typography-strong), Adobe Firefly (licensed-data, enterprise), Recraft V3, Flux Pro Ultra.

6.5 Comparison table

★ 2026 SOTA update — Native / MMDiT T2I frontier (2025)

• Qwen-Image: 20B MMDiT; data pipeline + curriculum for complex text rendering (esp. Chinese) and precise editing; strong general T2I frontier-open.
• HunyuanImage 3.0: 80B-param MoE (13B active), unified autoregressive text+image (not DiT) with native chain-of-thought; largest open image generator, rivals closed models.

7. Video Generation

7.1 Architecture pattern

Causal 3D VAE encodes video to latents (typically T/4 × H/8 × W/8). Spatiotemporal MM-DiT operates on latents with 3D-RoPE. Loss: diffusion or flow matching.

7.2 Open lineup

• ModelScope T2V (2023): early open.
• AnimateDiff (2023): plug temporal modules into SD U-Net.
• SVD / SVD-XT (Stability 2023): image-to-video.
• ZeroScope: ModelScope improvements.
• CogVideoX / 1.5 (Tsinghua 2024): DiT, T5 conditioning.
• Open-Sora / Open-Sora-Plan: Sora replications.
• Mochi 1 (Genmo 2024): 10B AsymmDiT, fully open.
• Hunyuan Video (Tencent 2024): 13B, frontier-open quality.
• LTX-Video (Lightricks 2024): real-time on A100.
• Wan 2.1 (Alibaba 2025): open frontier.
• Step-Video, Allegro: other open frontier.

7.3 Closed frontier

• Sora / Sora 2 (OpenAI 2024–25): 60s+ clips.
• Veo 2 / Veo 3 (Google 2024–25): with audio.
• Runway Gen-3 / Gen-4.
• Kling 1.6 / 2 (Kuaishou).
• Pika 1 / 2.
• MovieGen (Meta 2024): joint video + audio.
• MiniMax Hailuo, Luma Dream Machine, Vidu.

7.4 Image-to-video, animation, controllable

AnimateAnyone, MimicMotion, Champ, MagicAnimate: pose-driven character animation. Live Portrait: image + driving video → animated portrait. EMO, Audio2Photoreal: audio-driven talking head.

7.5 World-model framing for video

Sora, Veo, GAIA, Cosmos all double as learned simulators when conditioned on actions. Used for AV closed-loop simulation, robotics RL, embodied training.

8. Audio and Music Foundation Models

8.1 Speech / ASR

• Wav2Vec 2.0, HuBERT, WavLM: SSL speech encoders.
• Whisper (OpenAI 2022): 680k-hour ASR; multilingual; robust to noise. Standard open ASR.
• Conformer: convolution-augmented Transformer for ASR.

8.2 TTS (text-to-speech)

• NaturalSpeech, NaturalSpeech 3.
• VALL-E, VALL-E X.
• F5-TTS: flow matching for TTS.
• XTTS, Tortoise.

8.3 Audio / music generation

• AudioLM (Google 2022): hierarchical (semantic + acoustic) Transformer.
• MusicGen, AudioGen (Meta): single-stage Transformer over EnCodec tokens.
• AudioLDM / 2: latent diffusion for audio.
• Stable Audio / Stable Audio Open: latent DiT for music.
• Suno, Udio (closed): frontier music gen; vocals + instruments.
• MMAudio, V2A models: video-to-audio generation.

8.4 Audio understanding

• CLAP (audio-text contrastive).
• AudioFlamingo, Qwen-Audio, Qwen2-Audio: audio LLMs.
• GPT-4o, Gemini 2.5: native audio understanding.

9. NeRF, 3DGS, and 3D Scene Foundation

9.1 NeRF lineage

• NeRF (2020): MLP-based neural radiance field.
• Instant-NGP (2022): hash-grid encoding; trains in seconds.
• Mip-NeRF, Mip-NeRF 360: anti-aliasing + unbounded scenes.
• Zip-NeRF: combines hash + anti-alias.
• Plenoxels, TensoRF: explicit / decomposed grids.

9.2 3D Gaussian Splatting (Kerbl SIGGRAPH 2023)

Each Gaussian: position \(\mu\), anisotropic covariance \(\Sigma = R S S^\top R^\top\), opacity \(\alpha\), SH color \(c(d)\). Differentiable rasterization; alpha-composite front-to-back.

9.3 3DGS variants

Major 3DGS variants

• Mip-Splatting: 2D mip + 3D smoothing for anti-alias.
• 2D-GS: planar disks for surfaces.
• Scaffold-GS, Octree-GS: anchor-based, memory-efficient.
• 4D-GS / Deformable 3DGS: dynamic scenes with time-dep \(\mu(t)\), \(\Sigma(t)\).
• SuGaR / Gaussian Frosting: mesh extraction.
• Relightable 3D Gaussians, GS-IR: per-Gaussian BRDF + lighting.
• LightGaussian, CompGS: compression to \(<\) 10% storage.
• CityGaussian, Hierarchical 3DGS: city-scale / kilometer scenes via LoD.

9.4 NeRF vs 3DGS decision

• Real-time render: 3DGS wins.
• Large dynamic scenes: 4D-GS or NeRF + deformation MLP.
• Relighting: GS-IR / 3D-NeuS leading; both still early.
• Mesh integration: 3DGS with extraction.
• Storage compressed: NeRF often wins.
• Training time: 3DGS faster (\(\sim\) 30 min vs hours).

9.5 3D-aware diffusion

IBRNet, GNT: render novel views by attending across reference images. ReconFusion, ZeroNVS: diffusion priors for sparse-view reconstruction.

10. Feed-Forward 3D (Single / Few-View Reconstruction)

10.1 LRM family (large reconstruction models)

• LRM (Adobe 2023): single image → triplane neural rep in one transformer pass.
• InstantMesh: extracts mesh from LRM output.
• MeshLRM: predicts mesh directly.
• TripoSR, CRM, SF3D, SPAR3D: variants.
• GS-LRM, Long-LRM: predict 3D Gaussians directly.

10.2 DUSt3R / MASt3R / VGGT (the SfM-eclipse)

Already covered in §4. Predict per-pixel pointmaps from N images directly; camera + depth + matches all decoded.

2024–25 dominant approach.

10.3 3D generation (text/image-to-3D)

3D generation lines

• Optimization-based (SDS): DreamFusion, Magic3D, ProlificDreamer (VSD). Hours per scene; Janus problem.
• Multi-view diffusion: MVDream, ImageDream, Wonder3D, SyncDreamer, Zero123 / Zero123++ / Stable Zero123. Generate consistent views; reconstruct.
• Native 3D diffusion (the 2025 wave): Trellis (Microsoft), Hunyuan3D-2 (Tencent), CLAY, Direct3D, Rodin Gen-1.5. Seconds per asset; commercial-grade.

10.4 Mesh generation Transformers

MeshGPT, MeshXL, MeshAnything V2, EdgeRunner, BPT: autoregressive face-vertex generation; native mesh topology.

★ 2026 SOTA update — Single-image + native 3D generation

• SAM 3D: Meta visually-grounded single-image 3D; predicts geometry, texture, pose/layout even under occlusion/clutter; SAM 3D Objects + SAM 3D Body; synthetic pretrain then real-world alignment.
• Hunyuan3D 2.5: Tencent shape foundation model LATTICE up to 10B params + PBR multi-view texturing; sharp detailed meshes with precise image-3D alignment.

11. Avatars and Humans

11.1 Photoreal avatars

• Codec Avatars 2.0 / 3.0 (Meta): photoreal full-body and face from multi-view captures.
• Apple Persona: 3D head/shoulder from few-second enrollment; FaceTime in Vision Pro.
• Gaussian Avatars / GaussianHead / FlashAvatar / LiveHead: 3DGS-based, lower capture cost, realtime.
• Animatable Gaussians, IMavatar, MonoGaussianAvatar: from monocular video.

11.2 Body models

• SMPL, SMPL-X, SMPL-H: parametric body models.
• PHALP, SMPLer-X, OSX, NLF (Neural Localizer Fields), CLIFF: pose / shape regressors.
• 4D-Humans, 4D-DRESS, GauHuman: full-body 4D reconstruction.

11.3 Audio / image-driven animation

• EMO (Alibaba 2024): audio-driven talking-head.
• Audio2Photoreal (Meta): audio → photoreal full-body.
• Live Portrait: image + driving video → animated portrait.
• V-Express, Hallo: similar lines.

11.4 Pose-driven character animation

AnimateAnyone, MimicMotion, Champ, MagicAnimate: drive a static character with a pose video. Standard for short-form content creation.

12. World Models

12.1 What is a world model?

A generative model conditioned on actions / state that predicts future observations. Used as a learned simulator for RL and as a substrate for embodied AI.

12.2 Genie family (DeepMind)

Genie (2024): playable video world models trained from internet game footage. Discover "actions" unsupervised.

Genie 2 (2025): consistent world models from a single image; supports interactive play.

12.3 GAIA (Wayve)

GAIA-1 / GAIA-2: video diffusion conditioned on past frames + actions; closed-loop AV simulator.

12.4 Cosmos (NVIDIA)

Cosmos World Foundation Models (2025): both diffusion and autoregressive variants; video tokenizer (continuous and discrete); positioned as the world-model platform for robotics + AV.

12.5 Other world models

Vista, MagicDrive3D, EmerNeRF: scene-level synthesis with action / camera control. DriveDreamer, DriveDreamer-2: text-controlled driving scenarios.

12.6 World models for control

Dreamer V3 (DeepMind): model-based RL with RSSM; scales across >150 tasks. IRIS, DIAMOND: tokenizer / diffusion world models for Atari.

★ 2026 SOTA update — Self-supervised structured world models

• PSI: Probabilistic Structure Integration; learns promptable world model from 1.4T video tokens via probabilistic prediction + causal structure extraction + integration; yields SOTA optical flow, depth, segmentation zero-shot.

13. Vision-Language-Action (VLA) Models

13.1 The recipe

Take a pretrained VLM, add an action head, fine-tune on robot trajectories. Action head: discrete action tokens or continuous via flow matching.

13.2 Google Robotics line

• RT-1 (2022): small Transformer, discretized actions.
• RT-2 (2023): VLM (PaLI-X / PaLM-E) with action tokens in vocab; co-trained on web + robot.
• RT-X (2023): Open-X-Embodiment cross-embodiment training.

13.3 Open VLAs

• OpenVLA (2024): 7B Llama 2 + DINOv2 + SigLIP + discretized actions. Strong baseline.
• Octo: smaller policy.
• RDT-1B: 1B-param diffusion-policy Transformer.

13.4 Physical Intelligence's π0 family

• π0 (2024): VLM + flow-matching action head:

\[\mathcal{L}_{\mathrm{FM}} = \mathbb{E}\,\big\| v_\theta(a_t, t, o) - (a_1 - a_0) \big\|^2 .\]

Pretrained on \(\sim\) 10k hours of robot data.

• π0.5 (2025): hierarchical control for long-horizon tasks.

13.5 Humanoid VLAs

• GR00T (NVIDIA): open VLA + action stack with Cosmos sim.
• Helix (Figure): two-network humanoid VLA (slow planner + fast executor).

13.6 Spatial-reasoning VLMs for robotics

SpatialVLM, SpatialBot, RoboPoint, SpatialVLA, RoboVLM: fine-tune VLMs with synthetic 3D-grounded spatial QA. Improves spatial reasoning for robot tasks.

13.7 Older embodied VLMs

PaLM-E (Google 2023): early multimodal embodied LLM; predates RT-2.

★ 2026 SOTA update — Open VLA frontier (2025)

• GR00T N1: NVIDIA open humanoid VLA; dual-system Eagle-2 VLM backbone + DiT flow-matching action policy; single weights drive single/bi-arm/humanoid; trained on robot + human video + synthetic.
• SmolVLA: ~450M compact open VLA (trimmed SmolVLM-2 + action expert) trained on community LeRobot data; async inference for real-time control on single GPU/CPU.

14. Embedding / Retrieval Foundation Models

14.1 Multimodal embedders

• CLIP / SigLIP: standard for image-text retrieval.
• BGE-VL, EVA-CLIP-VL: open larger / better.
• Voyage AI multimodal-3, OpenAI text-embedding-3 multimodal: closed APIs.
• Cohere Embed v3 multimodal.

14.2 Multi-vector / late-interaction

• ColBERT, ColBERTv2: per-token vectors + max-similarity per query token.
• ColPali, ColQwen2: vision-text multi-vector for document-image retrieval; emerging standard for OCR-heavy docs.

14.3 Document understanding

• Donut, Pix2Struct: OCR-free document understanding.
• LayoutLM v3: layout + text + image.
• Kosmos-2.5: text-to-image grounding for documents.

15. Specialized / Domain-Specific Foundation Models

15.1 Medical imaging

• Med-PaLM, Med-PaLM 2 (Google): medical LLM.
• Med-Gemini: native multimodal medical.
• MedSAM: SAM fine-tuned for medical segmentation.
• RETFound, EyeFM: ophthalmology foundation models.
• ChestX-Det, RadFM: radiology.

15.2 Earth observation / remote sensing

• SatlasPretrain, Prithvi (NASA-IBM): satellite imagery foundation models.
• EarthFormer, ClimaX: Earth science / weather.

15.3 Scientific imaging

ESM-3 (Meta): protein structure + function. AlphaFold-2 / 3 (DeepMind): protein structure / interactions.

16. Action / Motion Foundation Models

16.1 Human motion

• MotionDiffuse, MDM (Motion Diffusion Model), PhysDiff: text-to-motion via diffusion.
• HumanML3D, Motion-X: datasets that anchor the field.
• TMR, MotionCLIP: motion-text contrastive embedders.

16.2 Robot manipulation policies

Diffusion Policy, ACT, 3D Diffuser Actor, RDT-1B: imitation-learning policy classes (covered in robotics section).

17. Specialized Vision Models Worth Knowing

17.1 Human-related

• Hands: MediaPipe Hands, HaMeR, WiLoR.
• Faces: ArcFace (recognition), MICA, EMOCA.
• Pose: OpenPose, RTMPose, HRNet.

17.2 Tracking

ByteTrack, OC-SORT, BoT-SORT, MeMOTR, Cutie, SAM 2.

17.3 Stylization

StyleGAN-3, StyleGAN-T, GigaGAN: legacy GAN family. Largely displaced by diffusion for new releases.

17.4 Restoration

Real-ESRGAN, SwinIR, Restormer, Stable-Diffusion-Upscaler, FLUX-Upscaler: super-res, denoise, deblur.

18. Cross-Cutting Architectural Patterns

18.1 The 2026 default block

• Pre-norm with RMSNorm.
• RoPE positional encoding (1D text, 2D vision, 3D video).
• GQA / MQA / MLA for KV efficiency.
• SwiGLU MLP.
• FlashAttention 3 kernels.
• BF16 weights, FP8 forward where supported.

18.2 Vision tokenizers

• Continuous: ViT patch embed; 2D-RoPE positions.
• Discrete (for AR generation): VQ-GAN, MAGVIT-v2 (LFQ), FSQ, Cosmos Tokenizer.
• Compressive: TiTok 1D (\(\sim\) 32 tokens / image).
• Video: Causal 3D VAE (T/4 × H/8 × W/8).

18.3 Conditioning for diffusion

• adaLN-Zero: scale + shift LayerNorm from condition; gate residual init zero.
• Cross-attention to text: U-Net standard.
• Joint attention: MM-DiT (SD3, FLUX).
• ControlNet: clone encoder + zero-init connections.
• IP-Adapter: parallel image cross-attn.

19. The 2026 Production Stack

Appendix A: Names to Recognize on Sight

If an interviewer says these, you should know exactly what they refer to:

• CLIP / SigLIP / DINOv3: vision encoders.
• SAM 2: promptable segmentation.
• Grounding DINO: open-vocab detection.
• DUSt3R / MASt3R / VGGT: feed-forward 3D.
• 3DGS / 3D Gaussian Splatting: scene rep.
• MAE / V-JEPA: SSL recipes.
• LLaVA / InternVL / Qwen-VL: open VLMs.
• GPT-4o / Gemini 2.5 / Claude Opus: closed VLMs.
• Chameleon / Show-o / Emu3: native multimodal.
• SDXL / SD3 / FLUX: image diffusion.
• MM-DiT: image diffusion architecture.
• Sora / Veo 3 / Kling 2 / Hunyuan Video / Wan 2.1: video gen.
• MovieGen: joint video + audio.
• LRM / GS-LRM / MeshLRM: feed-forward 3D.
• Trellis / Hunyuan3D-2: native 3D generation.
• π0 / OpenVLA / GR00T / Helix: VLAs.
• GAIA / Cosmos / Genie 2: world models.
• Whisper: ASR.
• Stable Audio / MusicGen / Suno: audio gen.
• Codec Avatars / EMO / Live Portrait: avatars.
• R1 / VLM-R1 / Vision-R1: reasoning models.
• ControlNet / IP-Adapter: conditioning.

Appendix B: Family Tree (One-Page Mental Map)

Vision encoders

CLIP → ALIGN → OpenCLIP → SigLIP / SigLIP 2 / EVA-CLIP / MetaCLIP / DFN.

DINO → DINOv2 → DINOv3.

MAE → BEiT v2 → EVA / EVA-02 → FCMAE.

I-JEPA → V-JEPA → V-JEPA 2.

VLMs

Flamingo (cross-attn) / BLIP-2 (Q-Former) → LLaVA (MLP) → LLaVA-NeXT → LLaVA-OneVision.

Qwen-VL → Qwen2-VL (2D-RoPE) → Qwen2.5-VL → Qwen3-VL.

InternVL → InternVL2/2.5/3.

Chameleon (early-fusion VQ) / Janus / Show-o / Transfusion / Emu3.

Image gen

GLIDE / DALL-E 1/2 → SD 1.5 (LDM) → SDXL → PixArt-α/Σ (DiT) → SD3 (MM-DiT + RF) → FLUX.

Video gen

ModelScope / AnimateDiff / SVD → CogVideoX / Mochi / Open-Sora → Hunyuan Video / Wan 2.1 / Sora / Veo 3 / Kling 2 / MovieGen / Cosmos.

3D

NeRF → Instant-NGP → 3DGS → Mip-Splatting / 4D-GS / Scaffold-GS / GS-IR.

LRM → MeshLRM / GS-LRM → Trellis / Hunyuan3D-2.

DUSt3R → MASt3R / Spann3R → VGGT.

Robotics

RT-1 → RT-2 → RT-X / OpenVLA → π0 / π0.5 / GR00T / Helix.

Reasoning VLMs

LLaVA-CoT / LLaVA-o1 / Mulberry / Insight-V / Vision-R1 / VLM-R1 / R1-V / MM-EUREKA / LMM-R1 / Video-R1.

Appendix C: Comparison Tables

Open VLMs (2026)

Open T2I models (2026)

Open T2V models (2026)

Closed frontier (April 2026)