VLA Models — Vision-Language-Action

Updated July 2026 with 2025–2026 SOTA additions — new entries marked ★. Algorithm names link to their papers (arXiv / project page).

April 2026 · Version 1.0

Contents

1. What Is a VLA Model?
2. Anatomy of a VLA
3. Action Representations
4. Action Chunking
5. Major VLA Models: Google's Robotics Line
6. Open VLA Lineage
7. π0 Family (Physical Intelligence)
8. NVIDIA GR00T
9. Figure's Helix
10. Spatial-Aware VLAs
11. Pretraining and Post-Training Pipeline
12. Datasets and Embodiments
13. Sim-to-Real for VLA
14. Real-Time Inference
15. RL Fine-Tuning of VLAs
16. Multi-Embodiment and Generalization
17. Evaluation
18. Deployment Patterns
19. Frontier 2025–2026
20. The 2026 Production Stack Appendix A: Twenty-Five Things to Know Appendix B: Decision Tree — "Which VLA?" Appendix C: Year-by-Year VLA Milestones

1. What Is a VLA Model?

1.1 Working definition

A Vision-Language-Action (VLA) model is a single neural network that takes (a) visual observations and (b) a natural-language instruction (and often (c) proprioception), and outputs (d) actions for a robot.

\[\pi_\theta : (o_t^{\text{img}},\ c_{\text{instr}},\ o_t^{\text{prop}}) \to a_t.\]

1.2 Why VLA?

• Multi-task: one model handles many tasks via language conditioning.
• Multi-embodiment: one model across robots (with adaptation).
• Generalization: VLM backbone provides web-knowledge prior.
• Composability: chain instructions in long-horizon plans.
• Foundation-model trajectory: matches the pattern that worked for LLMs (pretrain + fine-tune at scale).

1.3 The recipe (in one sentence)

Key

Take a pretrained VLM, attach an action head, train on robot trajectories. The pretrained VLM provides visual + language understanding; the action head produces (or predicts) control signals.

1.4 VLA vs traditional robotics

1.5 The 2024–26 inflection

• 2022 (RT-1): discrete action tokens via small Transformer.
• 2023 (RT-2, PaLM-E): VLM + action token vocab; co-train with web data.
• 2024 (OpenVLA, Octo, π0): open foundation VLAs; flow matching for actions.
• 2025 (GR00T, Helix, π0.5): humanoid foundation VLAs; production deployment.

1.6 The principal-level claim

Key

Robotics is having its ImageNet moment via cross-embodiment data (Open-X-Embodiment) + VLM pretraining + scalable action heads. By 2026, frontier robotics is dominated by VLA pretraining + per-task fine-tuning, not per-task RL.

2. Anatomy of a VLA

2.1 Visual encoder

• Pretrained vision encoder (DINOv2, SigLIP, CLIP, custom ViT).
• Multi-camera: wrist cam + 3rd-person cam.
• History: K recent frames concatenated.
• Sometimes 3D: depth, point cloud, voxel.

2.2 Language understanding

• Text instruction tokenized (with VLM's tokenizer).
• Sometimes extended via task / context tokens.
• Free-form language is the standard interface.

2.3 LLM / VLM backbone

• Pretrained on web-scale data.
• Common choices: Llama 2 / 3 (OpenVLA, RT-2), PaliGemma (π0), PaLM-X (RT-2-X).
• Frozen base + LoRA, or full fine-tune.

2.4 Proprioception

• Joint angles, velocities, gripper state, end-effector pose.
• Encoded as small vector → embedded → injected via concatenation or cross-attn.
• Critical for fine manipulation.

2.5 Action head

The most varied component. Options:

• Discrete: per-dim binning, output via vocab logits.
• Continuous regression: MLP on backbone hidden state.
• Mixture density network: predict GMM.
• Diffusion policy: condition diffusion on backbone output.
• Flow matching (π0): FM on action chunk.

2.6 Output format

• Single-step action: \(a_t \in \mathbb{R}^d\).
• Action chunk: \(a_{t:t+H} \in \mathbb{R}^{H\times d}\) with \(H = 8\text{–}64\).
• Action with reasoning trace: "Thought: ... Action: ...".
• High-level plan + low-level executor: hierarchical (Helix, π0.5).

2.7 Action space dimensionality

• 6-DoF arm + gripper: 7 dims.
• 14-DoF dual-arm + 2 grippers: 16.
• 19-DoF humanoid (e.g., Unitree H1): 19.
• Full-body humanoid (Atlas): 28+.
• Plus base velocity if mobile.

2.8 Frequency / control rate

• Most VLAs output at 5–30 Hz.
• Action chunking + temporal ensembling amortizes inference.
• Onboard low-level controller runs at ≥1 kHz.

3. Action Representations

3.1 Discrete action tokens (RT-1, RT-2, OpenVLA)

Per-dim binning: each action dim discretized into 256 bins; 7-dim arm → 7 tokens per timestep. Embedded into LLM vocabulary so VLM predicts actions as it predicts text.

Pros: integrates cleanly with LLM; cross-embodiment by remapping bins. Cons: discretization noise; \(256^7\) effective combinations.

3.2 Continuous regression

Linear / MLP head on backbone last hidden state → continuous action vector. Cheap; loses uncertainty; collapses to mean for multi-modal actions.

3.3 Mixture Density Networks (MDNs)

Predict GMM over actions:

\[p(a|x) = \sum_k \pi_k(x)\, \mathcal{N}(a;\, \mu_k(x),\, \Sigma_k(x)).\]

Captures multi-modality. Less common in modern VLAs (replaced by diffusion / FM).

3.4 Diffusion policy (Chi et al. 2023)

Generate action chunk via conditional diffusion:

\[\mathcal{L} = \mathbb{E}_{k,\epsilon,a_0,o} \left\| \epsilon - \epsilon_\theta\!\left(\sqrt{\bar\alpha_k}\, a_0 + \sqrt{1-\bar\alpha_k}\,\epsilon,\ k,\ o\right) \right\|^2.\]

Multi-modal action distributions captured naturally. Receding-horizon execution.

3.5 Flow matching action heads (π0)

Faster than diffusion; train with FM objective:

\[\mathcal{L}_{\text{FM}} = \mathbb{E}_{t,\epsilon,a_0,a_1,o} \left\| v_\theta(a_t, t, o) - (a_1 - a_0) \right\|^2.\]

Single-step or few-step inference; continuous actions. Standard in π0 family.

3.6 FAST action tokenizer (Pi)

FAST: BPE-like tokenizer over action sequences. Produces compact discrete tokens that the VLM can predict.

Bridges discrete-token simplicity with continuous-action expressiveness.

3.7 VQ action tokenizer

Learn discrete codebook over action chunks; predict code indices. Used in some recent open-source attempts.

3.8 Action token cheat table

4. Action Chunking

4.1 The idea

Predict H actions at a time, not one. Reduces per-step inference cost; smooths actions; reduces compounding error; captures temporal correlation.

4.2 ACT (Action Chunking Transformer, Zhao et al. 2023)

• Predict \(H = 100\) future actions at once.
• CVAE over action chunk:

\[\mathcal{L}_{\text{ACT}} = \mathbb{E}\left\| \hat a_{t:t+H} - a_{t:t+H} \right\|_1 + \beta\, \mathrm{KL}\!\left(q(z|a,o)\,\|\,p(z)\right).\]

• Strong on bimanual ALOHA tasks.

4.3 Temporal ensembling

At each timestep, average overlapping action chunks generated at different past steps. Smooths jitter; effectively a low-pass filter.

4.4 Diffusion Policy chunks

Predict \(H = 8\text{–}16\) actions via conditional diffusion. Receding-horizon: execute first \(h < H\), then re-plan.

4.5 π0 chunks

Predict 50-step action chunk via FM head. 50 Hz control from a 5 Hz inference.

4.6 Trade-offs

• Larger H: amortizes inference; harder to predict accurately at horizon end.
• Smaller H: more reactive; higher per-step cost.
• Standard: \(H = 8\text{–}50\).

5. Major VLA Models: Google's Robotics Line

5.1 RT-1 (Robotics Transformer, 2022)

• Small Transformer: image + instruction → discretized actions.
• Trained on 130k episodes from 13 robots over 17 months.
• 35M params; modest scale.
• Established the multi-task language-conditioned baseline.

5.2 RT-2 (2023)

• Backbone: PaLI-X (55B) or PaLM-E (12B).
• Action tokens embedded in LLM vocabulary (256 bins per dim).
• Co-trained on web data + robot data: web data prevents catastrophic forgetting; transfers semantic knowledge to robot tasks.
• Achieves emergent capabilities: "move to extinct animal" (knows giraffe is one).

5.3 RT-X / RT-2-X (2023)

• Same architecture trained on Open-X-Embodiment: 1.4M trajectories from 22 embodiments / 60 datasets.
• Demonstrates positive transfer across embodiments.
• Cross-embodiment becomes standard.

5.4 PaLM-E (2023)

• Embodied multimodal language model (562B-param at full scale).
• Multi-modal sentences (text + images + robot states) interleaved.
• Plans over multiple steps; downstream low-level controller.
• Predates RT-2 architecturally; influences subsequent VLAs.

5.5 SARA-RT, RT-Trajectory, RT-Sketch

RT family extensions for various conditioning (drawn trajectories, sketches, etc.).

5.6 ALOHA / Mobile ALOHA / ALOHA Unleashed

Hardware platform (low-cost teleoperation) for collecting bimanual demonstrations. Used to train ACT, Diffusion Policy. Influential reference platform for many open VLAs.

★ 2026 SOTA update — Gemini Robotics (Google DeepMind)

• Gemini Robotics: Gemini-2.0-based generalist VLA + Gemini Robotics-ER (embodied reasoning) variant; dexterous, reactive manipulation, strong open-vocab instruction following and zero/few-shot generalization; on-device version for local inference.
• Gemini Robotics 1.5: dual-model stack — GR-ER 1.5 (embodied reasoning/planner that 'thinks' before acting) + GR 1.5 (action VLA); adds agentic multi-step reasoning and Motion Transfer for cross-embodiment skill sharing.

6. Open VLA Lineage

6.1 OpenVLA (Stanford 2024)

• Backbone: Llama 2 7B + DINOv2 + SigLIP.
• Action head: discretized 7-dim actions (256 bins each).
• Trained on Open-X-Embodiment.
• LoRA fine-tuning for new tasks.
• Open weights + training code; the open RT-2 reference.

6.2 Octo (Stanford / Berkeley 2024)

• Smaller than OpenVLA (27M / 93M params).
• Diffusion action head.
• Open Octo-base / Octo-small.
• Cross-embodiment via padding action dims.

6.3 RDT-1B / RDT-2B (Robotics Diffusion Transformer)

• 1B-param diffusion policy Transformer.
• Pretrained on 46 robot datasets.
• Strong on bimanual / dexterous tasks.
• Open release.

6.4 Octo + similar small open VLAs

• Trade scale for accessibility.
• Easier to fine-tune on consumer hardware.
• Quickly adopted by academic labs.

6.5 CogACT

Diffusion Action Transformer with video pretraining. Improves multi-modal action distribution capture.

6.6 TraceVLA

Conditioned on visual "traces" (drawn paths) for object-relative motion. Useful for demonstration-style conditioning.

★ 2026 SOTA update — OpenVLA successors & compact open VLAs

• OpenVLA-OFT: Optimized Fine-Tuning recipe — parallel decoding + action chunking + continuous L1-regression head; lifts LIBERO 76.5%→97.1% and ~26× faster inference; now a standard open fine-tuning baseline.
• SmolVLA: 450M open VLA (HF/LeRobot) on trimmed SmolVLM-2 + flow-matching action expert; trained on ~23K community episodes; asynchronous inference stack; runs/trains on consumer hardware (~18ms/step, <1GB VRAM).
• MolmoAct: AI2 'Action Reasoning Model' — grounds perception into spatial reasoning tokens then predicts actions; fully open weights + data + reasoning traces; explainable, steerable manipulation.
• Dita: scalable Diffusion Transformer generalist policy denoising continuous action chunks with in-context conditioning on raw visual tokens; strong cross-embodiment scaling and 10-shot real-world adaptation.

7. π0 Family (Physical Intelligence)

7.1 The premise

Build a single foundation model for robotics analogous to GPT-3 for language. Train at scale on diverse embodiments.

7.2 π0 (2024)

• Backbone: PaliGemma-3B (Google's open VLM).
• Flow-matching action head: small expert network on top of VLM.
• Trained on ∼10k hours of robot data across 7 robot platforms.
• Predicts 50-step action chunks.
• Continuous actions; FM enables single-step or few-step inference.
• Fine-tunes on 1–10 hours of new task data for production-quality.

7.3 π0 FM action head equation

\[\mathcal{L}_{\text{FM}} = \mathbb{E}_{t\sim U(0,1),\, a_0,\, a_1,\, o} \left\| v_\theta\!\left((1-t)a_0 + t a_1,\ t,\ o\right) - (a_1 - a_0) \right\|^2,\]

\(a_0 \sim \mathcal{N}(0, I)\), \(a_1\) = real action chunk, conditioned on observation \(o\).

7.4 π0.5 (2025)

• Hierarchical: high-level VLM plans + low-level FM head executes.
• Long-horizon manipulation (multi-minute tasks).
• Better generalization to unseen environments.
• Folding laundry, full kitchen tasks demoed.

7.5 Why FM for actions?

• Multi-modal action distributions captured (multiple valid ways to pick up a cup).
• Continuous outputs (no discretization noise).
• Few-step inference (vs many-step diffusion).
• Composes cleanly with VLM backbone.

7.6 Practical impact

π0 deployed in production for several Physical Intelligence customers (warehouse, kitchen tasks). π0.5 pushes long-horizon humanoid manipulation.

★ 2026 SOTA update — π*0.6 — VLA that learns from experience

• π*0.6: Physical Intelligence's self-improving π-family VLA trained with RECAP (RL with Experience & Corrections via Advantage-conditioned Policies): demos + real-time expert corrections + autonomous RL on-robot; ~2× throughput and ~half the failure rate on hard tasks (espresso, laundry, factory box assembly) — 'RL is back' for VLAs.

8. NVIDIA GR00T

8.1 The platform play

GR00T = General Humanoid Robot platform. Combines:

• Foundation VLA (GR00T N1, N2).
• Cosmos World Foundation Models for sim.
• Isaac Lab for physics.
• Open-X-Embodiment-style data.
• NVIDIA hardware (Jetson Thor edge GPU).

8.2 GR00T N1 (2025)

• Two-network architecture: System 2 (deliberate, slow) + System 1 (reactive, fast).
• Diffusion Transformer action head.
• Pretrained on humanoid-specific data.
• Open weights for some versions.

8.3 GR00T N2

Scaled successor; better long-horizon, better cross-embodiment.

8.4 Cosmos integration

Cosmos World Foundation Models generate counterfactual rollouts; GR00T trained / fine-tuned in Cosmos as world simulator.

8.5 Isaac Lab + Sim-to-real

Asymmetric A-C with privileged state in sim; distill to vision-only student; deploy on real humanoid.

8.6 Jetson Thor

On-board edge GPU for running VLA inference at ∼10 Hz on the robot. Critical for autonomy.

9. Figure's Helix

9.1 The two-network split

Helix uses System 1 / System 2 architecture explicitly:

• S2 (slow, large VLM): high-level reasoning, scene understanding, plan.
• S1 (fast, small policy): low-level reactive control, on-board real-time.
• Inspired by Kahneman's dual-process model.

9.2 Why split?

• S2 latency too high for real-time control.
• S1 fast enough but lacks reasoning.
• Combined: best of both.

9.3 On-board inference

Helix runs entirely on-robot (no cloud dependency). Critical for production humanoid deployment in homes / warehouses.

9.4 Multi-robot coordination

Demonstrated: two Figure robots collaborating on a single task with shared S2 planning. Foreshadows multiagent VLA.

9.5 Industrial deployment

Figure has BMW partnership for industrial humanoid use; Helix is the policy.

10. Spatial-Aware VLAs

10.1 The problem

Standard VLMs (LLaVA, Qwen-VL) trained on internet text+images are weak at 3D spatial reasoning critical for robot tasks.

10.2 SpatialVLM (Google)

Fine-tune VLM on synthetic 3D-grounded spatial QA data:

• "How far apart are these two objects?"
• "Which is closer to the camera?"
• "Where is X relative to Y in 3D space?"

Massively improves spatial reasoning. Useful for any robot task requiring 3D understanding.

10.3 SpatialVLA, SpatialBot, RoboPoint, RoboVLM

Various follow-ups with different data / architectures. SpatialVLA is full VLA; SpatialBot is VLM only.

10.4 3D-VLA

Use 3D representation (point cloud, voxel grid, or 3DGS) as additional input to VLA. Better object-relative reasoning.

10.5 Affordance prediction

Predict "where to grasp" / "where to pour" / "where to push" as auxiliary head. Improves manipulation.

10.6 Depth-aware VLAs

Some VLAs add a depth encoder branch. Useful for cluttered manipulation. Trade-off: more parameters; not all sensors have depth.

11. Pretraining and Post-Training Pipeline

11.1 The four stages

1. VLM pretraining: standard web-scale image+text training (often inherited).
2. Action co-training: train on robot data + web data simultaneously.
3. Robot-specific fine-tuning: on target embodiment / task.
4. RL fine-tuning (optional): residual RL to surpass demonstrations.

11.2 Why co-train with web data?

• Prevents catastrophic forgetting of language / visual knowledge.
• Maintains semantic transfer (model knows what a "glass of milk" is).
• RT-2 demonstrated this is critical for emergent capabilities.

11.3 Per-embodiment adaptation

• Different robots have different action dims, kinematics.
• Approaches: per-robot embedding token, per-robot action heads, padding common action dims, embodiment-aware tokenizer.

11.4 LoRA fine-tuning

• Train rank-8 to rank-256 LoRA adapters on attention + MLP.
• Base model frozen; per-task / per-embodiment LoRAs.
• Stack multiple LoRAs for compositionality.
• Standard for OpenVLA-class deployment.

11.5 Sample-efficient task fine-tuning

• 100–1000 task demonstrations typical.
• LoRA + frozen base.
• Few hours of training on consumer GPU.
• Production-quality on simple tasks.

11.6 Hierarchical fine-tuning (π0.5, Helix)

• Fine-tune only S1 (fast policy) per task.
• S2 (slow planner) frozen across tasks.
• Faster + cheaper than full-model fine-tuning.

12. Datasets and Embodiments

12.1 Open-X-Embodiment (OXE, 2024)

• ∼1.4M trajectories.
• 22 robot embodiments.
• 60 datasets aggregated.
• Standardized format (RLDS).
• The foundation for OpenVLA / Octo / RDT / RT-X.

12.2 DROID (2024)

• 76k demos from 13 institutions.
• Distributed labeling; common Franka platform.
• Diverse environments (kitchens, offices, etc.).

12.3 Bridge V2 (Berkeley)

• 60k demos across many tabletop tasks.
• Pre-OXE; influential.

12.4 ALOHA / Mobile ALOHA datasets

Bimanual demos collected on ALOHA hardware. Training data for ACT, Diffusion Policy.

12.5 AgiBot World (2024)

Large-scale humanoid dataset from AgiBot (China). Driving Asian-led humanoid VLA development.

12.6 RH20T (2024)

• 110k diverse trajectories.
• 147 task types.
• 7 robot configurations.
• Strong open dataset.

12.7 DexCap, UMI

• DexCap: dexterous-hand demos with motion capture.
• UMI (Universal Manipulation Interface): handheld grippers with cameras for in-the-wild demos.

12.8 1X World Model dataset

Humanoid robot footage from 1X. Used for both VLA training and world model.

12.9 Embodiment landscape

13. Sim-to-Real for VLA

13.1 Why sim?

• Real data expensive: ∼$5–50 per demo.
• Sim is essentially free at scale.
• Counterfactual scenarios (rare events).
• RL training (millions of episodes).

13.2 The reality gap

Visual + dynamics differences between sim and real. VLA-specific challenges:

• Visual: textures, lighting, materials.
• Dynamics: friction, mass, motor characteristics.
• Sensor: camera intrinsics, depth noise.
• Embodiment: real robot has wear, calibration drift.

13.3 Domain randomization

Per episode, randomize:

• Visual: textures, lighting, camera poses.
• Dynamics: mass, friction, damping.
• Sensor: depth noise, image noise.
• Initial conditions: object placement, robot pose.

Trained policy invariant to these variations → transfers.

13.4 Asymmetric Actor-Critic

• Critic uses privileged info (full state).
• Actor uses egocentric observations only.
• Variance reduction without privileged info leak.

13.5 Privileged-to-vision distillation

After RL, distill privileged actor (sees state \(s\)) into vision-only student (sees \(o\)):

\[\mathcal{L}_{\text{distill}} = \mathbb{E}\left\| \pi_S(o) - \pi_T(s) \right\|^2.\]

Standard for production sim-to-real.

13.6 Cosmos / Isaac Lab integration

Cosmos World Foundation Models provide photoreal video; Isaac Lab provides physics. Combine for rich training environments.

13.7 Mixed sim+real training

• Pretrain on sim at scale.
• Fine-tune on real.
• Mix sim+real batches in fine-tuning.
• Anneal sim ratio down over time.

13.8 World-model-as-simulator (frontier 2025)

Use video world model (Sora-class) as the simulator instead of physics engine. Higher visual realism; physics fidelity still emerging.

14. Real-Time Inference

14.1 Latency budget

• Action chunking: predict 8–50 actions; execute over a window.
• Inference rate: 2–10 Hz.
• Effective control rate: 10–100 Hz via chunking.
• Low-level controller (PD, MPC): 1+ kHz on-board.

14.2 Distillation for deployment

• Distill 7B VLA into 1B / 0.5B for edge.
• Especially compelling because robot policies don't need broad knowledge — just task-relevant.
• Per-deployment distillation common.

14.3 Quantization

• INT8 / INT4 weights for VLA backbone.
• FP8 on H100 / Jetson Thor (with FP8 support).
• Action head usually kept FP for precision.

14.4 On-board vs off-board

• Off-board (cloud / desktop GPU): higher quality, latency penalty.
• On-board (Jetson / edge GPU): low latency, smaller model.
• Hybrid (Helix): S2 off-board / S1 on-board possible but complex.

14.5 Tokenization speed

Image encoding can dominate inference. Approaches:

• Lower-resolution input.
• Pixel unshuffle for token compression.
• Cache image features across timesteps when possible.
• Lightweight vision encoders.

14.6 S1 / S2 separation latency model

• S2 fires every 1–2 seconds (planning).
• S1 fires every 10–100ms (control).
• S2 output cached / interpolated for S1.

★ 2026 SOTA update — Unified dual-system fast/slow VLA

• FiS-VLA: embeds System 1 inside System 2 by repurposing the final VLM transformer blocks as the fast reactive head (shared params), keeping the full VLM for slow reasoning; high-freq inputs (state, images, point clouds) to S1; ~118 Hz control while beating prior dual-system SOTA.

15. RL Fine-Tuning of VLAs

15.1 Why RL on top of BC?

BC bounded by demonstration quality; RL can surpass. But RL from scratch on real robots is sample-prohibitive.

RL fine-tuning of pretrained VLA is the sweet spot.

15.2 Residual policy

• Pretrained VLA \(\pi_{\text{base}}\) frozen or slow-updated.
• Small residual network \(\pi_{\text{res}}\) trained via RL.
• Output: \(a = \pi_{\text{base}}(o) + \pi_{\text{res}}(o)\).
• Sample-efficient because base provides strong prior.

15.3 Q-chunking

RL on action chunks (not single steps). Each chunk treated as one "meta-action." Easier credit assignment for long-horizon tasks.

15.4 Diffusion DPO for actions

• Collect preference pairs (trajectory A vs B).
• Run Diffusion-DPO on diffusion policy or FM head.
• Aligns to human preference / safety.

15.5 DPPO (Diffusion Policy PO)

Cast diffusion policy as multi-step MDP; PPO with reward = task success.

15.6 GRPO on action sequences

Sample multiple rollouts per task; compare; train with group-relative advantage. Verifiable reward = task success / completion / safety.

15.7 IRL / inverse RL

Recover reward from demos; then RL with that reward. Useful when explicit reward is hard to specify.

15.8 Practical recipes

• Pretrain VLA on π0-style scale.
• LoRA fine-tune on target task BC data.
• Residual RL with massive parallel sim.
• Distill to deployable model.

★ 2026 SOTA update — RL post-training of VLAs (2025)

• SimpleVLA-RL: scalable veRL-based RL framework for VLAs with sparse success rewards; raises OpenVLA-OFT from 17.3→91.7 on LIBERO with one-trajectory cold-start SFT; beats π0/RDT on RoboTwin and real robots; mitigates SFT data scarcity (ICLR 2026).
• RIPT-VLA: Reinforcement Interactive Post-Training with dynamic rollout sampling + leave-one-out advantage; sparse binary rewards push OpenVLA-OFT to 97.5%; turns a 4% one-demo SFT model into 97% within ~15 iterations.
• ConRFT: two-stage consistency-policy reinforced fine-tuning (offline BC+Q-learning, then online consistency policy with human-in-the-loop); 96.3% avg on 8 real tasks in 45–90 min (RSS 2025).

16. Multi-Embodiment and Generalization

16.1 Cross-embodiment learning

• Different robots have different kinematics, action spaces, cameras.
• Naive training fails: positive transfer rare.
• Open-X-Embodiment showed positive transfer across embodiments.

16.2 Embodiment representation

• One-hot / learnable embodiment token in input.
• Per-embodiment LoRA on shared backbone.
• Action-space normalization (canonical action space, decode per-robot).

16.3 Action canonicalization

Normalize across embodiments:

• End-effector pose deltas (universal).
• Per-robot decoder from canonical → joint commands.

16.4 Camera frame canonicalization

Standardize camera mounting / FOV across embodiments.

16.5 What transfers

• Visual scene understanding (universal).
• Object semantics (universal).
• High-level plans (mostly universal).
• Low-level dynamics (embodiment-specific).

16.6 What doesn't transfer (yet)

• Fine motor skill (hand-specific).
• Robot-specific calibration.
• Soft-body / dexterous tasks.
• Multi-arm coordination across hardware.

★ 2026 SOTA update — Latent-action cross-embodiment learning

• UniVLA: learns task-centric latent actions unsupervised from any-embodiment (incl. actionless human) video in DINO feature space, then decodes per-robot; SOTA on manipulation+navigation with <1/20 pretrain compute and 1/10 downstream data vs OpenVLA.
• GO-1: AgiBot's ViLLA (Vision-Language-Latent-Action) generalist — MoE with a Latent Planner (Chain-of-Planning over latent action tokens from cross-embodiment + human data) feeding an Action Expert; trained on AgiBot World Colosseo (1M+ demos).

17. Evaluation

17.1 Benchmarks

17.2 Real-world success rate

• Per-task success % over K trials (typically 10–50).
• Robust to seed / initial conditions.
• Per-task vs aggregate.
• Generalization: novel object / scene / instruction.

17.3 SimplerEnv (Stanford)

Calibrated sim that closely matches real benchmarks. Cheap iteration on policy without robot time.

17.4 Fine-grained evaluation

• Reach success rate.
• Grasp success rate.
• Sub-task completion.
• Time to complete.
• Recovery from failure.

17.5 Robustness probes

• Lighting variation.
• Distractor objects.
• Novel object instances.
• Instruction paraphrasing.
• Camera perturbations.

17.6 Multi-step / long-horizon

• Composite tasks (cook breakfast = > 10 subtasks).
• Failure recovery: how often policy recovers from sub-failure.
• Time efficiency.

18. Deployment Patterns

18.1 Production deployment topologies

• Cloud-only: high quality, latency penalty.
• Edge-only: low latency, smaller model.
• Hybrid: S2 cloud / S1 edge (Helix-style).
• Per-robot adaptation: LoRA per robot at deploy.

18.2 Safety architectures

• Hard limits: collision avoidance via classical planner overlay.
• E-stop: human / sensor triggered.
• Speed limiter: cap velocity in human spaces.
• Behavior monitoring: detect anomalous actions.
• Force / torque safety: stop on excessive contact.

18.3 Continuous learning loop

• Deploy initial policy.
• Collect telemetry on failures.
• Annotate / re-demo failed scenarios.
• Re-train (LoRA) on augmented data.
• Deploy updated policy.
• Iterate weekly / monthly.

18.4 Multi-tenant fleet patterns

• Per-robot LoRA + shared base.
• Per-task LoRA + shared embodiment.
• Per-deployment LoRA.
• LoRA stacking at runtime.

18.5 Failure-mode logging

• Action / observation logs.
• Task success / failure.
• Time-to-failure.
• Operator interventions.
• Used for offline analysis + retraining.

19. Frontier 2025–2026

19.1 Humanoid foundation models

GR00T / Helix / 1X / π0.5: foundation models specifically for humanoids. Massive investment; several startups + NVIDIA + Tesla.

19.2 Long-horizon manipulation

π0.5: folding laundry, cleaning, multi-minute tasks. Hierarchical S2/S1 architectures emerging.

19.3 Multi-robot coordination

Helix demoed two robots working together. VLA + multi-agent communication / shared planning emerging.

19.4 World-model + VLA hybrids

Train VLA inside Cosmos / GR00T world model. Imagine + execute.

19.5 Dexterous manipulation

Tactile sensing + VLA: in-hand manipulation, fine assembly. Active research; production-grade still rare.

19.6 In-context learning for robots

Few-shot demos for new tasks via prompt engineering / in-context learning. Generalizes ICL paradigm to robots.

19.7 RL on top of VLA at scale

Sample-efficient RL fine-tuning of pretrained VLAs. Surpasses BC ceiling.

19.8 Safety + alignment

• Refusal of dangerous requests.
• Human-in-the-loop confirmation for high-stakes actions.
• Constitutional methods adapted to actions.

19.9 Open research questions

• Cross-embodiment scaling laws.
• Long-horizon memory / planning.
• Multi-modal observation (force, audio, smell).
• Continual / lifelong learning on real robots.
• Sim-to-real for fine motor skill.
• Tactile + VLA integration.

★ 2026 SOTA update — Unified world-model + action VLAs

• WorldVLA: single autoregressive model unifying action generation and image/world prediction; world-model and action-model mutually reinforce; attention-mask strategy suppresses error accumulation in autoregressive action chunks.

20. The 2026 Production Stack

Appendix A: Twenty-Five Things to Know

1. VLA = vision + language + action in single model.
2. Recipe: pretrained VLM + action head + robot data.
3. RT-1 (2022): discrete actions, 35M params, baseline.
4. RT-2 (2023): VLM + action vocab; co-train with web data.
5. RT-X / Open-X-Embodiment: 1.4M trajectories, 22 embodiments.
6. OpenVLA: open RT-2 reference; Llama 2 + DINOv2 + SigLIP.
7. Octo: smaller diffusion-policy open VLA.
8. RDT-1B / RDT-2B: 1B/2B-param diffusion VLA.
9. π0: PaliGemma + flow-matching action head.
10. \(\pi_0\) FM loss: \(\left\| v_\theta - (a_1 - a_0) \right\|^2\).
11. \(\pi_{0.5}\): hierarchical S2/S1 for long-horizon.
12. GR00T: NVIDIA humanoid platform; Cosmos sim integration.
13. Helix: Figure's S2/S1 split for on-board inference.
14. PaLM-E: predates RT-2; embodied multimodal LM.
15. Discrete action tokens: 256 bins per action dim.
16. Diffusion Policy: multi-modal action chunks.
17. Flow matching: π0's choice; few-step inference.
18. ACT: action chunking transformer + CVAE.
19. Temporal ensembling: average overlapping chunks.
20. Spatial VLMs: SpatialVLM / SpatialBot / RoboPoint.
21. Sim-to-real: DR + asymmetric A-C + privileged-to-vision distill.
22. Cross-embodiment via OXE; positive transfer.
23. LoRA per task / per embodiment, shared base.
24. RL fine-tune via Q-chunking / residual / Diffusion DPO / GRPO.
25. Edge: Jetson Thor + distillation + INT8.

Appendix B: Decision Tree — "Which VLA?"

1. Research baseline / academic project? → OpenVLA (open + reproducible).
2. Want diffusion policy specifically? → Octo (small) or RDT-1B/2B (frontier).
3. Production manipulation, commercial? → \(\pi_0\) / \(\pi_{0.5}\) (Physical Intelligence).
4. Humanoid robot, NVIDIA stack? → GR00T N1 / N2 + Cosmos + Isaac Lab.
5. Industrial humanoid, on-board? → Helix (Figure, closed) or build S2/S1 with π0/GR00T.
6. Bimanual table-top demos? → ACT or Diffusion Policy on ALOHA.
7. Need 3D / spatial reasoning? → SpatialVLM fine-tune or 3D-VLA with point-cloud branch.
8. Cross-embodiment scaling? → Open-X-Embodiment + RT-X pattern.
9. RL fine-tuning? → Q-chunking / Residual RL / Diffusion DPO / GRPO on pretrained VLA.
10. Edge / on-robot deployment? → Distill + INT8 + Jetson Thor.

Appendix C: Year-by-Year VLA Milestones

• 2022: RT-1 (Google, 35M); ACT (Stanford); Diffusion Policy (Columbia).
• 2023: RT-2 (Google); PaLM-E (Google); Mobile ALOHA (Stanford); RDT-1B; AnyGrasp.
• 2024 (early): RT-X / Open-X-Embodiment; OpenVLA (Stanford); Octo (Berkeley/Stanford); π0 (Physical Intelligence) released.
• 2024 (late): GR00T announced (NVIDIA); Helix (Figure) demoed; AgiBot World; DexCap; SimplerEnv.
• 2025 (early): GR00T N1 released; π0.5 (long-horizon humanoid); Cosmos World Foundation Models (NVIDIA); GR00T integration with Isaac Lab.
• 2025 (mid): Helix multi-robot coordination demo; 1X World Model + VLA integration; SpatialVLA / RoboPoint mainstream.
• 2025–2026: Production deployment of π0 / π0.5 in customer sites; Figure BMW partnership; Tesla Optimus VLA roll-outs.
• 2026: Humanoid VLA standard; cross-embodiment foundation models; world-model + VLA hybrids; RL fine-tuning routine; Jetson Thor as the standard edge target.