The 2026 reality: end-to-end is the new baseline for academic / startup work. Modular hybrid (with neural perception + classical safety overlay) dominates production at most large players. Neither is purely winning yet.

1.6 Industry split

Vision-only end-to-end: Tesla FSD, Wayve, Comma.ai, Xpeng.
Sensor-rich modular: Waymo, Cruise (paused), Zoox, Pony.ai, WeRide.
Vision + REM map: Mobileye SuperVision / Chauffeur.
Hybrid: most others trying to do both.

2. Sensor Stack

2.1 Camera

Monocular: cheap; weak depth.
Stereo: passive depth from binocular.
Surround (6–12 cameras): full 360 view.
Standard production: 6–12 cameras at 1080p, 30 fps.

2.2 LiDAR

Mechanical (Velodyne-style): 32–128 beams, 360-rotation. Legacy.
Solid-state: no moving parts; more reliable.
4D imaging LiDAR: gives velocity (Doppler).
Range typically 100–250m; cost $$$ to $.

2.3 Radar

3D radar: range + Doppler velocity.
4D imaging radar: + high angular resolution.
Weather-robust (rain / fog).
Lower resolution than LiDAR.

2.4 Ultrasonic / IMU / GNSS

Ultrasonic: parking, very short range.
IMU: ego-motion estimation.
GNSS / RTK: global position; cm-level with RTK.

2.5 HD map (controversial)

Static-info layer: lane geometry, signs, traffic lights.
Pre-built; updated periodically.
Used by Waymo, Cruise, Mobileye REM.
Tesla bet: “mapless”; rely on online perception only.
Trend (2024–26): online HD mapping (MapTR) replacing pre-built.

2.6 Sensor calibration

Intrinsic: per-sensor.
Extrinsic: sensor-to-vehicle (rotation + translation).
Time sync: PTP / GPS time.
Drift: re-calibrate periodically.

2.7 Sensor trade-offs (Tesla vs Waymo)

	Vision-only (Tesla)	Sensor-rich (Waymo)
Hardware cost	low ($100s)	high ($10k+)
Weather robustness	lower	higher
Long-range detection	limited	strong
Scale economics	better	worse
Compute requirement	high (vision is heavy)	moderate
Edge-case generalization	TBD	better in some

3. Camera-Only Perception (BEV)

3.1 The BEV approach

Convert per-camera image features into a top-down “bird’s-eye-view” representation. Standardized output format for downstream prediction / planning.

3.2 LSS (Lift-Splat-Shoot)

First major BEV-from-camera method (Philion & Fidler 2020):

For each pixel, predict categorical depth distribution α ∈∆D−1.
Lift: 2D feature →frustum at multiple depths.
Splat: scatter+sum into BEV cells.
Shoot: BEV-conv head for detections / map.

3.3 BEVFormer (Li et al. 2022)

Cross-attention from BEV queries to multi-camera features: DA(q, p) = X m Wm X k Amk Vm(p + ∆pmk), deformable attention with predicted offsets. Temporal self-attention pulls from BEVt−1. Standard since 2022.

3.4 BEVFormer V2

Adds perspective view auxiliaries; longer temporal context. Stronger.

3.5 BEVDet, BEVDepth

Variants of LSS with improved depth estimation. BEVDepth: explicit depth supervision via LiDAR.

3.6 PETR / PETRv2 / StreamPETR

Position-encoded queries; multi-view 3D detection without explicit BEV. StreamPETR: streaming temporal. State-of-the-art camera-only 3D detection.

3.7 Far3D

Long-range detection (up to 150m) via two-stage refinement.

3.8 SparseBEV

Sparse query-based BEV (one query per object). Cheaper compute, comparable quality to dense BEV.

3.9 Camera-only frontier (2026)

StreamPETR-class for 3D detection.
Combined with online HD mapping (MapTR).
Foundation-model backbones (DINOv3-class).

4. LiDAR Perception

4.1 VoxelNet / SECOND

VoxelNet: 3D voxels + 3D conv encoder.
SECOND: efficient sparse 3D conv. Standard early baseline.

4.2 PointPillars (Lang et al. 2019)

Pillar (vertical column) encoding:

Each point cloud column →pillar (1D feature).
Pillars form 2D pseudo-image.
2D conv for detection.
Fast; ∼60 fps.
Standard production baseline.

4.3 CenterPoint (Yin et al. 2021)

Anchor-free; predict center heatmap of objects + per-center offsets / boxes. Two-stage refinement on second pass. Strong on Waymo / nuScenes.

4.4 PV-RCNN++

Point + voxel hybrid; strong on long-tail.

4.5 TransFusion

Transformer-based; fuses LiDAR + camera. Standard fusion baseline.

4.6 DSVT (Dynamic Sparse Voxel Transformer)

Sliding-window voxel attention; balances dense / sparse.

4.7 SAFDNet

2024 SOTA fully sparse detector. Comparable accuracy at much faster runtime.

4.8 LiDAR-only choice

Production: PointPillars or CenterPoint.
Benchmark-pushing: DSVT / SAFDNet / TransFusion.
Mobile / efficient: PointPillars.

5. Sensor Fusion

5.1 The fusion problem

Combine camera (rich semantics, no depth) + LiDAR (dense depth, sparse semantics) + radar (velocity / weather) into one BEV representation.

5.2 BEVFusion (MIT and PKU variants)

Independently encode each modality to BEV.
Concatenate / feature-fuse.
BEV head for downstream tasks.
Strong production baseline.

5.3 TransFusion (Bai et al.)

LiDAR-first; camera as cross-attention conditioning. Each LiDAR query attends to relevant camera regions.

Token-level fusion; LiDAR + camera tokens jointly attended.

5.5 FUTR3D

3D detection via cross-modal transformer; supports camera + LiDAR + radar.

5.6 SparseBEV-fusion

Sparse query-based BEV fusion; cheap.

5.7 Camera-LiDAR alignment

Critical: precise extrinsic calibration; time synchronization. Calibration drift hurts fusion quality dramatically.

5.8 Late vs early fusion

Early: feature-level (BEVFusion).
Mid: query-level (TransFusion).
Late: detection-level. Simpler; less effective.

5.9 Radar fusion

Active research:

HRFuser, RadarBEV, K-Radar: radar-aware fusion.
Critical for weather robustness.
Less mature than camera-LiDAR fusion.

6. Occupancy Networks

6.1 The occupancy formulation

3D voxel grid V ∈{0, 1}X×Y ×Z + multi-class semantics c ∈{1, . . . , C}. Predict per-voxel occupancy + class. Catches general obstacles regardless of category.

6.2 Why occupancy?

Detection-by-class misses long-tail (debris, weird shapes).
Free-space estimation critical for planning.
Dense surface representation.

6.3 TPVFormer

Tri-plane (XY, YZ, XZ) representation instead of full voxel grid. Compact and effective.

6.4 OccFormer / SurroundOcc / FB-OCC

Variants on dense voxel prediction:

OccFormer: Transformer-based.
SurroundOcc: surround-view input.
FB-OCC: Forward-Backward fusion (2024 SOTA).

6.5 SparseOcc / OPUS

Sparse occupancy: only predict non-empty voxels. ∼100× memory savings. Production-friendly.

6.6 Occupancy loss

Locc = CEvoxel + λg Lov´asz + λa Affinity, where Lov´asz handles class imbalance, affinity preserves local structure.

6.7 COTR, CVT-Occ

2024+ improvements; better long-tail handling.

6.8 Tesla’s occupancy network

Occupancy network is a core component of Tesla FSD perception (revealed at AI Day 2022). General obstacle detection without box-style classification.

6.9 Occupancy + flow

Some methods predict occupancy flow (per-voxel velocity). Useful for motion prediction: OccWorld, FlowOcc3D.

7. Online HD Mapping

7.1 The mapless trend

Replace pre-built HD maps with online perception. Tesla’s bet; increasingly adopted.

7.2 HDMapNet (early 2022)

Predict semantic BEV map (lanes, road boundaries, pedestrian crossings).

7.3 MapTR (Liao 2022)

Predict vectorized polylines for lane / road geometry.
Set-prediction with permutation-invariant Hungarian matching.
Standard for online HD mapping.

7.4 MapTRv2

Improved: hierarchical predictions, PV (perspective-view) auxiliary.

7.5 StreamMapNet

Streaming temporal aggregation; better consistency over time.

7.6 MapEX

Map prior conditioning; combine pre-built + online.

7.7 PivotNet

Pivot-point prediction for cleaner topology.

7.8 VectorMapNet

Earlier vectorized map prediction.

7.9 Online map components

Lane centerlines.
Lane boundaries.
Pedestrian crossings.
Stop lines.
Road boundaries.
Traffic lights / signs (sometimes).

7.10 Why online mapping?

No HD-map vendor dependency.
Robust to map outdating.
Generalizable across cities.
Cheaper deployment.

7.11 HD map vs online: hybrid

Many production stacks use both:

HD map as prior.
Online perception fills gaps / corrects.
Mobileye REM: crowd-sourced HD map.

8. Tracking and Multi-Object Tracking

8.1 Tracking-by-detection

SORT / DeepSORT: simple Kalman + Hungarian.
ByteTrack: low-confidence tracking via two-stage association.
OC-SORT: motion-aware re-identification.
BoT-SORT: + camera motion compensation.

8.2 Transformer trackers

TransTrack, MOTR, MeMOTR: end-to-end track-by-attention. Track queries propagate across frames.

8.3 3D MOT

AB3DMOT (3D Kalman + 3D IoU).
Multi-sensor fusion track.
Joint detection + tracking.

8.4 Streaming temporal aggregation

Cumulative state across past frames. StreamPETR / StreamMapNet pattern.

8.5 Long-tail tracking

Rare object classes (animals, debris) hard. Open-vocabulary tracking emerging (OVTrack).

8.6 Track-then-detect vs detect-then-track

Detect-then-track: classical; per-frame detect + associate. Robust. Track-then-detect: track queries propagate; emergent tracking. End-to-end.

9. Motion Prediction

9.1 The problem

Given current ego-state + tracked objects, predict their next-T trajectory distribution.

9.2 Vector representations

VectorNet: encode all map + agent context as polyline vectors with attention. TNT (Trajectory NN with Targets): target-driven trajectory. MultiPath++: multi-modal trajectory + endpoints.

9.3 Wayformer (Waymo)

Transformer over diverse inputs (map, agents, traffic lights). Multi-axis attention. Strong baseline.

9.4 MTR (Motion TransfoRmer)

Anchor-trajectory queries refined per-step. SOTA on Waymo Open Motion 2023.

9.5 MTR++

Multi-agent joint prediction. Models interactions explicitly.

9.6 QCNet (Query-Centric Network)

Query-centric encoding for invariance to ego frame. Strong generalization.

9.7 SmartRefine

Iterative trajectory refinement.

9.8 Output structure

Predict K trajectory modes with probabilities πk + per-step Gaussians: p(τ|h) = X k πk(h) Y t N(τt; µk,t, Σk,t).

9.9 Loss

Min-of-K: only the best mode trains for displacement; mode classification on closest mode (winner-takes-all). Avoids mode-averaging. L = min k ∥ˆτk −τ ∗∥2 + λc CE(ˆπ, k∗).

9.10 Multi-agent joint prediction

Predict joint trajectory of all agents. Models interactions (e.g., one car yielding to another). MTR++, JFP, M2I.

9.11 Closed-loop integration

Motion prediction →planning input. Reactive planning depends on accurate prediction.

10. Planning and Control

10.1 Classical planning

Search-based: A*, RRT*.
Sampling-based: lattice planner.
Optimization-based: MPC (Model Predictive Control).
Behavior tree / FSM: decision logic.
Rule-based: hand-coded.

10.2 ML-based planning

Imitation learning: BC on expert driving logs.
RL: reward-based; rare in pure form.
End-to-end: pixels →controls.

10.3 Hierarchical planner-controller

Planner: high-level trajectory (route, lane choice).
Controller: low-level steering / brake / throttle.
Standard split since 1990s.

10.4 Control

Lateral: PID / MPC for steering.
Longitudinal: PID / MPC for throttle / brake.
Joint MPC for both.

10.5 Reactive vs deliberative

Reactive: respond to immediate environment.
Deliberative: plan over horizon.
Most stacks combine both.

10.6 Safety overlay

Hard limits: never exceed.
Collision avoidance: independent of main planner.
Emergency braking.
RSS rules (Mobileye).

11. End-to-End Driving

11.1 The premise

Single neural network from sensor input to control output. Joint training on all stages.

11.2 Why end-to-end?

Avoid information loss between stages.
Joint optimization for downstream task.
Simpler engineering / fewer interfaces.
Scaling laws apply.

11.3 UniAD (CVPR 2023 best paper)

Unified architecture with shared BEV features driving:

Detection.
Tracking.
Mapping.
Occupancy.
Motion prediction.
Planning. All trained jointly. Pioneering large-scale end-to-end paper.

11.4 VAD / VADv2* (Vectorized Autonomous Driving)**

Replace dense BEV with vectorized representations throughout. Faster + lighter.

11.5 GenAD

Generative architecture: predict the trajectory distribution explicitly.

11.6 GraphAD

Scene graph for compositional reasoning.

11.7 SparseDrive***

Sparse queries throughout. Production-friendly.

11.8 DriveTransformer ***

Heavy Transformer architecture for end-to-end.

11.9 Hydra-MDP

Multi-Diversity Planning with multiple anchor trajectories. Hedged prediction.

11.10 PARA-Drive***

Parallel architecture for end-to-end.

11.11 Tesla FSD v12+ (2024)

Reportedly drops C++ planner code.
End-to-end neural net (“photons in, controls out”).
Mostly imitation-learned from millions of human driving hours.
Major shift away from modular pipeline.

11.12 End-to-end loss

Le2e = X τ λτLτ + λpLplan, Lplan = ∥ˆτ −τ ∗∥2 + λcCollisionPenalty(ˆτ). Sum of auxiliary task losses + planning loss + safety penalty.

11.13 Scaling story

More driving data →better policy.
Tesla has ∼10B+ miles of fleet driving data.
Wayve trained on ∼5M km.
Data scale matters more than architecture.

11.14 Limitations

Hard to debug.
Long-tail edge cases.
Safety verification difficult.
Distribution shift.
Rare-event handling.

12. VLM-Based Reasoning Driving

12.1 The premise

Use a VLM to reason about complex driving scenarios. Combine perception + language reasoning.

12.2 Wayve LINGO-1 / LINGO-2

LINGO-1: VLM commentary on driving scenes.
LINGO-2: integrates VLM with planning; can explain its decisions in natural language.
Wayve’s bet on language-as-supervision.

12.3 DriveGPT4 / DriveLM / DriveVLM

VLMs fine-tuned on driving Q&A. Produce reasoning + plans:

“Why are you slowing down?” →“A pedestrian is crossing.”
“What should I do at the intersection?” →structured plan.

12.4 LMDrive (closed-loop)

LM in closed-loop driving. Conditional generation of trajectories.

12.5 Senna (Wayve)

VLM as high-level planner + low-level control net.

12.6 Reason2Drive

Causal-reasoning benchmark + dataset.

12.7 DriveMM, DriveDPO

Various VLM-based driving frameworks.

12.8 The two-network split (Helix-style)

S2 (slow VLM): high-level reasoning, long-term plan.
S1 (fast policy): real-time low-level control.
Mirrors humanoid robotics architecture.

12.9 Benefits

Explainability.
Long-tail reasoning.
Following complex instructions.
Better generalization.

12.10 Challenges

Latency (VLMs are slow).
Hallucination in safety-critical.
Compute on-vehicle.
Verification.

13. World Models for AV

13.1 Why world models?

Closed-loop simulation: re-render real driving log under perturbed trajectories. Train policies in counterfactual scenarios. Plan over imagined futures.

13.2 GAIA-1 / GAIA-2 (Wayve)

Driving-specific video diffusion.
Conditioned on past frames + actions (steering / throttle / brake).
GAIA-2: scaled, multi-modal control.
Used for closed-loop evaluation.

13.3 DriveDreamer / DriveDreamer-2

Open AV world models. Text + action-controlled scenario generation.

13.4 Vista

Driving / scene-level video synthesis with controllability.

13.5 MagicDrive / MagicDrive3D

3D-aware video synthesis for AV.

13.6 Cosmos (NVIDIA)

General world-model platform; positioned as substrate for AV + robotics. Open weights for some variants.

13.7 WoVoGen

World-volumetric generation. Voxel-grid-style world model.

13.8 DriveWorld / OccWorld

Occupancy-grid-based world models. Predict future occupancy.

13.9 Closed-loop AV simulation pipeline

1. Capture real driving log.

2. Reconstruct scene (NeuRAD / EmerNeRF / StreetGaussians).

3. Replay with counterfactual actions.

4. Render via world model (or neural simulator).

5. Evaluate policy.

13.10 Long-tail edge case generation

World models can:

Generate rare scenarios.
Augment training data.
Test policy safety.

14. Closed-Loop Simulation

14.1 Why closed-loop?

Open-loop evaluation (replay logs) doesn’t capture how policy interacts with environment. Closed-loop is essential for true policy testing.

14.2 Classical simulators

CARLA: open; widely used. (project)
LGSVL / SVL Simulator: enterprise; deprecated.
IsaacSim: NVIDIA’s general; AV-extended.
Apollo / Baidu Apollo: open for Chinese ecosystem.
Waymax (Waymo): synthetic motion simulation.

14.3 Benchmarks

Bench2Drive: end-to-end evaluation.
NAVSIM: standardized AV benchmark.
DriveArena: holistic.
nuPlan: Motional’s planning benchmark.

14.4 Neural simulators (the new wave)

NeuRAD: NeRF-based AV simulation.
UniSim: NVIDIA’s AV-specific.
StreetGaussians: 3DGS for streets.
EmerNeRF: dynamic + static decomposition. (project)
DrivingGaussian: 3DGS for AV.
OmniRe: open-source comprehensive.
S-NeRF: scalable scene NeRF for AV.

14.5 Hybrid: physics + neural

Physics (CARLA / Isaac) for dynamics.
Neural (Cosmos / Sora-class) for visuals.
Best of both.

14.6 Cosmos Predict (NVIDIA)

Diffusion + autoregressive variants. Action-conditioned video. Closed-loop ready.

14.7 Sim-to-real transfer

Domain randomization.
Photorealistic simulation.
Real-data co-training.
Adversarial perturbations.

14.8 Production simulator stacks

Waymo: internal Waymax + neural simulators.
Cruise: internal physics + neural hybrid.
Tesla: massive shadow-mode fleet (real-world simulator).
Wayve: GAIA + on-road testing.
NVIDIA: Cosmos + Isaac as platform.

15. Auto-Labeling and Data Engines

15.1 The data problem

AV needs orders of magnitude more data than typical CV:

Long-tail edge cases.
Diverse weather / lighting / regions.
3D ground truth expensive.
Manual labeling cost-prohibitive.

15.2 Tesla’s data engine

1. Fleet captures driving.

2. Rare-event triggers (intervention, near-miss, model uncertainty).

3. Rare events uploaded.

4. Auto-labeled by larger offline ensemble.

5. Used to train production model.

6. Iterate weekly / monthly.

15.3 Long-tail mining

High-loss frames in production logs.
Rare class detections.
Disengagement events.
Weather / scenario tags.

15.4 Auto-labeling

Larger / slower offline model labels frames; production model trained on labels.

4D auto-labeling: track objects across time + sensors.
Multi-sensor cross-checking (LiDAR sanity-checks camera).
HD-map + tracker disagreement to find errors.

15.5 Active learning

Rank candidate frames by:

Model uncertainty.
Ensemble disagreement.
Class rarity.
Geographic / weather coverage.

15.6 Programmatic supervision

HD-map overlap to detect errors.
Multi-modal cross-checking.
Snorkel-style label functions.

15.7 Synthetic data

Rare scenarios (CARLA).
Diffusion-augmented.
Cosmos-generated counterfactuals.
Sim-to-real bridging.

15.8 Industry labeling vendors

Scale AI, Surge, Mercor, internal at Waymo / Cruise. Combination of human + auto.

16. Neural Reconstruction for AV

16.1 Why neural reconstruction?

Reconstruct real driving logs as differentiable scenes. Re-render under perturbations. Free closed-loop testing.

16.2 Block-NeRF (Tancik et al.)

Decompose city-scale scene into spatial blocks; train per-block NeRF; composite. SF / Mission Bay results.

16.3 Mega-NeRF

Multi-NeRF for large scenes; spatial partitioning + visibility prediction.

16.4 StreetGaussians

3DGS for streets. Dynamic vehicle handling via separate Gaussians per object.

16.5 EmerNeRF

Decompose static / dynamic; emerge object motion from supervision.

16.6 S-NeRF

Scalable per-scene NeRF for AV.

16.7 DrivingGaussian

3DGS for driving with multi-camera fusion.

16.8 OmniRe

Open-source comprehensive AV scene reconstruction.

16.9 NeuRAD

NeRF-based AV reconstruction with sensor fusion.

16.10 Production application

Reconstruct real log.
Edit ego trajectory (counterfactual).
Edit other agents (pedestrian appears).
Re-render full scene.
Test policy in counterfactual.

16.11 Combined with world models

Cosmos / GAIA + neural reconstruction = full closed-loop simulator.

17. Foundation Models for AV

17.1 Vision foundation models

DINOv3 features: as backbone for AV perception.
SigLIP: zero-shot class detection.
SAM 2: open-vocabulary segmentation.
Grounding DINO 1.6: open-set 3D detection.

17.2 VLM grounding for AV

DriveVLM, Senna: VLM grounded in driving.
LINGO family: explanation + planning.
DriveLM, DriveGPT4: Q&A driving.

17.3 Cosmos World Foundation Models (NVIDIA)

Cosmos Predict (Diffusion + AR).
Cosmos Transfer (sim-to-real).
Cosmos Reason (physical commonsense).
Cosmos Tokenizer. Positioned as platform for AV simulation + perception.

17.4 Pretrained encoders for AV

Use frozen DINOv3 / SigLIP as feature extractor; train AV-specific head. Saves data + compute.

Native multimodal (Gemini-style) for end-to-end.
Pretrain on internet driving + general data.
Fine-tune for AV.

17.6 Trends 2025–26

Foundation-model backbones standard for new AV systems.
VLM reasoning integrated for explainability.
World models as central simulation substrate.

18. Safety and Verification

18.1 Safety frameworks

ISO 26262: functional safety standard.
ISO/PAS 21448 (SOTIF): Safety of the Intended Functionality.
UL 4600: AV safety case.
ISO 21434: cybersecurity.

18.2 RSS (Responsibility-Sensitive Safety, Mobileye)

Formal mathematical model of safe driving:

Per-agent safety distance.
Pre-defined safe responses to dangerous situations.
Deterministic safety guarantees on top of ML.

18.3 Statistical safety case (Waymo’s approach)

Million-mile testing in-domain.
Disengagement rate analysis.
Per-scenario failure analysis.
Public safety reports.

18.4 Adversarial testing

Stress-testing rare events.
Adversarial perturbations.
Edge-case generation via simulators.
Red-team scenarios.

18.5 Driver state, attentiveness

Driver-monitoring (DMS): cameras observe driver.
Hands-on / hands-off detection.
Eyes-on-road detection.
Required by EU regulation, increasingly elsewhere.

18.6 Disengagement metrics

Miles per disengagement (CA reports).
Per-scenario disengagement type.
Weather / time-of-day breakdowns.

18.7 Long-tail coverage

Long-tail rare events the bottleneck for L4+.
Most “solved” for normal driving; edge cases dominate.
Synthetic / generative data for rare scenarios.

18.8 Formal verification

Formal methods for ML still emerging.
Differential testing across model versions.
Reachability analysis on classical components.

19. On-Vehicle Compute

19.1 Latency budgets

Perception: ∼50–100 ms.
Prediction: ∼50 ms.
Planning: ∼50 ms.
Total round-trip: ∼200 ms target.
Lower for emergency response.

19.2 NVIDIA AV hardware

Tegra Xavier: legacy.
Orin: 254 TOPS; current Tesla / Mercedes.
Thor (B100-based): 2000 TOPS; next-gen.

19.3 Tesla AV chip

HW3 (FSD chip): 2019; 144 TOPS.
HW4: 2023; ∼2× HW3.
AI5 (next): rumored 2025.
Custom in-house design.

19.4 Mobileye EyeQ

EyeQ5: 24 TOPS, deployed.
EyeQ6: 80+ TOPS, current.
EyeQ Ultra: production target for SuperVision.

19.5 Other ASIC players

Qualcomm Snapdragon Ride, Black Sesame, Horizon Robotics, NVIDIA Drive Concierge.

19.6 Optimization for vehicles

Quantization (INT8 / FP8).
Pruning + sparsity.
Distillation (teacher-student).
TensorRT compilation.
Multi-stream concurrency.

19.7 Energy / cooling

On-vehicle compute power draw: ∼100–500 W. Range impact + heat dissipation matters. Drives chip selection.

20. Industry Players and Approaches

20.1 Tesla

FSD (Full Self-Driving): vision-only end-to-end (since v12).
HW4 chip; Dojo training cluster.
10B+ miles fleet data.
Mapless approach.
L2+ supervised; pushing to L4.

20.2 Waymo (Alphabet)

Sensor-rich (LiDAR + cam + radar).
Modular pipeline + ML.
L4 robotaxi in Phoenix, SF, LA, Austin.
Heaviest engineering safety case.
HD maps + online perception.

20.3 Wayve

Vision-only end-to-end.
GAIA world model.
LINGO VLM reasoning.
Partner with Uber, OEMs.

20.4 Mobileye

Vision + REM crowd-sourced HD map.
EyeQ chip portfolio.
Tier-1 supplier; deployed in millions of vehicles.
SuperVision (L2+) and Chauffeur (L3).
RSS safety framework.

20.5 Cruise (paused)

Multi-sensor stack; SF robotaxi paused after 2023 incident; rebuilding under GM.

20.6 Zoox (Amazon)

Multi-sensor; purpose-built robotaxi pod; SF deployment.

20.7 Aurora

Trucking + ride-hailing focus. Sensor-rich.

20.8 Pony.ai, WeRide

Chinese players; robotaxi in Beijing, Guangzhou, Shenzhen.

20.9 Xpeng, Li Auto, Nio, BYD

Chinese OEMs with proprietary AV stacks. Aggressive end-to-end deployment.

20.10 Huawei AD/ADS

Provides AV stack to Chinese OEMs.

20.11 Mercedes Drive Pilot, Honda Sensing Elite

L3 systems certified in some regions.

20.12 Comma.ai

Open-source-ish; OpenPilot driver assistance for retrofit.

20.13 NVIDIA Drive

Platform for OEMs (chips + software). Used by Mercedes, Volvo, others.

21. Datasets

21.1 Public AV datasets

Dataset	Size	Notes
KITTI (2012)	22 sequences	legacy; small
KITTI-360 (2020)	320 km	360-camera + LiDAR
nuScenes (2019)	1000 scenes	6 cam + LiDAR + radar
Waymo Open (2019)	2000+ scenes	5 LiDAR + 5 camera
Argoverse 1/2 (2019/2022)	varies	motion forecasting
Lyft Level 5 (2020)	24k frames	motion + maps
ApolloScape	varies	Chinese roads
Cityscapes (2016)	5000 imgs	semantic seg
BDD100K (2018)	100k videos	diverse
Mapillary Vistas	25k	street-level
ZOD (Zenseact 2023)	varies	Sweden
DriveLM, DriveVLM-data	—	language QA for VLM

21.2 Synthetic datasets

CARLA: full simulator; varies.
Synscapes: synthetic Cityscapes.
GTA-V renders: classic.
TartanAir-driving: synthetic.

21.3 Motion forecasting

Argoverse 2 motion.
Waymo Open Motion.
nuPlan.
Lyft Level 5 motion.

21.4 Closed-loop benchmarks

Bench2Drive.
NAVSIM.
DriveArena.
nuPlan.

21.5 HD map data

HERE.
TomTom.
Google internal.
Mobileye REM (crowd-sourced).
OpenStreetMap (OSM).

22. Training Infrastructure for AV

22.1 Tesla Dojo

Tesla’s custom training cluster. D1 chip. 10× scaling roadmap. Trains FSD on 100k+ H100-equivalent.

22.2 Cloud training

Waymo: Google TPU.
Wayve: Microsoft Azure (NVIDIA H100).
Cruise: AWS.
NVIDIA: in-house DGX.

22.3 Distillation pipelines

Train large offline model on full data.
Auto-label rare events.
Distill into deployable model.
Iterate.

22.4 Continuous training

New data daily / weekly.
Re-train on rare events.
Canary deployment.
Gradual rollout.

22.5 A/B testing

Shadow mode (new model decisions logged but not executed).
Per-scenario performance comparison.
Disengagement / intervention metrics.

22.6 Closed-loop training

Reinforcement learning with simulator.
Combined with imitation.
Wayve, Cosmos, Carla all enable.

23. Recent Frontier (2025–2026)

23.1 End-to-end is mainstream

Tesla FSD v12+ proved end-to-end works. Wayve / Xpeng / Li Auto follow. Modular stacks under pressure.

23.2 VLM reasoning integration

DriveVLM / Senna / LINGO-2 demonstrate VLM-based explanation + planning. Latency vs reasoning trade-off improving.

23.3 World models for sim

GAIA-2, Cosmos, NeuRAD: closed-loop neural simulation. Replaces some real-world testing.

23.4 Online HD mapping mainstream

MapTRv2 / StreamMapNet: production-grade. Reduces dependency on pre-built maps.

23.5 Foundation-model perception

DINOv3 / SigLIP / SAM 2 as backbones. Less custom architecture; more data + scale.

23.6 NVIDIA Cosmos as platform

Cosmos Predict / Transfer / Reason for AV simulation + perception. Open weights.

23.7 Robotaxi expansion

Waymo: 200k+ rides per week (LA, SF, Phoenix, Austin).
Pony.ai, WeRide: Chinese cities.
Tesla Cybercab announced.
Robotaxi era beginning.

23.8 Open research

Long-tail edge cases.
Safety verification of E2E.
Multi-agent prediction at scale.
Sim-to-real for vision.
World models with physics.

24. Production Stack 2026

Stage	Default approach	Notes
3D detection (camera)	StreamPETR / SparseBEV	state-of-the-art
3D detection (LiDAR)	PointPillars (prod) or DSVT / SAFDNet	SOTA vs cheap
Sensor fusion (BEV)	BEVFusion / TransFusion / CMT	camera + LiDAR
Occupancy	FB-OCC / SparseOcc / OPUS	general obstacles
Online HD mapping	MapTRv2 / StreamMapNet	vectorized
Tracking	ByteTrack / OC-SORT / TransTrack	+ 3D variants
Motion prediction	Wayformer / MTR++ / QCNet	multi-modal
End-to-end driving	UniAD / VAD / Hydra-MDP / Tesla FSD	joint train
VLM-based reasoning	Wayve LINGO-2 / DriveVLM / Senna	explainable
World model (closed sim)	GAIA-2 / Cosmos Predict	action-conditioned
Neural reconstruction	StreetGaussians / EmerNeRF / OmniRe	3DGS
Sim platform	Waymax / NAVSIM / Bench2Drive / CARLA	open
Open-source backbone	DINOv3 / SigLIP	frozen features
On-vehicle compute	NVIDIA Thor / Tesla AI5 / Mobileye EyeQ Ultra	1000+ TOPS
Long-tail mining	active learning + auto-label + data engine	Tesla-style continuous

Appendix A: Twenty-Five Things to Know

1. Levels of autonomy: L0 (none) to L5 (full).

2. Sensor stack: camera + LiDAR + radar + IMU + GNSS.

3. BEV (Bird’s-Eye-View): unified top-down representation.

4. LSS / BEVFormer / PETR for camera-only 3D detection.

5. PointPillars / CenterPoint / SAFDNet for LiDAR.

6. BEVFusion / TransFusion for sensor fusion.

7. Occupancy networks (FB-OCC / SparseOcc) for general obstacles.

8. MapTR / MapTRv2 for online HD mapping.

9. Tracking-by-detection (ByteTrack) + Transformer trackers (TransTrack).

10. Motion prediction: Wayformer / MTR++ / QCNet.

11. Min-of-K loss for multi-modal trajectory prediction.

12. UniAD (CVPR 2023) pioneered end-to-end joint training.

13. VAD / Hydra-MDP / SparseDrive: end-to-end variants.

14. Tesla FSD v12+ is end-to-end neural, mostly imitation-learned.

15. Wayve LINGO-2 / DriveVLM / Senna: VLM-based driving.

16. GAIA-1/2 / Cosmos / DriveDreamer: AV world models.

17. StreetGaussians / EmerNeRF / OmniRe: neural AV simulators.

18. RSS (Mobileye): formal safety framework.

19. Tesla data engine pattern: rare-event mining + auto-label.

20. NVIDIA Thor 2000 TOPS / Tesla HW4 / Mobileye EyeQ for on-vehicle.

21. Waymo: sensor-rich modular; Tesla: vision-only end-to-end.

22. Tesla mapless vs Waymo HD-map: opposing bets.

23. Closed-loop sim (GAIA + Cosmos + Waymax): replaces some real testing.

24. Foundation model backbones (DINOv3 / SigLIP) increasingly in AV.

25. Long-tail edge cases dominate L4 deployment effort.

Appendix B: Decision Tree — “Which AV Component?”

1. Camera-only 3D detection? →StreamPETR or SparseBEV.

2. LiDAR-only 3D detection? →PointPillars (production) or DSVT/SAFDNet (SOTA).

3. Sensor fusion? →BEVFusion / TransFusion / CMT.

4. Free-space + general obstacles? →Occupancy network (FB-OCC / SparseOcc).

5. Online HD mapping? →MapTRv2 / StreamMapNet.

6. Multi-object tracking? →ByteTrack / OC-SORT / TransTrack.

7. Motion prediction? →Wayformer / MTR++ / QCNet.

8. End-to-end driving (research)? →UniAD / VAD / Hydra-MDP.

9. VLM-based reasoning? →Wayve LINGO-2 / DriveVLM / Senna.

10. World model for closed-loop? →GAIA-2 / Cosmos Predict.

11. Neural reconstruction (sim re-render)? →StreetGaussians / EmerNeRF / OmniRe.

12. Closed-loop benchmark? →Bench2Drive / NAVSIM / Waymax.

13. On-vehicle compute platform? →NVIDIA Thor or Tesla AI5 or Mobileye EyeQ Ultra.

Appendix C: Year-by-Year AV Milestones

2009: Google self-driving project starts (now Waymo).
2014: Mobileye SDK in Tesla; AP1 launches.
2015: NVIDIA Drive PX1; Tesla Autopilot v7 with vision-driven driving.
2017: Waymo Phoenix early rider program; DeepLearning + LiDAR mainstream.
2018: PointPillars; Apollo open source; Cruise / Zoox milestones.
2019: nuScenes dataset; CenterPoint; PETR-precursor work; Tesla HW3 chip.
2020: LSS (Lift-Splat-Shoot); Waymo Open Dataset; Mobileye REM at scale.
2021: BEVFormer; CenterPoint mainstream; Tesla AI Day reveals occupancy networks.
2022: BEVFormer / BEVDet / PETR; PointPillars production; HD-map decline begins; UniAD prototype.
2023: UniAD (CVPR best paper); VAD; Wayve GAIA-1; Tesla FSD v11; nuPlan benchmark; SparseBEV / SparseDrive.
2024: Tesla FSD v12 end-to-end (major shift); Wayve LINGO-2; GAIA-2; Cosmos World Foundation Models; FB-OCC SOTA; MapTRv2; Hydra-MDP; OmniRe; StreetGaussians; DriveVLM; Senna.
2025: end-to-end mainstream; VLM reasoning integration; Cosmos as AV platform; Waymo robotaxi 200k+ rides/week; Tesla Cybercab announced; Mobileye SuperVision in production; Chinese OEMs catch up rapidly.
2026: foundation-model backbones standard; world-model closed-loop sim production-grade; long-tail edge cases the central frontier; L4 robotaxi expansion across cities; safety verification still hardest open problem.

Converted from 3D-Autonomous_Driving_All_Variants_with_Links.md3D-Autonomous_Driving_All_Variants_with_Links.md

1. Foundations: The AV Stack

1.1 Levels of autonomy (SAE)

1.2 The classical AV stack

1.3 The 2024–26 trend: end-to-end

1.4 Three architectural approaches

1.5 Hybrid is dominant