3D / Multi-View Generation Models — All Variants & Tricks

Updated July 2026 with 2025–2026 SOTA additions — new entries marked ★. Algorithm names link to their papers (arXiv / project page).

April 2026 · Version 1.0

Contents

1. Foundations
2. 3D Representations
3. Score Distillation Sampling (SDS)
4. Multi-View Diffusion
5. Multi-View Diffusion + 3D Reconstruction (the 2024 wave)
6. Feed-Forward 3D: LRM Family
7. Native 3D Diffusion (the 2025 wave)
8. Mesh Generation Transformers
9. Feed-Forward Geometry from Images (DUSt3R Family)
10. Single-Image to 3D Pipelines
11. Text-to-3D
12. 3D Gaussian Splatting Generation
13. Avatar Generation (Specialized 3D)
14. Scene-Level 3D Generation
15. 4D / Dynamic 3D Generation
16. Multi-View Consistency Tricks
17. Texture Generation
18. Datasets
19. Evaluation
20. Production Pipelines (2026)
21. Frontier 2025–2026
22. Production Stack 2026 Appendix A: Twenty-Five Things to Know Appendix B: Decision Tree — "Which 3D Generation?" Appendix C: Year-by-Year Milestones

1. Foundations

1.1 The 3D / multi-view generation problem

Goal: from limited 2D supervision (text, image, or few views), produce a 3D representation usable for rendering, editing, and downstream graphics.

1.2 Why it exploded in 2023–26

• 3DGS made 3D rendering real-time + differentiable.
• Internet-scale 2D image diffusion (SD, FLUX) provided strong priors.
• Objaverse / Objaverse-XL provided >10M 3D assets for supervised training.
• Use cases: gaming, AR / VR, e-commerce, robotics, VFX.

1.3 The four paradigms

1. Optimization-based (SDS, VSD): differentiable scene optimized to match a 2D diffusion prior across rendered views. Slow (hours), strong quality.
2. Multi-view diffusion + reconstruction: generate consistent multi-view images, then 3D-reconstruct.
3. Feed-forward (LRM family): single transformer pass image \(\to\) 3D representation. Fast (seconds), needs supervised data.
4. Native 3D diffusion (Trellis, Hunyuan3D-2): diffuse directly in 3D latent space. Fast + high quality.

Dominant 2025–26.

1.4 The 2025–26 winning recipe

Key

Native 3D diffusion (Trellis / Hunyuan3D-2 / CLAY) for asset generation; feed-forward 3D (VGGT / MASt3R) for scene reconstruction; 3DGS as the rendering substrate; SDS as a fallback for unbounded scenes.

1.5 Key trade-offs

2. 3D Representations

2.1 Explicit

• Mesh: vertices + faces; standard graphics asset.
• Voxel grids: dense or sparse octree.
• Point clouds: unordered points; lacks topology.
• Hash grids: multi-resolution hash table (Instant-NGP).

2.2 Implicit

• Signed Distance Function (SDF): \(f(x)\) = distance to surface; surface at \(f = 0\).
• Occupancy field: \(f(x) \in [0, 1]\) = inside probability.
• Neural Radiance Field (NeRF): \(f(x, d) = (c, \sigma)\).

2.3 Hybrid

• Triplane: 3 axis-aligned 2D feature planes.
• TensoRF: tensor decomposition (CP / VM).
• K-Planes / Hexplane: 4D space-time decomposition.
• Hash-grid + MLP: Instant-NGP combo.

2.4 3D Gaussian Splatting (the dominant 2024–26 substrate)

Each Gaussian: \((\mu, R, S, \alpha, \mathrm{SH})\). Real-time differentiable rasterization. Now the default rendering primitive for many generative pipelines.

2.5 Generative-friendly representations

• Triplane: clean to predict via Transformer (LRM).
• Sparse latent grids (Trellis): compact, structured.
• Volumetric latents (CLAY): voxel-style for diffusion.
• Mesh-vertex sequence: for autoregressive (MeshGPT).
• Gaussians: GS-LRM, MeshLRM-style.

2.6 Choosing a representation for generation

• Need editability / topology: mesh / autoregressive mesh.
• Need photorealism: 3DGS / triplane radiance.
• Need physics / simulation: mesh + materials.
• Need fast generation: native 3D diffusion in compact latent.
• Need scene-scale: Gaussians / NeRF.

3. Score Distillation Sampling (SDS)

3.1 DreamFusion (Poole et al. 2022)

The original optimization-based 3D-from-2D method. Differentiable scene \(\theta\) (NeRF or 3DGS), render at view \(v\) \(\to\) image \(x\), pass through frozen 2D diffusion teacher.

3.2 The SDS gradient

Key

\[\nabla_\theta \mathcal{L}_{\mathrm{SDS}} = \mathbb{E}_{t,\epsilon,v}\left[w(t)\big(\epsilon_\phi(x_t,t,c) - \epsilon\big)\,\partial x/\partial\theta\right].\]

Teacher's noise prediction guides \(\theta\) toward consistency with the diffusion prior.

3.3 Limitations

• Slow: hours per scene.
• Janus problem: front-facing view replicated across angles.
• Mode collapse: lacks diversity.
• Saturated colors (high CFG bias).
• Geometry artifacts (floaters, hollow shells).

3.4 Mitigations / variants

• Magic3D (Lin et al. 2022): two-stage — low-res NeRF then high-res mesh + textures.
• Fantasia3D: decouple geometry + appearance.
• ProlificDreamer (VSD): variational SDS; replaces \(\epsilon\) target with learned distribution; reduces collapse.
• Score-Jacobian Chaining: revised gradient computation.
• HiFA, NFSD, CSD: variants tuning timestep schedule.
• Perp-Neg, DreamTime: timestep-aware sampling.

3.5 ProlificDreamer / VSD formula

\[\nabla_\theta \mathcal{L}_{\mathrm{VSD}} = \mathbb{E}\left[w(t)\big(\epsilon_\phi(x_t,t,c) - \epsilon_{\mathrm{lora}}(x_t,t,c,v)\big)\,\partial x/\partial\theta\right],\]

where \(\epsilon_{\mathrm{lora}}\) is a LoRA-fine-tuned diffusion model that captures the current scene's distribution. Reduces the bias of using a single noise sample.

3.6 Sketch / image conditioning

Latent-NeRF, ControlNet-SDS: condition the SDS prior on additional control (text + sketch + image).

Keeps geometry constrained.

3.7 When to still use SDS in 2026

• Unbounded scenes.
• Custom art styles requiring strong 2D-prior bias.
• No suitable native-3D model for the domain.
• Research / one-off content.

4. Multi-View Diffusion

4.1 The premise

Generate multiple consistent views of an object from one input (image or text); then 3D-reconstruct from those views. Sidesteps SDS optimization at inference time.

4.2 MVDream (Shi et al. 2023)

4-view diffusion model trained on Objaverse renders. Generates 4 consistent views jointly via cross-view attention.

4.3 ImageDream

Single-image \(\to\) 4 views. Same architecture conditioned on input image.

4.4 Wonder3D

Joint diffusion of color + normal images at 6 views; recover SDF mesh from the multi-view normals.

4.5 SyncDreamer (Liu et al.)

Diffusion model generates 16 views simultaneously in a single denoising; uses 3D-aware feature volume internally.

4.6 Zero123 / Zero-1-to-3

Zero123: novel-view synthesis from single image conditioned on relative camera pose. Foundation of many follow-ups.

Zero123++: 6-view at once for better consistency.

Stable Zero123: Stability AI's distilled / refined.

Zero123-XL: scaled to Objaverse-XL.

4.7 Era3D

6-view + cross-attention, with explicit camera embedding. Improved consistency.

4.8 SV3D (Stable Video 3D, Stability AI)

Treat orbit around object as a video; use video diffusion (SVD) backbone. Smooth multi-view; better identity preservation.

4.9 V3D, VFusion3D

Video-diffusion-based multi-view generation. Larger temporal context \(\to\) smoother results.

4.10 DreamComposer, Direct2.5

Compositional / fast multi-view variants.

4.11 Cross-view attention pattern

• Concatenate views along batch dim with shared denoising.
• Cross-attention across views at each block.
• Camera-pose conditioning via Plücker rays.
• Reference-image attention for input-view conditioning.

4.12 Reconstruction from multi-view

After multi-view diffusion:

• NeRF or 3DGS optimization (DreamGaussian).
• Direct triplane regression (LRM-style).
• Sparse-view reconstruction (Splatt3R, NoPoSplat).
• Mesh extraction (Wonder3D's normal-based SDF).

★ 2026 SOTA update — Generalist View-Synthesis Diffusion - Stable Virtual Camera (Seva): Stability AI generalist NVS diffusion; any number of input views to arbitrary target cameras with trajectory control, no per-scene 3D.

5. Multi-View Diffusion + 3D Reconstruction (the 2024 wave)

5.1 DreamGaussian (Tang et al. 2023)

1. Generate ~4 views from input.
2. Optimize 3DGS for ~1 minute.
3. Extract mesh + texture refinement.

Took 3D generation from hours (DreamFusion) to ~1 minute. Major impact.

5.2 GaussianDreamer

3DGS-based SDS variant. Faster than NeRF-SDS due to GS rendering speed.

5.3 LucidDreamer

Improved SDS gradient (Interval Score Matching) + 3DGS. Higher fidelity than vanilla SDS.

5.4 LGM (Large Gaussian Model)

Predicts 3D Gaussians directly from multi-view diffusion outputs. Feed-forward; very fast.

5.5 Splatter Image

Per-pixel Gaussian regression from a single image. Direct 3DGS without optimization loop.

5.6 One-2-3-45 / One-2-3-45++

Single-image \(\to\) Zero123 multi-view \(\to\) feed-forward 3D reconstruction. End-to-end pipeline.

5.7 InstantMesh / CRM / TripoSR / SF3D / SPAR3D

Family of fast image-to-3D pipelines combining multi-view diffusion + feed-forward reconstruction. Output mesh + texture in seconds. Common pattern in commercial products.

5.8 Multi-view diffusion + LRM hybrid

1. Image \(\to\) multi-view diffusion (4–6 views).
2. Multi-view \(\to\) feed-forward LRM \(\to\) triplane / Gaussians.
3. Optional: SDS refinement.

The standard 2024 pipeline.

6. Feed-Forward 3D: LRM Family

6.1 LRM (Adobe 2023)

First feed-forward 3D-from-image model:

• ViT encoder over single image.
• Cross-attention from learned triplane queries.
• Output: triplane neural representation.
• Render via volumetric integration.
• Trained on Objaverse + Objaverse-XL.

Triggered the feed-forward 3D wave.

6.2 Instant3D

Two-stage: 4-view diffusion \(\to\) LRM-style triplane regression. Combines multi-view + feed-forward.

6.3 InstantMesh

LRM output \(\to\) mesh extraction via FlexiCubes / marching cubes. Game-ready.

6.4 MeshLRM

Predicts mesh directly (not triplane \(\to\) mesh). Higher topology quality.

6.5 GS-LRM

Predicts 3D Gaussians directly. Faster downstream rendering than triplane.

6.6 Long-LRM

Larger context window; supports more input views. Better quality at scale.

6.7 PF-LRM (pose-free LRM)

Doesn't require known camera poses. Predicts both 3D and the implicit poses. Useful for in-the-wild captures.

6.8 TripoSR (Tripo / Stability)

Open small LRM-class. 0.5s per asset. Strong open frontier (2024).

6.9 CRM (Convolutional Reconstruction Model)

UNet-based instead of transformer. Different inductive bias.

6.10 SF3D, SPAR3D

SF3D: "Stable Fast 3D" (Stability). Very fast, decent quality. SPAR3D: improved architectural choices.

6.11 GeoLRM

Geometry-emphasizing variant; uses normal supervision.

6.12 Comparison

7. Native 3D Diffusion (the 2025 wave)

7.1 The premise

Diffuse directly in a 3D latent space. No SDS, no multi-view intermediate. Fastest at inference; highest quality.

7.2 Trellis (Microsoft 2024)

• Structured 3D latent: sparse-voxel feature grid.
• Two-stage: structure then details.
• Flow-matching loss.
• Single-image / text conditioning.
• Produces 3D Gaussians, mesh, or radiance representation.
• ~5–10 seconds per asset.
• Open weights; commercial-grade quality.

7.3 CLAY

Volumetric latent diffusion. SDF representation; exports to mesh. Strong on hard surface objects.

7.4 Hunyuan3D (Tencent)

Hunyuan3D: native 3D diffusion + texture model.

Hunyuan3D-2: improved version; mesh + texture; strong open frontier.

Hunyuan3D-mini: smaller / faster.

7.5 Direct3D

Direct latent diffusion in 3D. Compact representation; fast.

7.6 3DTopia, 3DTopia-XL

Open native 3D diffusion baselines. Triplane latent.

7.7 SDFusion, Diffusion-SDF

SDF-based diffusion. Voxel-grid SDF latents.

7.8 LASA, GaussianAnything

Various native-3D-diffusion attempts; Gaussian-based.

7.9 Cube (Roblox 2024)

Roblox's open native-3D-diffusion + texture for game assets.

7.10 Rodin Gen-1.5 (Deemos)

Commercial 3D-asset generation product; native 3D diffusion + heavy post-processing.

7.11 Why native 3D wins (in 2025)

• Fast: seconds per asset.
• No multi-view consistency issues (works in 3D).
• No SDS optimization.
• High quality with sufficient 3D training data.
• Texture + geometry jointly possible.

7.12 Common architecture (Trellis-style)

1. Encode 3D asset to compact latent (sparse voxel + features).
2. Train DiT-style diffusion on the latent.
3. Conditioning: image / text via cross-attention.
4. Decode: latent \(\to\) 3D representation (mesh / Gaussians).

★ 2026 SOTA update — Sparse & Scalable Native 3D Diffusion - Direct3D-S2: gigascale sparse-volume 3D DiT with Spatial Sparse Attention (SSA); \(1024^3\) latents, ~3.9x/9.6x fwd/bwd speedup, trainable on 8 GPUs (NeurIPS 2025). - Sparc3D: unified sparse cube representation + topology-preserving remeshing for high-res (\(1024^3\)) shape modeling, plugs into latent diffusion. - Hunyuan3D 2.5: LATTICE 10B-param shape foundation model, sharp geometry with precise image-3D following and clean meshes.

★ 2026 SOTA update — Part-Level & Controllable 3D Diffusion - PartCrafter: compositional latent DiT jointly generating multiple semantic parts from one image, no pre-segmentation; recovers occluded parts (NeurIPS 2025). - Hunyuan3D-Omni: unified controllable generation on Hunyuan3D 2.1, conditions on point clouds, voxels, bounding boxes, and skeletal pose.

8. Mesh Generation Transformers

8.1 The autoregressive mesh paradigm

Tokenize mesh as a sequence (vertex + face indices); train a Transformer to predict the next token. Native mesh topology output.

8.2 PolyGen (Nash et al. 2020)

Pioneering: separate vertex and face Transformers; autoregressive over mesh.

8.3 MeshGPT (Siddiqui et al. 2023)

• VQ-VAE on mesh patches.
• GPT-style autoregressive prediction.
• Generates clean topology (triangle strips).

8.4 MeshXL

Scaled MeshGPT; larger Transformer; improved quality on complex shapes.

8.5 MeshAnything / MeshAnything V2

Image- or point-cloud-conditioned mesh generation. V2 adds adjacent mesh tokens for cleaner topology.

8.6 EdgeRunner

Edge-based tokenization (BPE-like for mesh edges). More efficient than vertex-by-vertex.

8.7 BPT (Bayesian Persistence Tree)

Hierarchical mesh tokenization; better long-range topology.

8.8 LLaMA-Mesh, MeshArt

Various follow-ups; LLM-style architectures.

8.9 Why autoregressive mesh?

• Native mesh output (no isosurface extraction).
• Better topology (no marching-cubes artifacts).
• Edit-friendly.
• Fits game pipelines directly.

8.10 Limitations

• Slow inference (token-by-token).
• Token-vocabulary limits geometric detail.
• Hard surfaces work; organic / complex topology harder.

★ 2026 SOTA update — RL & Tree-Sequenced Mesh Transformers - DeepMesh: autoregressive artist-mesh generation with efficient tokenization + RL/DPO human-preference alignment; point-cloud and image conditioned (ICCV 2025). - TreeMeshGPT: autoregressive tree sequencing over face adjacency, ~22% token compression, up to 11k faces (CVPR 2025).

9. Feed-Forward Geometry from Images (DUSt3R Family)

9.1 The phase transition (2024)

Skip classical SfM/MVS; predict 3D structure directly via Transformer.

9.2 DUSt3R (Naver 2024)

Predict per-pixel 3D pointmaps from 2 images, in camera 1's frame.

\[X^{1,1},\, X^{2,1} \in \mathbb{R}^{H\times W\times 3}.\]

Camera intrinsics, extrinsics, depth, point cloud all decode from the pointmap predictions; pixel matches are nearest neighbors in 3D.

9.3 MASt3R

• DUSt3R + explicit dense matching head.
• Better quality on textureless surfaces.

9.4 MASt3R-SfM

Global SfM via hundreds of pairwise predictions + global optimization. Replaces COLMAP for many use cases.

9.5 Spann3R

Incremental: processes one new view at a time. Online 3D reconstruction.

9.6 Splatt3R

Directly predicts 3D Gaussians from image pairs. End-to-end pose-free 3DGS.

9.7 NoPoSplat

No-pose Gaussians: skips pose estimation; predicts Gaussians + camera jointly.

9.8 Fast3R

Optimized for speed; many-view (50+) support; orders-of-magnitude faster than DUSt3R.

9.9 VGGT (Visual Geometry Grounded Transformer, Meta 2025)

• Large transformer.
• Maps unposed N-image set to depth + cameras + per-pixel 3D in one forward pass.
• Trained on diverse 3D datasets.
• By 2025 the dominant feed-forward 3D model.

9.10 Pi3 (π3, Meta 2025)

Fully feed-forward, pose-free 3D reconstruction. Scaled up.

9.11 StreamingT3R, DepthCrafter, video-DUSt3R

Video-based variants; temporal coherence.

9.12 MoGe / MoGe-2

Metric-scale monocular geometry. Removes affine-ambiguity from classical mono-depth.

9.13 Comparison

★ 2026 SOTA update — Universal Feed-Forward Geometry - MapAnything: single transformer regressing metric multi-view geometry + cameras from images plus optional priors (intrinsics/pose/depth); one backbone for SfM, MVS, depth, localization. - WorldMirror: any-prior-prompted universal 3D world reconstruction, feed-forward from arbitrary combinations of geometric cues.

10. Single-Image to 3D Pipelines

10.1 Pipeline pattern

1. Input: single image.
2. Foreground segmentation (SAM / SAM 2).
3. Multi-view diffusion or native 3D diffusion.
4. Reconstruction (LRM / mesh extraction).
5. Texture refinement.
6. Output: textured mesh / Gaussians.

10.2 Common products

• Tripo (now part of Stability).
• Meshy.
• Rodin Gen-1.5.
• CSM (Common Sense Machines).
• Adobe Substance 3D Sampler.

10.3 Quality vs speed trade-off

• TripoSR: 0.5s, decent quality.
• Trellis: 5–10s, high quality.
• Hunyuan3D-2: 5–10s, high quality + texture.
• SDS-based: hours, sometimes higher quality but Janus risk.

10.4 Failure modes

• Back-face hallucination (model never saw the back).
• Hollow / floater geometry.
• Texture seams.
• Symmetry breaks.
• Topology issues for non-convex shapes.

11. Text-to-3D

11.1 Pipeline pattern

1. Text \(\to\) image (SDXL / FLUX).
2. Image \(\to\) 3D (Trellis / Hunyuan3D-2).

Sometimes text \(\to\) 3D directly via native models.

11.2 Direct text-to-3D models

• DreamFusion (text-only SDS).
• Magic3D (text + 2-stage).
• Trellis (text or image).
• Hunyuan3D-2 (text or image).
• GraphDreamer: scene graph \(\to\) 3D.

11.3 Why text-to-image then image-to-3D often wins

• Text-to-image is a much more mature pipeline.
• Image-to-3D leverages 2D priors.
• Iteration: user can edit image first.
• Reduces ambiguity in text prompt.

11.4 Text + image / sketch combined

ControlNet-style for 3D: provide both text and reference. Better controllability.

12. 3D Gaussian Splatting Generation

12.1 Per-scene optimization (DreamGaussian, etc.)

1. Multi-view generation.
2. Initialize sparse Gaussians.
3. Photometric optimization.
4. Densification.
5. Mesh extraction (optional).

12.2 Feed-forward Gaussian generation (LGM, GS-LRM)

• Input: image(s) or text.
• Direct prediction of N Gaussians.
• No per-scene optimization.
• Seconds per asset.

12.3 Splatter Image

Per-pixel Gaussian: each pixel produces a 3D Gaussian. Simple and effective for object-centric.

12.4 Pixel-aligned Gaussians (NoPoSplat, Splatt3R)

Combine DUSt3R-style geometry with per-pixel Gaussian prediction.

12.5 Streaming Gaussian generation

Add Gaussians as new views become available. Useful for scene scanning.

★ 2026 SOTA update — Pose-Free Gaussian Generation - AnySplat: feed-forward 3DGS from unconstrained, uncalibrated views; one pass yields Gaussians plus per-image intrinsics/extrinsics, sparse-to-dense in seconds.

13. Avatar Generation (Specialized 3D)

13.1 Photoreal facial avatars

• Codec Avatars 2.0/3.0 (Meta).
• Apple Persona.
• Gaussian Avatars / GaussianHead / FlashAvatar.
• Animatable Gaussians, IMavatar, MonoGaussianAvatar.

13.2 Single-image avatar generation

• PortraitGen, Portrait3D.
• Realistic 3D head from a phone photo.
• Driven by audio (EMO / Live Portrait) or pose.

13.3 Body avatars

• GauHuman / 4D-Humans / 4D-DRESS.
• HumanGaussian.
• SMPL-X-conditioned generation.

13.4 Avatar pipeline (2026)

1. Capture / generate face image.
2. Predict SMPL-X / FLAME parameters.
3. Generate 3DGS avatar bound to template.
4. Animate via audio / pose.

13.5 Capture rigs (premium)

ICT Light Stage, Beeble, Disney; full reflectance capture for highest fidelity. Combined with neural representations.

14. Scene-Level 3D Generation

14.1 The challenge

Object-level (Trellis, Hunyuan3D-2) works well; scene-level (kitchens, cities) is harder due to:

• Scale and unbounded extent.
• Many object instances.
• Lighting / shadows.
• Persistent context.

14.2 Scene-level methods

• SceneDreamer: 3D-aware GAN for landscapes.
• LucidDreamer (scene): SDS + camera trajectory for explorable scenes.
• Set-the-Scene: layout + 3D placement + composition.
• GraphDreamer: scene graph \(\to\) 3D objects + relations.
• InfiniCity, CityDreamer: city-scale procedural + neural.
• ZoneDream, Director3D: cinematic scenes from text.
• World Labs: large-world generation (closed previews).

14.3 Scene-from-image / video reconstruction

Use VGGT / MASt3R + 3DGS to reconstruct real scenes. Often combined with native 3D for object replacement / editing.

14.4 Compositional generation

Decompose scene into objects + relations + lighting; generate each independently; compose. More controllable; harder to integrate.

★ 2026 SOTA update — Explorable 3D World Generation - HunyuanWorld 1.0: text/image to immersive explorable 3D worlds via panoramic proxies + semantically layered mesh; exportable, disentangled objects. - Matrix-3D: panorama-based omnidirectional explorable world generation, combining conditional video generation with panoramic 3D reconstruction.

15. 4D / Dynamic 3D Generation

15.1 The 4D problem

Generate time-varying 3D representations: animated objects, dynamic scenes.

15.2 4D-fy

SDS for 4D: extend DreamFusion to time-varying NeRF. Slow but works.

15.3 Dream-in-4D, DreamScene4D

Improvements: better temporal consistency.

15.4 Animate3D

Make a static 3D model dance / move. Conditioned on motion sequences.

15.5 4D Gaussian Splatting generation

Predict time-dependent Gaussian parameters via:

• MLP deformation field.
• Polynomial / RBF temporal bases.
• Hexplane decomposition.
• Per-Gaussian temporal trajectory.

15.6 Video diffusion + 3D lifting

1. Generate a video clip with text-to-video.
2. Lift each frame's 3D via VGGT.
3. Track Gaussians across time.
4. Output 4D representation.

15.7 Genie 2 (DeepMind)

3D playable worlds from a single image; up to 1 minute interactive. Implicitly 4D world model.

15.8 Limitations

• Quality lags object 3D significantly.
• Long temporal coherence open problem.
• Topology changes (cloth, fluids) hard.
• Compute-heavy.

★ 2026 SOTA update — Video-Diffusion 4D Generation - 4Real-Video-V2: fused view-time attention + feed-forward reconstruction for photorealistic 4D scene generation from video diffusion priors. - ShapeGen4D: high-quality 4D (dynamic) shape generation directly from monocular videos.

16. Multi-View Consistency Tricks

16.1 Camera conditioning

• Plücker coordinates per pixel: ray-based conditioning. Standard.
• Camera embedding (token): learnable per-camera token.
• Relative pose: relative-to-input-view.
• Absolute pose: less common.

16.2 Cross-view attention

• Self-attention across views (concatenated).
• Cross-attention from view \(i\) to view \(j\).
• Reference-only attention (only condition on input view).
• Epipolar attention (constrained to epipolar lines).

16.3 3D-aware noise

Initialize noise consistently across views (instead of independent random per view). Reduces inconsistency.

16.4 3D feature volume

Maintain a shared 3D feature volume across views during diffusion. Used in SyncDreamer, MV3D, etc.

16.5 Joint denoising

All views denoised together with a shared diffusion process. Standard in MVDream, ImageDream.

16.6 Reference attention

Inject input image as reference via cross-attention or KV concatenation. Standard for I2V / 3D-from-image.

16.7 ReferenceNet pattern

Trainable copy of U-Net runs on reference; KVs concatenated. Ported from animation to 3D generation.

17. Texture Generation

17.1 The texture problem

After 3D geometry is generated, need to apply high-quality textures. Challenges:

• Multi-view consistency on the surface.
• UV-space rendering issues.
• Fine detail preservation.
• Material parameters (PBR).

17.2 Text2Tex, TexFusion

Iteratively paint textures on a 3D mesh using a 2D diffusion model + view consistency.

17.3 SyncMVD

Synchronized multi-view diffusion specifically for texturing. Joint denoising across views in UV space.

17.4 Paint3D

3D-aware texture diffusion with PBR output (albedo, roughness, metallic).

17.5 TextureAnything

Native 3D texture diffusion. Trained on textured 3D dataset.

17.6 Hunyuan3D-2 texturing

Joint geometry + texture native generation. Strong open frontier.

17.7 Material decomposition

Recover albedo / normal / roughness from generated textures via inverse rendering. See "Delighting & Relighting" cheat sheet.

17.8 Texture refinement

SDS-style refinement at the texture stage to add fine detail post-coarse-generation.

17.9 UV mapping

Auto-UV generation: critical for usability. xatlas, BFF, automatic UV via NeRF-derived parameterization.

18. Datasets

18.1 3D asset datasets

• Objaverse (Allen AI 2022): 800k 3D models from Sketchfab.
• Objaverse-XL: scaled to 10M+ assets.
• ABO (Amazon Berkeley Objects): ~8k product 3D.
• ShapeNet, ShapeNet-Sem: legacy CAD-style.
• Google Scanned Objects (GSO): 1k+ scanned everyday.
• ABC Dataset: ~1M CAD models.
• Thingi10K: 3D printable.
• ModelNet, PartNet: structural.

18.2 Scene datasets

• ScanNet, ScanNet++: indoor 3D.
• Matterport3D, Replica: indoor.
• Hypersim: synthetic indoor.
• 3D-FRONT, 3D-FUTURE: layouts + furniture.
• Habitat-Matterport, Gibson: navigation.
• ETH3D, Tanks and Temples: outdoor.
• nuScenes, KITTI-360, Waymo Open: driving 3D.

18.3 Multi-view image datasets

• CO3D: object-centric multi-view.
• MVImgNet: multi-view images of objects.
• DTU: object MVS.
• NeRF Synthetic + Real360: standard novel-view eval.
• LLFF: forward-facing real scenes.

18.4 Avatar / human

• Sketchfab humans: subset of Objaverse.
• 4D-DRESS, 4D-Humans datasets: full-body 4D.
• HumanML3D, Motion-X: motion + body.
• NeRSemble: light-stage facial.

18.5 Synthetic data (renders)

• Render Objaverse-XL with Blender / Mitsuba for multi-view supervision.
• Random poses + lighting.
• PBR-aware rendering for material learning.

19. Evaluation

19.1 Image quality (per view)

• PSNR, SSIM, LPIPS for novel-view synthesis.
• FID across views for generation diversity.

19.2 Geometry quality

• Chamfer Distance: bidirectional NN distance.
• F-Score: precision/recall at distance threshold.
• Normal consistency: predicted vs GT normals.
• IoU: voxel / SDF.

19.3 Multi-view consistency

• Re-projection error across views.
• Identity-embedding distance across views.
• DUSt3R / VGGT-derived 3D residual.

19.4 T3Bench

Text-to-3D evaluation benchmark; combines quality + alignment metrics.

19.5 GPTEval3D

LLM-based scoring of 3D outputs.

19.6 ULIP-T3D

3D-text alignment via ULIP.

19.7 User studies

Pairwise preferences. Often required for subjective quality. Gold standard for generation.

19.8 Game-engine viability

• Mesh quality (manifold, water-tight, low-poly).
• Texture seam-freeness.
• Topology validity.
• Importable to Unity / Unreal.
• Triangle / vertex count.

20. Production Pipelines (2026)

20.1 Open-source asset gen pipeline

1. Text \(\to\) image (FLUX / SDXL).
2. Background remove (SAM 2).
3. Image \(\to\) 3D (Trellis / Hunyuan3D-2 / TripoSR).
4. Mesh extraction + retopology.
5. Auto-UV.
6. Texture refinement (Paint3D / SyncMVD).
7. Export to OBJ / GLB / USD.

20.2 Commercial 3D-asset products

• Tripo / TripoAI.
• Meshy.
• CSM (Common Sense Machines).
• Rodin Gen-1.5 (Deemos).
• Spline AI.
• Krea 3D.

20.3 Scene reconstruction pipeline

1. Capture: photos / video.
2. Geometry frontend: VGGT / MASt3R-SfM (replaces COLMAP).
3. 3DGS optimization or feed-forward GS prediction.
4. Mesh extraction (SuGaR, 2D-GS) if needed.
5. Compression (LightGaussian).

20.4 AR / VR worldbuilding

• Capture room with phone (Polycam, Luma AI, Scaniverse).
• 3DGS reconstruction.
• Edit / generate additions.
• Compress + stream.
• Render in headset.

20.5 Game asset pipeline

1. Concept image (artist + 2D diffusion).
2. Image \(\to\) 3D (Trellis / Hunyuan3D-2).
3. Retopology to game-ready triangle count.
4. Auto-UV.
5. Bake PBR maps.
6. Engine import (Unreal / Unity).

20.6 VFX / film

• Scan real props / locations (photogrammetry).
• Generate digital doubles (avatars).
• Compose CG into live action with consistent lighting.
• LED volumes (Stagecraft) for in-camera realism.

20.7 E-commerce

• Single product photo \(\to\) 3D model.
• AR try-on / configurator.
• Replaces traditional product photography.

20.8 Robotics

• Generate procedural environments for training.
• Cosmos World Foundation Models (NVIDIA).
• Synthetic objects via Trellis-class for sim.

21. Frontier 2025–2026

21.1 Trends

• Native 3D diffusion (Trellis / Hunyuan3D-2 / CLAY) replaces SDS for production asset generation.
• Feed-forward 3D (VGGT / MASt3R) replaces classical SfM.
• 3DGS is the dominant rendering substrate.
• Mesh transformers (MeshGPT lineage) competitive with native diffusion + extraction.
• Texture + geometry jointly generated.
• 4D generation maturing for short clips.
• Avatars production-ready (Apple Persona, Codec Avatars).

21.2 Open problems

• Long-form 4D coherence.
• Scene-level (rooms, cities).
• Dynamic topology (cloth, fluids).
• Editability of generated 3D.
• Real-time generation (sub-second).
• Texture detail at game-asset quality.
• Multi-object / compositional scenes.
• Physics-aware generation.

21.3 Active research directions

• Hybrid diffusion + autoregressive mesh.
• Native mesh diffusion (skip extraction).
• World Labs-style large world models.
• Generative city-scale (CityDreamer, SceneDreamer follow-ups).
• Editing / inpainting / extending existing 3D.
• Real-time on consumer hardware.

21.4 Closed frontier (April 2026)

• World Labs (Fei-Fei Li): 3D world generation; closed previews.
• Meta (internal): scaled VGGT successors.
• Tencent / Alibaba: continual Hunyuan / Wan 3D releases.
• NVIDIA: Cosmos + Edify 3D.

22. Production Stack 2026

Appendix A: Twenty-Five Things to Know

1. Four paradigms: SDS / multi-view diffusion / feed-forward / native 3D.
2. DreamFusion (2022): SDS gradient \(w(t)(\epsilon_\phi - \epsilon)\,\partial x/\partial\theta\).
3. Janus problem: front-face replication; major SDS issue.
4. ProlificDreamer (VSD): replaces noise target with learned distribution.
5. Magic3D: two-stage NeRF \(\to\) mesh.
6. MVDream / ImageDream: multi-view consistent diffusion.
7. Zero123 / Zero123++: novel-view synthesis from single image.
8. SV3D: video-diffusion approach to multi-view.
9. LRM (Adobe): single image \(\to\) triplane via Transformer.
10. InstantMesh / MeshLRM / GS-LRM / Long-LRM / TripoSR / CRM / SF3D / SPAR3D: LRM family.
11. Native 3D diffusion (2025 wave): Trellis / Hunyuan3D-2 / CLAY / Direct3D.
12. Trellis: structured sparse-voxel latent + flow matching.
13. Hunyuan3D-2: native 3D + texture; open frontier.
14. Mesh autoregressive: MeshGPT / MeshXL / MeshAnything / EdgeRunner / BPT.
15. DUSt3R: pointmap from 2 images (camera 1 frame).
16. MASt3R / MASt3R-SfM: + matching head + global SfM.
17. VGGT (2025): dominant feed-forward 3D.
18. π3: pose-free scaled VGGT successor.
19. Plücker coordinates per pixel for camera conditioning.
20. Cross-view attention + reference attention for multi-view consistency.
21. DreamGaussian: multi-view + 3DGS optimization in ~1 min.
22. LGM / Splatter Image: feed-forward Gaussians.
23. Texture: SyncMVD / Paint3D / Hunyuan3D-2 joint.
24. Objaverse / Objaverse-XL: 800k–10M+ 3D dataset.
25. T3Bench / GPTEval3D / ULIP for 3D-text alignment evaluation.

Appendix B: Decision Tree — "Which 3D Generation?"

1. Single image \(\to\) 3D asset, open-source? \(\to\) Trellis or Hunyuan3D-2.
2. Fastest open-source (< 1s)? \(\to\) TripoSR / SF3D / SPAR3D.
3. Text \(\to\) 3D? \(\to\) FLUX/SDXL \(\to\) Trellis / Hunyuan3D-2 (two-stage).
4. Need clean mesh topology (game asset)? \(\to\) MeshAnything V2 / MeshLRM / EdgeRunner.
5. Multi-view diffusion (4–6 views)? \(\to\) MVDream / Wonder3D / Zero123++ / SV3D.
6. Photogrammetry replacement? \(\to\) VGGT / MASt3R-SfM + 3DGS.
7. Scene scanning (phone)? \(\to\) Polycam / Luma AI + 3DGS.
8. Avatar (face)? \(\to\) Codec Avatars / Gaussian Avatars (premium) or Portrait3D (single image).
9. Custom art style, no 3D dataset? \(\to\) ProlificDreamer (VSD) / Magic3D (SDS-based).
10. Premium / commercial product? \(\to\) Rodin Gen-1.5 / Tripo / Meshy / CSM.
11. 4D / dynamic 3D? \(\to\) 4D-GS optimization / Animate3D / DreamScene4D.
12. Scene-level generation? \(\to\) World Labs (closed) / CityDreamer / Set-the-Scene (research).

Appendix C: Year-by-Year Milestones

• 2020: NeRF; PolyGen (autoregressive mesh).
• 2021: PixelNeRF; Mip-NeRF; Plenoxels; DVGO.
• 2022: DreamFusion (SDS); Magic3D; Latent-NeRF; Instant-NGP; Score-Jacobian Chaining; ScanNet++.
• 2023: 3D Gaussian Splatting (Kerbl SIGGRAPH); ProlificDreamer (VSD); Zero-1-to-3; MVDream; ImageDream; Wonder3D; SyncDreamer; LRM (Adobe); InstantMesh; DreamGaussian; LucidDreamer; Fantasia3D; Objaverse + Objaverse-XL.
• 2024 (early): MeshLRM; GS-LRM; CRM; Long-LRM; PF-LRM; DUSt3R (Naver); MASt3R; SV3D; Splatter Image; LGM; LucidDreamer-Scene; MASt3R-SfM.
• 2024 (mid–late): Trellis (Microsoft); Hunyuan3D / Hunyuan3D-2 (Tencent); CLAY; Direct3D; 3DTopia-XL; SF3D; SPAR3D; MeshGPT; MeshXL; MeshAnything / V2; EdgeRunner; BPT; Spann3R; Splatt3R; NoPoSplat; Fast3R; Cube (Roblox); Rodin Gen-1.5; PixMo / Molmo; Genie 2.
• 2025 (early): VGGT (Meta) — feed-forward 3D dominant; π3 pose-free; LLaMA-Mesh; DepthCrafter; StreamingT3R; MoGe / MoGe-2; texture-aware native 3D models.
• 2025 (mid–late): native 3D diffusion mainstream; Hunyuan3D-2 production deployments; Tripo / Meshy / CSM commercial maturity; Codec Avatars 3.0; World Labs founded.
• 2026: feed-forward 3D + native 3D diffusion replace classical pipelines for asset gen; AR/VR worldbuilding standard via 3DGS; web 3DGS deployment; PBR-aware native generation; physics-aware emerging.