Structure from Motion — Technologies & Tricks

Updated July 2026 with 2025–2026 SOTA additions — new entries marked ★. Algorithm names link to their papers (arXiv / project page).

April 2026 · Version 1.0

Contents

1. Foundations: What Is SfM?
2. Multi-View Geometry Foundations
3. Feature Detection
4. Feature Matching
5. RANSAC Family
6. Bundle Adjustment
7. COLMAP: The De Facto Pipeline
8. Global SfM
9. Multi-View Stereo (MVS)
10. Feed-Forward 3D (the 2024–26 Wave)
11. Visual SLAM
12. Camera Models and Calibration
13. Loop Closure and Place Recognition
14. Tools and Software
15. Datasets and Benchmarks
16. Production Pipelines
17. Common Failure Modes
18. Theoretical Foundations
19. Frontier 2025–2026
20. Production Stack 2026

• Appendix A: Twenty-Five Things to Know
• Appendix B: Decision Tree — "Which SfM?"
• Appendix C: Year-by-Year SfM Milestones

1. Foundations: What Is SfM?

1.1 Working definition

Given a set of images of a scene from different viewpoints, recover:

• Camera poses: position \(t \in \mathbb{R}^3\) and orientation \(R \in SO(3)\) per image.
• Camera intrinsics: focal length, principal point, distortion (often).
• Sparse 3D structure: point cloud of scene features.

1.2 Why SfM?

• Photogrammetry / 3D reconstruction.
• AR localization (visual SLAM is online SfM).
• Autonomous driving HD maps.
• VFX camera tracking.
• Robotics navigation (SLAM-flavored).
• Initialization for downstream 3DGS / NeRF.

1.3 The classical SfM pipeline

1. Feature detection per image.
2. Feature matching across image pairs.
3. Geometric verification.
4. Initial pair selection / two-view init.
5. Incremental: register new images via PnP.
6. Triangulate new 3D points.
7. Bundle adjustment.
8. (Optional) loop closure / global refinement.

1.4 The 2024–2026 transformation

Key

Feed-forward 3D models (DUSt3R, MASt3R, VGGT) are eclipsing classical SfM. Single transformer pass from N images \(\to\) depth + cameras + 3D point cloud. By 2025–26, COLMAP is becoming a fallback / refinement step, not the default.

1.5 The two paradigms today

1. Classical: COLMAP / Theia / OpenMVG. Mature, accurate, slow, fragile on textureless / repetitive.
2. Feed-forward: DUSt3R / MASt3R / VGGT. Fast, robust, requires training data, less interpretable.

1.6 Inputs / outputs

1.7 Use cases by paradigm

• COLMAP: precision-critical, well-textured, professional 3D scanning.
• VGGT / MASt3R: phone capture, in-the-wild, fast, AR/VR / 3DGS init.
• Visual SLAM: real-time on robot / vehicle.
• Hybrid: VGGT for init, COLMAP refine.

2. Multi-View Geometry Foundations

2.1 Pinhole camera model

World point \(X \in \mathbb{R}^3\) projects to pixel \(x\) via:

\[\lambda \begin{pmatrix} u \\ v \\ 1 \end{pmatrix} = K\,[R \mid t] \begin{pmatrix} X \\ 1 \end{pmatrix},\]

where intrinsic matrix \(K = \begin{pmatrix} f_x & s & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{pmatrix}\), 5 DoF (4 if \(s = 0\)).

2.2 Distortion

Brown-Conrady (radial-tangential):

\[x_d = x\,(1 + k_1 r^2 + k_2 r^4 + k_3 r^6) + 2 p_1 x y + p_2 (r^2 + 2 x^2), \quad r^2 = x^2 + y^2.\]

Kannala-Brandt (fisheye): polynomial in \(\theta = \arctan(r)\).

2.3 Calibration (Zhang's method)

• Image a planar checkerboard from \(\ge 3\) poses.
• Each plane gives homography \(H = K[r_1, r_2, t]\).
• Constraints from \(r_1, r_2\) orthonormal yield Cholesky-recoverable \(B = K^{-\top} K^{-1}\).
• Refine via non-linear minimization.

2.4 Epipolar geometry

For two cameras with relative pose \((R, t)\):

\[\mathbf{x}_2^{\top} E\, \mathbf{x}_1 = 0, \quad E = [t]_\times R.\]

• \(E\): essential matrix (calibrated). 5 DoF.
• \(F\): fundamental matrix (uncalibrated): \(F = K_2^{-\top} E K_1^{-1}\). 7 DoF.
• Rank-2 constraint: \(\det F = 0\).

2.5 Eight-point algorithm

For \(\ge 8\) correspondences: linear system \(A f = 0\). Solve via SVD; enforce rank-2 by zeroing smallest singular value.

Watch out

Hartley normalization essential: normalize image coords (centroid at origin, average distance \(\sqrt{2}\)). Without it, 8-point is numerically catastrophic.

2.6 Five-point algorithm (Nistér 2004)

For calibrated case, 5 correspondences suffice.

Up to 10 candidate solutions via Gröbner basis or Macaulay matrix. Standard for two-view init in modern pipelines.

2.7 Triangulation

Given correspondence \((x_1, x_2)\) and projection matrices \(P_1, P_2\):

• DLT (linear): stack \(A = [\,u_1 \mathbf{p}_3^{\top} - \mathbf{p}_1^{\top};\ \dots\,]\) rows; solve via SVD.
• Mid-point: midpoint of closest segment between two rays.
• Optimal (Hartley-Sturm): minimize geometric reprojection in both images via 6th-order polynomial.

2.8 PnP (Perspective-n-Point)

Given 3D points \(X_i\) and 2D observations \(x_i\) + intrinsics, recover \(R, t\).

• P3P (Gao): minimum case, 4 solutions; disambiguate with 4th point.
• EPnP: \(O(n)\) closed form via control points.
• Iterative refinement via Gauss-Newton on reprojection error.

3. Feature Detection

3.1 Classical features

• Harris corners (1988): classical corner detector via second-moment matrix.
• SIFT (Lowe 2004): scale-invariant; DoG pyramid; gradient histogram. The gold-standard classical descriptor.
• SURF: faster SIFT alternative.
• ORB (Rublee 2011): FAST corners + BRIEF binary descriptor; very fast.
• AKAZE: nonlinear scale space; better than ORB, slower.
• FAST + BRIEF: lightest combination; binary.

3.2 Modern learned features

• SuperPoint (DeTone 2018): jointly learn detector + descriptor; trained on synthetic shapes + real images.
• R2D2 (Revaud 2019): separately learn reliability + repeatability.
• DISK (Tyszkiewicz 2020): discrete attention with reinforcement learning.
• ALIKED (Zhao 2023): efficient + sparse; production-grade.
• XFeat (Potje 2024): lightweight; mobile-friendly; SOTA at low compute.

3.3 SIFT vs modern learned

• SIFT: zero-shot, no training; surprisingly competitive in 2026 on many benchmarks.
• Learned: better on extreme conditions (illumination, weather).
• Production often combines: SIFT for general + learned for hard.

3.4 Detector vs descriptor

• Detector: where to extract features (corners, blobs).
• Descriptor: how to describe a patch (SIFT 128-d, ORB 256-bit).
• Modern learned methods often joint detector + descriptor.

3.5 Scale-invariance and rotation

• SIFT: scale-space DoG; orientation assignment via gradient histogram.
• Learned: usually relies on training-data augmentations.

3.6 What features to use in 2026

• Classical SIFT or ORB: still default in COLMAP.
• ALIKED + LightGlue: modern open standard.
• XFeat: edge / mobile.
• DUSt3R / MASt3R: matching-free (matches as byproduct).

4. Feature Matching

4.1 Classical matching

• Brute-force: \(O(N^2)\) pairwise comparison.
• FLANN (Fast Library for Approximate NN): KD-tree / hierarchical k-means. Standard in OpenCV.
• Lowe's ratio test: keep \(d_1/d_2 < 0.7\) (best vs second-best) to avoid ambiguous matches.
• Mutual NN check: feature \(i\) matches \(j\), and \(j\) matches \(i\) back.

4.2 Geometric verification

After putative matches:

• Estimate fundamental / essential matrix via RANSAC.
• Keep matches consistent with the geometry.
• Reject outliers.

4.3 Modern learned matchers

• SuperGlue (Sarlin 2020): graph neural network on top of SuperPoint; differentiable optimal-transport matching. Major leap.
• LightGlue (Lindenberger 2023): same architecture, faster (depth-adaptive); production default since 2024.
• LoFTR (Sun 2021): detector-free; dense Transformer matching directly on images.
• ASpanFormer, MatchFormer, ROMA: variants.

4.4 Detector-free matching

LoFTR / ROMA / DKM: skip explicit feature detection; produce dense matches end-to-end. Better on textureless / repetitive scenes.

4.5 Image-pair matchers as 3D estimators

DUSt3R / MASt3R: short-circuit matching — predict pixel-aligned 3D pointmaps; matches fall out as nearest neighbors in 3D. The 2024+ paradigm.

4.6 Evaluation metrics

• Precision (correct matches / total matches).
• Recall (correct matches / GT matches).
• AUC at various error thresholds (3px, 5px, 10px).
• HPatches benchmark.

4.7 Practical recipe (2026)

1. Detect features (SIFT, ALIKED, or DISK).
2. Match with LightGlue.
3. Geometric verify with MAGSAC++.
4. Or: use MASt3R / VGGT for direct 3D inference.

★ 2026 SOTA update — Dense-SfM: detector-free dense SfM

• Dense-SfM: SfM from dense detector-free matches with Gaussian-Splatting track extension + transformer/Gaussian-Process multi-view kernelized track refinement. Denser, more accurate than SOTA on ETH3D and texture-poor scenes. CVPR 2025.

5. RANSAC Family

5.1 Basic RANSAC (Fischler & Bolles 1981)

1. Sample \(s\) minimal correspondences.
2. Fit model.
3. Count inliers (within threshold \(\tau\)).
4. Repeat \(N\) iterations; return model with most inliers.

5.2 Number of iterations

For inlier ratio \(w\), sample size \(s\):

\[N = \frac{\log(1 - p)}{\log(1 - w^s)},\]

where \(p\) is the desired confidence (\(p = 0.99\) standard). Adapt \(N\) as \(w\) estimate improves.

5.3 Variants

• MSAC: M-estimator score; per-residual weight.
• MLESAC: maximize likelihood under inlier/outlier mixture.
• LO-RANSAC: local optimization on inliers between iterations.
• PROSAC: progressive sampling guided by feature quality scores.
• MAGSAC++ (2020): marginalize over noise scale; smooth weighting; SOTA quality.
• GC-RANSAC: spatial coherence via graph cuts.
• USAC framework: modular template combining all of the above.
• DEGENSAC: handle degenerate configurations (e.g., planar scenes).

5.4 Modern defaults

• Default: MAGSAC++.
• For repetitive structure: DEGENSAC.
• For real-time: LO-RANSAC or vanilla on small problems.

5.5 Threshold \(\tau\)

• Pixel reprojection threshold: \(\tau \in [1, 4]\) pixels typical.
• Adaptive thresholds (MAGSAC++) avoid hard tuning.
• Keypoint-uncertainty-aware better.

5.6 Failure modes

• Degenerate configurations (planar scenes).
• Repetitive structure (matches consistent with multiple geometries).
• Very low inlier ratio (< 10%).
• Noisy correspondences.

5.7 Differentiable RANSAC

• NG-RANSAC, \(\nabla\)-RANSAC: differentiable for end-to-end learning.
• Used in some learned-matcher pipelines.

6. Bundle Adjustment

6.1 The objective

Minimize total reprojection error over cameras \(\{T_j\}\) and 3D points \(\{X_i\}\):

\[\min_{\{T_j, X_i\}} \sum_{(i,j) \in \Omega} \rho\!\left( \left\| \pi(T_j, X_i) - \mathbf{x}_{ij} \right\|_\Sigma^2 \right),\]

where \(\pi\) is the projection model, \(\rho\) is a robust kernel, \(\Sigma\) is the per-observation covariance.

6.2 Levenberg-Marquardt

Standard non-linear least squares solver:

\[\left( J^{\top} J + \lambda\, \mathrm{diag}(J^{\top} J) \right) \Delta\theta = -J^{\top} r,\]

\(\lambda\) damps between Gauss-Newton and gradient descent. Updated based on whether step decreased the loss.

6.3 Sparsity and Schur complement

• Stack parameters as \((\theta_C, \theta_P)\) for cameras + points.
• \(H_{PP}\) block-diagonal (each point observed independently).
• Eliminate via Schur complement:

\[\left( H_{CC} - H_{CP} H_{PP}^{-1} H_{CP}^{\top} \right) \Delta\theta_C = -b_C + H_{CP} H_{PP}^{-1} b_P.\]

• \(H_{PP}^{-1}\) cheap (per-point \(3 \times 3\) blocks).
• Reduced camera system dense but small.

6.4 Robust kernels

• Huber: quadratic for small residuals, linear for large.
• Cauchy: smooth saturation.
• Tukey: bounded influence.
• Standard choice: Huber.

6.5 Solvers

• Ceres Solver (Google): industry standard.
• g2o: widely used in SLAM.
• GTSAM: factor-graph based.
• Theseus (Meta): differentiable.

6.6 Local vs global BA

• Local BA: optimize recent few keyframes; SLAM-friendly.
• Global BA: optimize everything; SfM-style; more accurate, expensive.

6.7 Practical considerations

• Outlier filtering before BA (or via robust kernel).
• Parameterize \(R\) via quaternion or Lie algebra.
• Right perturbation: \(T \to T \exp(\hat{\delta})\) for cleaner Jacobians.
• Use sparse linear algebra (Eigen, SuiteSparse).
• Multi-threaded eigen-decomposition.

6.8 Compute scale

• Small (50 cameras, 5k points): seconds.
• Medium (500 cameras, 50k points): minutes.
• Large (10k cameras, 1M points): hours.
• Very large (city-scale, 100k cameras): distributed BA needed.

7. COLMAP: The De Facto Pipeline

7.1 What is COLMAP?

Open-source SfM + MVS framework by Schönberger (2016). De facto standard for incremental SfM since 2016.

7.2 The COLMAP SfM pipeline

1. Feature extraction (SIFT default).
2. Pairwise feature matching (vocabulary tree or sequential).
3. Geometric verification.
4. Initial pair selection (well-conditioned baseline).
5. Two-view reconstruction.
6. Incremental: register new images via PnP, triangulate new points.
7. Periodic local + global BA.
8. Outlier filtering.

7.3 COLMAP MVS pipeline

After SfM:

1. PatchMatch stereo per image pair.
2. Per-image dense depth maps.
3. Fusion into point cloud / mesh.

7.4 Initial pair selection

Critical step: choose two images with:

• Many matches.
• Sufficient baseline (good triangulation angle).
• Not too far apart (matching reliable).

Heuristic-based; sometimes fails.

7.5 Image registration (PnP)

• Find 2D-3D correspondences between new image features + existing 3D points.
• PnP + RANSAC.
• Triangulate new points from new image + old.

7.6 Bundle adjustment cadence

• After every \(\sim 10\) image registrations.
• Local BA: only recent cameras + nearby points.
• Global BA: occasionally.
• Final global BA at end.

7.7 Failure modes

• Repeated structures (warehouses, hallways).
• Textureless surfaces.
• Strong illumination changes.
• Dynamic scenes (people moving).
• Dense reflections.
• Very narrow baselines.

7.8 Mitigations

• More / better features (SuperPoint, LoFTR).
• Sequential matching (for video).
• Vocabulary-tree matching (for unordered).
• Mask out dynamic objects.

7.9 Why COLMAP dominated for 8 years

• Robust, precise, well-engineered.
• Open source.
• Scales to thousands of images.
• Accurate enough for VFX / photogrammetry.
• Initialization for NeRF / 3DGS.

7.10 Why COLMAP is being replaced (2024–26)

• Slow (hours for 100s of images).
• Fragile on textureless / repetitive.
• Requires good initial pair.
• No probabilistic uncertainty.
• Feed-forward methods now match accuracy with 100× speedup.

8. Global SfM

8.1 Vs incremental

Incremental (COLMAP-style): cameras added one-by-one; cumulative drift; sensitive to initialization.

Global: solve for all cameras jointly via rotation + translation averaging. Faster but potentially less accurate.

8.2 Rotation averaging

Given relative rotations \(R_{ij}\) from pairwise matching, recover absolute rotations \(R_i\):

\[\min_{\{R_i\}} \sum_{(i,j)} \left\| R_{ij} - R_j R_i^{-1} \right\|.\]

Robust solvers (Chatterjee + Govindu): L1 averaging on \(SO(3)\).

8.3 Translation averaging

Given relative directions \(t_{ij}\) (sign / scale ambiguous), recover \(t_i\):

• 1DSfM (Wilson & Snavely): outlier removal in 1D before averaging.
• Convex relaxation methods.
• LUD (Least Unsquared Deviation).

8.4 Frameworks

• Theia (UCSB): clean global SfM.
• OpenMVG: also has global mode.
• ETH3D pipeline.

8.5 Hybrid approaches

HSfM (hierarchical): cluster images, do global within clusters, merge incrementally.

8.6 When global wins

• Many images, well-distributed.
• Time-critical scenarios.
• Reasonably good pairwise matching.

8.7 When incremental wins

• Sequential capture (video).
• Difficult scenes.
• Need accuracy at all costs.

9. Multi-View Stereo (MVS)

9.1 What is MVS?

After SfM gives camera poses + sparse points, MVS produces dense depth / point cloud / mesh. Per-image depth via stereo matching.

9.2 PatchMatch MVS

• Per-pixel: random-init plane (depth + normal).
• Iterative: propagate good plane to neighbors; refine.
• NCC / ZNCC matching cost across views.
• Standard since Galliani et al. 2015.

9.3 COLMAP MVS

PatchMatch stereo + photometric / geometric consistency check + multi-view fusion into final point cloud.

9.4 Learned MVS

• MVSNet (Yao 2018): 3D feature volume + cost volume regularization.
• CasMVSNet: cascaded coarse-to-fine.
• PatchmatchNet: learned PatchMatch.
• TransMVSNet: transformer-based.

9.5 Modern feed-forward depth

• Depth Anything V2: monocular foundation depth.
• Marigold: diffusion-based monocular depth.
• MoGe / MoGe-2: metric monocular geometry.
• Combine with poses for "MVS via mono-depth alignment."

9.6 Mesh extraction

• Poisson surface reconstruction.
• Marching cubes on TSDF.
• Delaunay triangulation.
• Differentiable mesh extraction (FlexiCubes).

9.7 MVS quality factors

• Camera baseline (too narrow \(\to\) depth ambiguity).
• Texture quality.
• Lighting consistency.
• Resolution.
• View density.

10. Feed-Forward 3D (the 2024–26 Wave)

10.1 The phase transition

Single transformer pass from images \(\to\) 3D structure. Skips classical pipeline entirely.

10.2 DUSt3R (Naver 2024)

The breakthrough: predict per-pixel 3D pointmaps from 2 images, in camera 1's frame.

\[X^{1,1}, X^{2,1} \in \mathbb{R}^{H \times W \times 3}.\]

• Camera intrinsics, extrinsics, depth, point cloud all decode from pointmaps.
• Pixel matches: nearest neighbors in 3D.
• Trained on multi-view supervised data.
• Pose-free (doesn't need known cameras).

10.3 MASt3R

• DUSt3R + explicit dense matching head.
• Better quality on textureless surfaces.
• Stronger feature matching.

10.4 MASt3R-SfM

• Apply MASt3R pairwise to all image pairs.
• Global optimization to align pointmaps consistently.
• Replaces COLMAP for many use cases.
• Comparable accuracy at \(\sim 100\times\) speedup.

10.5 Spann3R (incremental)

• Process one new view at a time.
• Accumulates global 3D representation.
• Online / streaming applications.

10.6 Splatt3R (3DGS direct)

• Predicts 3D Gaussians from image pairs end-to-end.
• Skip pointmap intermediate.
• Scene rendering directly.

10.7 NoPoSplat / Fast3R

• NoPoSplat: jointly pose-free Gaussians.
• Fast3R: optimized for speed; many views (50+).

10.8 VGGT (Visual Geometry Grounded Transformer, Meta 2025)

• Large transformer.
• Maps unposed N-image set to depth + cameras + per-pixel 3D in one forward pass.
• State-of-art on most benchmarks 2025–26.
• Open weights.

10.9 π3 (Pi-cubed, Meta 2025)

• Fully feed-forward, pose-free.
• Scaled up.
• Latest in the lineage.

10.10 Comparison to classical SfM

★ 2026 SOTA update — MapAnything: universal metric 3D

• MapAnything: single transformer ingesting images + optional intrinsics/poses/depth; regresses metric 3D via factored depth + ray maps + poses + scale. One pass covers uncalibrated SfM, MVS, mono-depth, localization, depth completion (Meta, 2025).

★ 2026 SOTA update — CUT3R: stateful online pointmaps

• CUT3R: recurrent persistent-state transformer; ingests image stream, updates state, emits metric-scale pointmaps online. Handles video or unordered photos, static + dynamic; can probe unseen views. CVPR 2025.

★ 2026 SOTA update — Depth Anything 3: any-view geometry

• Depth Anything 3: plain DINO transformer, single depth-ray target predicts consistent geometry from any number of views, posed or unposed. Beats VGGT by ~44% pose / ~25% geometry accuracy. New SOTA (ByteDance Seed, 2025).

★ 2026 SOTA update — StreamVGGT: causal streaming VGGT

• StreamVGGT: causal-attention variant of VGGT with cached KV token memory for low-latency incremental streaming reconstruction; distilled from bidirectional VGGT teacher. Near-offline accuracy, online SOTA (2025).

★ 2026 SOTA update — AnySplat: feed-forward 3DGS init

• AnySplat: one forward pass from unposed/uncalibrated images to 3D Gaussians + intrinsics + extrinsics. Self-distilled from VGGT priors, no SfM/MVS supervision; real-time novel-view synthesis, skips COLMAP init. SIGGRAPH Asia 2025.

11. Visual SLAM

11.1 SLAM vs SfM

• SfM: offline, batch, optimize all cameras.
• SLAM: online, real-time, incremental.
• Same math; different runtime constraints.

11.2 Visual SLAM stacks

• ORB-SLAM3: feature-based; mature; Monocular / Stereo / RGB-D / IMU.
• VINS-Fusion / VINS-Mono: visual-inertial.
• OKVIS / OKVIS-2: VIO.
• DSO (Direct Sparse Odometry): photometric, no features.
• LSD-SLAM: large-scale direct.
• Kimera: with semantic 3D.

11.3 Visual-inertial SLAM

Pre-integrated IMU between keyframes gives motion priors. Drift reduces dramatically. Standard for AR, drones, robotics.

Joint cost:

\[\sum_k \left( r_{\text{vis}} + r_{\text{IMU}} \right) + \text{prior}.\]

11.4 Direct vs feature-based

• Feature-based (ORB-SLAM3): robust to motion blur, fewer correspondences needed.
• Direct (DSO): denser, photometric loss, sensitive to lighting.

11.5 Loop closure

Detect revisited locations; add constraint to pose graph; correct accumulated drift.

• Place recognition (NetVLAD, MixVPR, AnyLoc).
• Geometric verification (PnP + RANSAC).
• Pose graph optimization.

11.6 Neural SLAM (2023+)

• NICE-SLAM, Vox-Fusion: voxel-based neural map.
• NeRF-SLAM: NeRF map + classical front-end.
• MonoGS, GS-SLAM, SplaTAM: 3DGS-based.
• Photo-realistic dense map alongside SLAM.

11.7 Production SLAM stacks

• Apple ARKit: classical VIO.
• Google ARCore: classical VIO.
• Niantic Lightship VPS: classical + learned features.
• Quest Insight: classical hand-tuned.
• Vision Pro: classical + cloud assistance.

★ 2026 SOTA update — VGGT-SLAM: SL(4) submap SLAM

• VGGT-SLAM: dense uncalibrated monocular SLAM aligning VGGT submaps over the SL(4) manifold (15-DoF projective, not similarity). Enables long sequences infeasible for raw VGGT; VGGT-SLAM 2.0 adds real-time. MIT-SPARK, 2025.

12. Camera Models and Calibration

12.1 Pinhole

Most common; 4–5 DoF intrinsics (\(f_x, f_y, c_x, c_y\), optional skew).

12.2 Distortion models

• Brown-Conrady: radial \(k_1, k_2, k_3\) + tangential \(p_1, p_2\).
• Kannala-Brandt: fisheye polynomial.
• Mei: omnidirectional / catadioptric.
• Equirectangular: 360 panoramas.

12.3 Calibration methods

• Zhang's method: planar checkerboard from multiple views.
• Self-calibration: solve from SfM (less accurate).
• Tsai's method: classical; pre-Zhang.
• Charuco / AprilTag: marker-based.

12.4 Calibration toolboxes

• OpenCV calibrateCamera.
• Kalibr (multi-camera + IMU).
• MATLAB camera calibrator.
• COLMAP's built-in.

12.5 Self-calibration during SfM

COLMAP estimates intrinsics during reconstruction. Less accurate than explicit calibration but standard workflow.

12.6 Auto-calibration with feed-forward 3D

DUSt3R / VGGT: predict intrinsics implicitly. No calibration step needed.

13. Loop Closure and Place Recognition

13.1 The problem

After traveling, return to start; recognize "I've been here before"; correct drift.

13.2 Place-recognition pipeline

1. Compute global descriptor per image.
2. Index in retrieval structure.
3. Query: top-k nearest descriptors.
4. Geometric verification: check if 3D structure matches.

13.3 Global descriptors

• NetVLAD (2016): aggregated CNN features. Standard.
• MixVPR: improved MLP-based aggregation.
• AnyLoc: foundation-model features (DINOv2).
• Patch-NetVLAD: patch-level NetVLAD.

13.4 Loop closure mechanics

• Detect candidate via place recognition.
• Verify via PnP / 5-point.
• Add edge to pose graph.
• Pose graph optimization corrects drift.

13.5 Pose graph optimization

\[\min_{\{T_i\}} \sum_{(i,j) \in E} \rho\!\left( \left\| \log\!\left( T_j^{-1} T_i Z_{ij}^{-1} \right) \right\|_\Sigma^2 \right),\]

where \(Z_{ij}\) is the measured relative pose. Used in g2o, GTSAM.

13.6 Long-term localization

• Day-night: drastic illumination changes (Aachen Day-Night benchmark).
• Seasonal: snow / rain / vegetation changes.
• Modern methods: AnyLoc, learned descriptors handle these.

13.7 Visual Positioning Systems (VPS)

Persistent global localization:

• Niantic Lightship VPS.
• Google ARCore Cloud Anchors.
• Apple ARKit Location Anchors.
• Combines server-side mapping with on-device localization.

14. Tools and Software

14.1 Open-source frameworks

14.2 Commercial photogrammetry

• Reality Capture (Epic): film-quality scans.
• Agisoft Metashape: photogrammetry standard.
• Pix4D: drone / aerial.
• 3DF Zephyr: another option.
• Sphere: volumetric capture.

14.3 Mobile / consumer

• Polycam: on-phone photogrammetry + 3DGS.
• Luma AI: on-phone NeRF / 3DGS.
• Scaniverse: LiDAR-aided.
• KIRI Engine: photogrammetry + 3DGS.
• RealityScan (Epic): photogrammetry on phone.

14.4 Modern feed-forward 3D

• DUSt3R / MASt3R / Spann3R / Splatt3R: open Naver code.
• VGGT (Meta): open weights.
• π3: scaled VGGT successor.
• MoGe / MoGe-2: monocular geometry.

14.5 Choosing a tool (2026)

• Highest-precision photogrammetry: Reality Capture / Metashape.
• Open-source default: COLMAP or HLoc.
• Speed-critical: VGGT / MASt3R-SfM.
• Phone capture: Polycam / Luma AI / RealityScan.
• Research: HLoc + COLMAP + VGGT for comparison.

15. Datasets and Benchmarks

15.1 SfM benchmarks

• ETH3D: high-precision evaluation.
• Tanks and Temples: outdoor scenes.
• DTU: object-level MVS.
• Strecha MVS: classical.
• Aachen Day-Night: long-term loc.

15.2 Visual localization

• Aachen Day-Night: outdoor day/night.
• InLoc: indoor.
• RobotCar Seasons: seasonal robustness.
• 4Seasons: car-mounted.

15.3 SLAM benchmarks

• KITTI Odometry: outdoor driving.
• TUM RGB-D: indoor RGB-D.
• EuRoC MAV: drone.
• TartanAir: synthetic large-scale.
• ScanNet: indoor.

15.4 AV-specific

• KITTI-360: 360-degree HD maps.
• nuScenes, Waymo Open: full AV stack.
• Argoverse 2: HD map + SfM.

15.5 Mobile / AR

• ARKit / ARCore datasets: iOS / Android.
• Niantic Lightship: VPS-aligned.

15.6 Ground truth

• Total stations / tachymeters: mm-precision.
• LiDAR scans: dense ground truth.
• GPS / RTK: outdoor.
• Mocap: indoor.
• Synthetic (Hypersim, TartanAir): exact.

16. Production Pipelines

16.1 Photogrammetry workflow

1. Capture: 50–500 photos covering subject.
2. Run SfM + MVS (COLMAP / Reality Capture).
3. Mesh reconstruction (Poisson / TSDF).
4. Texture baking.
5. Export to graphics pipeline.

16.2 3DGS asset pipeline

1. Capture (handheld / turntable).
2. SfM init (COLMAP or VGGT).
3. Optimize 3DGS from SfM points.
4. Compress (LightGaussian).
5. Deploy.

16.3 AV HD map building

1. Vehicle drives test region with cameras + LiDAR + IMU + GNSS.
2. Multi-sensor SfM / SLAM (e.g., NeuRAD, OmniRe).
3. Lane / road / signs annotated.
4. Map served to fleet.
5. Continual updates.

16.4 VFX camera tracking

1. Production footage shot.
2. SfM recovers camera poses.
3. Composite CG aligned with real camera motion.
4. Standard for VFX since 1990s; modern tools include neural-aided tracking.

16.5 AR persistent localization

• Cloud builds map of region.
• Device captures, queries cloud.
• Returns global pose.
• Persistent AR experiences.

16.6 Mobile capture (phone \(\to\) 3D)

Polycam / Luma AI / Scaniverse:

1. User captures photos / video.
2. On-device or cloud reconstruction.
3. Output: 3DGS scene / mesh.
4. Share / export.

16.7 Drone / aerial

• Programmed flight path.
• RTK GPS for accurate poses.
• Photogrammetry tools (Pix4D, Metashape).
• Survey / construction / archeology.

17. Common Failure Modes

17.1 Textureless surfaces

Walls, sky, water. Not enough features. Fix: learned features, MVSNet, modern feed-forward.

17.2 Repetitive structures

Identical windows, parking lots. Fix: DEGENSAC, semantic constraints, larger context.

17.3 Dynamic scenes

Moving people / cars violate static-scene assumption. Fix: mask out dynamic objects (semantic segmentation), SLAM with motion modeling.

17.4 Drift

Cumulative error over long traversal. Fix: loop closure, global BA, IMU integration.

17.5 Scale ambiguity

Monocular SfM has unknown global scale. Fix: known object size, IMU, GPS, stereo, monocular depth model (MoGe).

17.6 Initialization failure

Bad initial pair \(\to\) catastrophic. Fix: try multiple pairs, global SfM.

17.7 Outliers

Mismatched features. Fix: RANSAC, robust kernels, learned matchers.

17.8 Occlusion

Features visible in some images, not others. Fix: handled by triangulation only on overlapping; not a fundamental issue.

17.9 Lighting / weather

Day-night, sun glare, rain. Fix: illumination-invariant features, learned matchers (better at this), AnyLoc-class.

17.10 Narrow baselines

Insufficient parallax. Fix: better baseline (move further).

18. Theoretical Foundations

18.1 Cramer-Rao bound

Lower bound on covariance of unbiased estimator. Used to predict expected SfM accuracy.

18.2 Triangulation uncertainty

• Larger baseline \(\to\) more accurate depth.
• Triangulation angle \(\theta\): depth-uncertainty \(\propto 1/\sin\theta\).
• Standard triangulation uncertainty model.

18.3 Reconstruction ambiguities

• Scale: monocular SfM has 1-DoF scale ambiguity.
• Bas-relief (uncalibrated photometric stereo): 3-DoF family.
• Projective: without calibration, only projective (15 DoF).
• Affine \(\to\) Metric: with intrinsics, recover up to scale.
• Metric: with scale, full reconstruction.

18.4 Degenerate configurations

• All cameras on a line.
• All points coplanar (essential matrix degenerate).
• Pure rotation (no translation).

18.5 Optimal triangulation

Hartley-Sturm: minimize geometric reprojection in both images via optimization on a 6th-order polynomial. Closed-form.

18.6 Uncertainty quantification

• Per-point covariance from BA Hessian.
• Per-camera pose covariance.
• Used in active-vision (next-best-view).

18.7 Robust statistics

• M-estimators: down-weight outliers.
• RANSAC: hard-classify outliers.
• Combination: RANSAC for initialization, robust kernel during BA.

19. Frontier 2025–2026

19.1 Trends

• Feed-forward 3D (VGGT / π3 / MASt3R) replaces COLMAP for many use cases.
• LightGlue + ALIKED + MAGSAC++ as the modern classical default.
• HLoc as standard wrapper for matcher + COLMAP.
• 3DGS init from VGGT instead of COLMAP.
• Mobile capture mainstream (Polycam / Luma / Scaniverse).

19.2 Hybrid pipelines

• VGGT init + COLMAP refine.
• Learned matchers + classical BA.
• Best of both: speed + precision.

19.3 Open research

• Scaling feed-forward 3D to thousands of images.
• Per-pixel uncertainty quantification.
• Online streaming SfM (Spann3R-style).
• Dynamic scene SfM.
• Cross-domain (sat \(\leftrightarrow\) ground).

19.4 End-to-end pipelines

• Image \(\to\) 3DGS via Splatt3R / NoPoSplat.
• Image \(\to\) mesh via VGGT + post-process.
• Skip SfM as intermediate step entirely.

19.5 Production reality

• COLMAP / Reality Capture / Metashape: still production-grade for high-precision.
• VGGT / Polycam: consumer / fast.
• HLoc + LightGlue + COLMAP: open research / academic.
• Apple Vision Pro / Quest: closed inside-out tracking.

19.6 Convergence with neural rendering

• Feed-forward 3D outputs \(\to\) direct 3DGS rendering.
• No SfM as separate pipeline.
• One model: image \(\to\) scene representation.
• Polycam / Luma already do this.

★ 2026 SOTA update — VGGT-Long: km-scale streaming

• VGGT-Long: chunk-loop-align scheme pushing VGGT to kilometer-scale unbounded outdoor RGB streams under fixed memory, with loop closure alignment. Runs on KITTI/Waymo/VKITTI where foundation models OOM; targets autonomous driving (2025).

20. Production Stack 2026

Appendix A: Twenty-Five Things to Know

1. Pinhole camera: \(\lambda x = K[R|t]X\).
2. Brown-Conrady distortion (radial + tangential).
3. Zhang's calibration: planar checkerboard from multiple views.
4. Epipolar constraint: \(x_2^{\top} E x_1 = 0\).
5. Essential matrix: \(E = [t]_\times R\), 5 DoF.
6. Fundamental matrix: \(F = K_2^{-\top} E K_1^{-1}\), 7 DoF, rank-2.
7. Eight-point algorithm with Hartley normalization.
8. Five-point algorithm (Nistér): minimal calibrated case.
9. DLT triangulation via SVD.
10. PnP: P3P (4 candidates) or EPnP \(O(n)\).
11. RANSAC iterations: \(N = \log(1 - p)/\log(1 - w^s)\).
12. MAGSAC++: marginalize over noise scale.
13. Reprojection error + Huber kernel + LM = bundle adjustment.
14. Schur complement reduces BA to camera-only system.
15. Ceres / g2o / GTSAM standard solvers.
16. COLMAP: incremental SfM + PatchMatch MVS.
17. HLoc: SuperPoint + SuperGlue / LightGlue + COLMAP.
18. LightGlue is the production-default learned matcher (2024+).
19. DUSt3R: predict pixel-aligned 3D pointmaps from 2 images.
20. MASt3R-SfM: replaces COLMAP at \(\sim 100\times\) speedup.
21. VGGT (Meta 2025): feed-forward N-image \(\to\) depth + cameras.
22. Visual SLAM = online SfM with real-time constraints.
23. ORB-SLAM3 / VINS-Fusion / OKVIS-2 are mature SLAM stacks.
24. Loop closure: place recognition + geom verify + pose graph opt.
25. NetVLAD / MixVPR / AnyLoc are place recognition standards.

Appendix B: Decision Tree — "Which SfM?"

1. Highest-precision photogrammetry? \(\to\) Reality Capture or Metashape.
2. Open-source max precision? \(\to\) COLMAP + HLoc + LightGlue + MAGSAC++.
3. Fast SfM (seconds)? \(\to\) VGGT or MASt3R-SfM.
4. 3DGS pipeline init? \(\to\) VGGT (replacing COLMAP).
5. Phone capture? \(\to\) Polycam / Luma AI / Scaniverse / RealityScan.
6. Real-time SLAM? \(\to\) ORB-SLAM3 (no IMU) or VINS-Fusion (with IMU).
7. Photoreal map needed? \(\to\) MonoGS / SplaTAM (3DGS-SLAM).
8. Aerial / drone survey? \(\to\) Pix4D / Metashape + RTK.
9. AR persistent localization? \(\to\) Niantic Lightship / ARKit Cloud Anchors.
10. Long-term loc (day-night)? \(\to\) AnyLoc + LightGlue for illum-robust.
11. Visual-inertial? \(\to\) VINS-Fusion / OKVIS-2 / Kimera.
12. VFX camera tracking? \(\to\) PFTrack / SynthEyes / Reality Capture.

Appendix C: Year-by-Year SfM Milestones

• 1981: Lucas-Kanade optical flow; Fischler & Bolles RANSAC.
• 1988: Harris corner detector.
• 1992: Tomasi-Kanade factorization (orthographic SfM).
• 1997: Hartley 8-point + normalization.
• 1999: SIFT (Lowe).
• 2000: Hartley & Zisserman "Multiple View Geometry" (the textbook).
• 2004: Nistér 5-point algorithm; SURF.
• 2007: PMVS / PMVS-2.
• 2008: Zhang's calibration; Bundler (precursor to COLMAP).
• 2011: ORB; FAST + BRIEF.
• 2015: COLMAP first release; PatchMatch MVS standardized; ORB-SLAM.
• 2016: NetVLAD; ORB-SLAM2.
• 2017: ORB-SLAM3 / VINS-Fusion mainstream; SuperPoint precursors.
• 2018: SuperPoint (DeTone); MVSNet; D2-Net.
• 2020: SuperGlue (Sarlin); R2D2; MAGSAC++; LoFTR (released 2021).
• 2022: HLoc mainstream; PixSfM.
• 2023: LightGlue (production-default learned matcher); AnyLoc; XFeat (released 2024).
• 2024 (early): DUSt3R (Naver); MASt3R; Spann3R; Splatt3R; NoPoSplat; Fast3R; MoGe.
• 2024 (mid–late): MASt3R-SfM replaces COLMAP for many use cases; GLOMAP; PixSfM; HLoc 1.5.
• 2025: VGGT (Meta) — feed-forward 3D dominant; π3; depthcrafter; MoGe-2; cross-modal SfM (sat + ground).
• 2026: feed-forward 3D + classical hybrid as the production default; mobile capture mainstream; SfM increasingly "solved" as separate problem; integrated with neural rendering.