Delighting & Relighting — All Variants & Tricks

Updated July 2026 with 2025–2026 SOTA additions — new entries marked ★. Algorithm names link to their papers (arXiv / project page).

July 2026 · Updated Edition

Contents

1. Foundations
2. Light Transport Math
3. Material Models for Real-Time & Offline
4. Intrinsic Image Decomposition
5. Photometric Stereo
6. Light Stage Capture
7. Image-Based Lighting (IBL)
8. Portrait / Face Relighting
9. Object Relighting
10. Scene Relighting
11. Generative / Diffusion-Based Relighting
12. Video Relighting
13. AR / Mixed Reality Lighting
14. VFX and Virtual Production
15. Capture Rigs and Methodologies
16. Material Capture and Datasets
17. Neural BRDF and Material Models
18. Evaluation
19. Production Workflows (2026)
20. Frontier 2025–2026 and Open Problems
21. Production Stack 2026

• Appendix A: Twenty-Five Things to Know
• Appendix B: Decision Tree — "Which Relighting Method?"
• Appendix C: Year-by-Year Milestones

1. Foundations

1.1 What is delighting?

Removing the influence of the original capture lighting from an image so that material properties (especially albedo) can be recovered. Outputs the "flat lit" or "albedo" version of an image.

1.2 What is relighting?

Synthesizing how a scene / portrait / object would appear under a new lighting condition. Often combined with delighting:

\[\text{capture} \to \text{delight} \to \text{relight}.\]

1.3 Why this matters

• VFX / film: composite virtual elements with consistent lighting.
• AR / Mixed Reality: insert virtual objects that match real scene lighting.
• E-commerce: relight product photos for different scenes / stories.
• Avatars: animatable + relightable digital humans.
• Game pipelines: photogrammetry assets need lighting removed before re-shading.
• Generative AI: relighting as a style edit (IC-Light, SwitchLight).

1.4 The decomposition problem

Image \(I\) is a function of geometry \(G\), material \(M\), and lighting \(L\):

\[I = R(G, M, L).\]

Inverse rendering: recover \((G, M, L)\) from \(I\). Fundamentally ill-posed: many combinations of \((G, M, L)\) produce the same \(I\). Strong priors / multi-view / explicit capture rigs needed.

1.5 Two paradigms

Physically-based: full inverse rendering with explicit material models, light transport, geometry. Slow, principled.

• Image-based / data-driven: learn delighting / relighting from data without explicit models. Fast, plausible, less controllable.

1.6 The 2024–2026 trend

Key

Diffusion-based generative relighting (IC-Light, SwitchLight, Relightful Harmonization) became the dominant approach in 2024–25 because it sidesteps the ill-posed inverse rendering problem with strong learned priors. Physically-based methods remain the standard for VFX where controllability is paramount.

2. Light Transport Math

2.1 The rendering equation (Kajiya 1986)

The fundamental equation of computer graphics:

\[L_o(x,\omega_o) = L_e(x,\omega_o) + \int_\Omega f_r(x,\omega_i,\omega_o)\, L_i(x,\omega_i)\, (\omega_i \cdot n)\, \mathrm{d}\omega_i.\]

• \(L_o\): outgoing radiance.
• \(L_e\): emitted radiance.
• \(f_r\): BRDF.
• \(L_i\): incoming radiance.
• \((\omega_i \cdot n)\): cosine attenuation (Lambert's law).
• \(\Omega\): hemisphere of incoming directions.

2.2 BRDF (Bidirectional Reflectance Distribution Function)

\[f_r(x,\omega_i,\omega_o) = \frac{\mathrm{d}L_o(x,\omega_o)}{L_i(x,\omega_i)\, (\omega_i \cdot n)\, \mathrm{d}\omega_i}.\]

Properties: non-negative, reciprocity (\(f_r(x,\omega_i,\omega_o) = f_r(x,\omega_o,\omega_i)\)), energy conservation:

\[\int f_r\, (\omega_i \cdot n)\, \mathrm{d}\omega_i \le 1.\]

2.3 Beyond BRDF

• BTDF: transmission (refraction).
• BSDF: scattering = BRDF + BTDF.
• BSSRDF: subsurface scattering (skin, marble).
• BTF (Bidirectional Texture Function): spatially varying + view-dependent.
• SVBRDF: spatially-varying BRDF (typical real-world surface).

2.4 Direct vs indirect illumination

Direct: light from sources directly. Indirect (global illumination): light bounced off other surfaces. Integral of rendering equation over multiple bounces. Path tracing, photon mapping, irradiance caching, screen-space global illum.

2.5 Diffuse vs specular

• Diffuse: equally scatters in all directions; view-independent; albedo-dominated.
• Specular: directional reflection; view-dependent; mirror at extreme.
• Real surfaces: combination.

2.6 Lambertian (perfect diffuse)

\[f_r = \frac{\rho}{\pi},\]

\(\rho =\) albedo. Constant BRDF; outgoing radiance proportional to \((\omega_i \cdot n)\) only.

2.7 Microfacet models

Surfaces modeled as collections of micro-mirrors:

\[f_r = \frac{F \cdot G \cdot D}{4(\omega_i \cdot n)(\omega_o \cdot n)},\]

• \(D\): normal distribution function (e.g., GGX, Beckmann).
• \(G\): geometry / shadowing-masking.
• \(F\): Fresnel reflectance.

2.8 GGX / Trowbridge-Reitz

\[D_{\mathrm{GGX}}(h) = \frac{\alpha^2}{\pi\big((h \cdot n)^2(\alpha^2 - 1) + 1\big)^2},\]

where \(\alpha = \text{roughness}^2\). Industry standard for micro-facet \(D\). Long tails (better for grazing angles than Beckmann).

2.9 Fresnel

Schlick approximation:

\[F(\theta) = F_0 + (1 - F_0)(1 - \cos\theta)^5,\]

\(F_0\) = base reflectivity. For dielectrics \(F_0 \sim 0.04\); for metals $F_0 = $ albedo.

3. Material Models for Real-Time & Offline

3.1 Phong / Blinn-Phong (legacy)

\[L_o = k_a + k_d(\omega_i \cdot n) + k_s(\omega_r \cdot \omega_o)^n.\]

Cheap; not energy-conserving; not physically correct. Replaced by PBR.

3.2 Cook-Torrance

Microfacet diffuse + specular split. Foundation of modern PBR.

3.3 Disney Principled BRDF

Single artist-friendly BRDF with parameters: base color, metallic, roughness, specular, anisotropic, sheen, clearcoat, subsurface. Industry standard since 2012; used in Maya, Houdini, Blender, Substance, Unreal, Unity.

3.4 PBR (Physically-Based Rendering)

Workflow with parameter set:

• Albedo / base color: diffuse color (no lighting baked).
• Metallic (0 or 1 typically): metallic vs dielectric.
• Roughness: micro-surface variance.
• Normal map: surface micro-geometry.
• AO (ambient occlusion): bake of self-occlusion.
• Optional: clearcoat, anisotropy, sheen, subsurface, transmission.

3.5 SVBRDF (Spatially-Varying BRDF)

Per-pixel material parameters across a surface. The standard output of material capture / estimation. Stored as texture maps: albedo, normal, roughness, metallic, etc.

3.6 Subsurface scattering (SSS)

Light enters surface, scatters, exits elsewhere. Critical for skin, wax, jade, milk. Models: separable diffusion (Burley), random walk, BSSRDF.

3.7 Anisotropic BRDFs

Direction-dependent reflection (brushed metal, hair, fabric weave). Anisotropic GGX adds tangent / bitangent roughness.

3.8 Hair, cloth, iridescence

• Hair: Marschner BCSDF (cylindrical fibers).
• Cloth: Disney sheen, ASTM standard models.
• Iridescence: thin-film interference (soap, beetle wings).

3.9 Neural BRDFs

Learned parametric or non-parametric BRDF representation. Examples: NeuMIP, Neural BTF, MaterialGAN.

Compact storage of measured BRDFs; differentiable.

4. Intrinsic Image Decomposition

4.1 The classical formulation

Decompose image \(I\) into reflectance \(R\) (albedo) and shading \(S\):

\[I = R \cdot S.\]

Often in log domain:

\[\log I = \log R + \log S.\]

\(R\) is what we want for delighting; \(S\) contains baked lighting.

4.2 Land's Retinex

Edges in log \(I\) that are sharp = reflectance edges; smooth gradients = shading. Classical algorithm: filter out smooth gradients to recover \(\log R\). Limitations: real-world edges aren't strictly classified.

4.3 Color Retinex

Use color information to disambiguate. A reflectance change usually changes hue; a shading change preserves hue. Improves over scalar Retinex.

4.4 Sparse R + smooth S priors

• Reflectance: piecewise-constant; sparse gradients.
• Shading: smooth; small Laplacian.

\[\|I - RS\|^2 + \lambda_R \|\nabla R\|_1 + \lambda_S \|\nabla^2 S\|^2.\]

4.5 Modern intrinsic decomposition

• IIW (Intrinsic Images in the Wild): large dataset of human-annotated relative reflectance.
• CGIntrinsics: synthetic photorealistic dataset.
• IIW + CGI training: most CNN methods combine.
• Direct Intrinsics, Bell et al., Li & Snavely: classical CNN baselines.

4.6 Modern neural intrinsics (2024–26)

• Ordinal-Intrinsics, \(\mathbf{RGB}{\to}\mathbf{X}\) (Adobe): predict normal, depth, albedo, roughness, metallic from a single image.
• IntrinsicAnything: diffusion-based intrinsic decomposition.
• IntrinsicDiffusion: latent diffusion specialized for intrinsics.

4.7 Beyond reflectance + shading

Modern decomposition outputs full PBR maps:

• Albedo (base color, no lighting).
• Normal (surface direction).
• Roughness (gloss).
• Metallic (dielectric vs metal).
• Often also depth, AO, lighting estimate.

5. Photometric Stereo

5.1 Classical Woodham (1980)

\(N \ge 3\) images of a Lambertian surface under different known directional lights. Per-pixel:

\[I_k = \rho \cdot (l_k \cdot n),\]

where \(l_k\) is the \(k\)-th light direction. Stack across \(K\) images:

\[\mathbf{I} = L \cdot (\rho\, \mathbf{n}).\]

Solve linear system for \(\rho\,\mathbf{n}\); normalize.

5.2 Outputs

• Per-pixel normal (from direction).
• Per-pixel albedo (from magnitude).
• Implicit shape (integrate normals).

5.3 Color photometric stereo

RGB lights at different positions in single shot (R, G, B from different angles). One-shot dynamic-scene capable.

5.4 Near-light photometric stereo

Light source close to surface; per-pixel light direction varies. More complex but captures fine detail.

5.5 Uncalibrated photometric stereo

Light directions unknown. Bas-relief ambiguity (Belhumeur, Kriegman, Yuille): solution determined only up to a 3-parameter family. Resolved with priors or extra constraints.

5.6 Deep photometric stereo

• PS-FCN: CNN regresses normals from \(K\) images.
• PS-Transformer: attention over \(K\) image stack.
• Universal Photometric Stereo (UPS): works with any number of lights, including 1.
• SDM-UniPS, SDPS-Net: more recent diffusion / set-transformer based.

5.7 Modern capture rigs

• Light stage (Debevec): hundreds of LEDs.
• Mobile photometric stereo: phone flash + ambient; typically 4–8 captures.
• Cross-polarized photo: removes specular; recovers diffuse + albedo.

6. Light Stage Capture

6.1 The Debevec Light Stage

USC ICT's hemispherical capture rig: hundreds of computer-controlled LEDs at known positions. Subject in center. Rapid capture under each light \(\to\) reflectance field.

6.2 Reflectance field

For each pixel \(p\), store \(T_p(\omega_i)\): how much that pixel responds to light from direction \(\omega_i\). Image under any new lighting:

\[I_p^{\mathrm{new}} = \int T_p(\omega_i)\, L^{\mathrm{new}}(\omega_i)\, \mathrm{d}\omega_i.\]

Linear in lighting; relighting is a matrix-vector multiply.

6.3 One-light-at-a-time (OLAT)

Capture image with each light alone (one-hot lighting). Stack of \(K\) images is a discrete reflectance field.

6.4 Time-multiplexed capture

Modern LED strobes flash at video frame rate; subject held still or moves slowly. Capture full reflectance field in seconds.

6.5 Gradient illumination (Ma et al.)

Instead of OLAT, use spatial-gradient lighting patterns:

• Constant.
• Linear gradients in X, Y, Z.
• Polynomial higher-order.

Few captures suffice (4–8) to estimate normals + diffuse + specular separation.

6.6 Polarization

Cross-polarized lights and camera filter:

• Parallel: full reflection (diffuse + specular).
• Cross: diffuse only.
• Difference: specular only.

Standard at light stages for clean separation.

6.7 Light-stage outputs (full pipeline)

• Per-pixel normal (gradient illumination + photometric).
• Albedo (cross-polarized).
• Specular roughness (polarization difference + analysis).
• Subsurface scattering (decay over distance).
• Reflectance field (OLAT) for direct relighting.

6.8 Production light stages

ICT Light Stage X, Lytro / Google (deceased), Disney Light Stage, Industrial Light & Magic Stagecraft (LED volumes). Used for film (Avatar, Avengers, Mandalorian), game cinematics, premium digital humans.

7. Image-Based Lighting (IBL)

7.1 Environment maps

A single panoramic HDR image captures the lighting at a point in space. Used to light synthetic objects placed at that point.

• Latitude-longitude (equirectangular): most common.
• Cube map: 6 faces of a cube; faster sampling.
• Dual paraboloid: 2 parabolic projections.

7.2 Capture

• HDR pano: stitched bracketed photos; common for VFX.
• Chrome ball: photograph a chrome sphere; unwrap to env map.
• 360 cameras with HDR: Insta360, Ricoh Theta with HDR mode.
• Probes / Theta One for VFX on-set.

7.3 Pre-filtered environment maps

For real-time use, pre-convolve env map with various roughness BRDFs:

• Diffuse irradiance map: pre-integrate diffuse cos kernel.
• Specular pre-filtered cubemap: mipmaps with progressively wider GGX kernels.

Lookup: just sample the cubemap at appropriate mip level.

7.4 Split-sum approximation (UE4-style)

\[\int L_i \cdot f_r\, (\omega_i \cdot n)\, \mathrm{d}\omega_i \approx \underbrace{\int L_i\, (\omega_i \cdot n)\, \mathrm{d}\omega_i}_{\text{pre-filtered env}} \; \cdot \; \underbrace{\int f_r\, (\omega_i \cdot n)\, \mathrm{d}\omega_i}_{\text{BRDF LUT (2D)}}.\]

Fast real-time IBL; standard since UE4.

7.5 Spherical Harmonics (SH) lighting

Project env map to SH basis (typically 9 coefficients for 2nd-order):

\[L_i(\omega) \approx \sum_{l=0}^{2} \sum_{m=-l}^{l} c_{lm}\, Y_{lm}(\omega).\]

For Lambertian BRDF: irradiance = closed-form linear combination of SH coefficients (Ramamoorthi & Hanrahan). Enables real-time IBL with \(\sim 9\) floats.

7.6 Spherical Gaussians (SG)

\[G(\omega; \mu, \lambda) = e^{\lambda(\omega \cdot \mu - 1)}.\]

Sum of SGs approximates env map. Better for specular than SH (which is band-limited). Used in PhySG, NeRO.

7.7 Modern lighting representations for neural rendering

• Direct env map.
• SH (9, 16, 25 coefficients).
• Spherical Gaussians (lobed).
• Voxel light grids (Lighthouse).
• Implicit / neural lighting (small MLPs).

8. Portrait / Face Relighting

8.1 Why portraits are special

• Subsurface scattering in skin.
• Sharp specular highlights from oily skin.
• Hair complex (Marschner).
• Eyes are mirrors (highlight critical).
• Face geometry well-priored (template or 3DMM).
• High user expectation for realism.

8.2 Sun et al. (Single Image Portrait Relighting, SIGGRAPH 2019)

Single-image deep portrait relighting:

• Encoder \(\to\) implicit features.
• Lighting parameters injected.
• Decoder \(\to\) relit image.
• Trained on light-stage data of 18 subjects under thousands of lighting conditions.

8.3 Total Relighting (Pandey et al. SIGGRAPH 2021)

Foreground matting + portrait relighting + background compositing in one pipeline. Production-quality results from a single image.

8.4 Lumos (Microsoft)

Single-image portrait relighting + edit; high-quality skin tone preservation.

8.5 SwitchLight (Beeble.ai 2024)

Foundation-model-style portrait relighting trained on massive light-stage data + diffusion priors. State-of-art quality; commercial product.

8.6 Relightful Harmonization (Pandey et al. 2024)

Composite a portrait into a new background and re-light to match. Combines relighting + harmonization.

8.7 DiFaReli (Diffusion Face Relighting)

Diffusion-based portrait relighting; single image; controllable lighting via conditioning on env map / direction.

8.8 IC-Light (Lvmin Zhang 2024)

Open-source diffusion-based relighting:

• Trained to "imagine" a foreground subject under arbitrary lighting.
• Two variants: text-conditioned + background-conditioned (composite + relight).
• Surprisingly robust on humans, objects, scenes.
• Used widely in indie creative tools by 2025.

8.9 Portrait-relighting capture rigs

• ICT Light Stage: gold standard, hundreds of LEDs.
• Google Light Stage: capture for Total Relighting + Pixel.
• Beeble Light Stage: SwitchLight training data.
• Mobile / DIY rigs: 4–16 LEDs around subject.

9. Object Relighting

9.1 Object-relighting use cases

• E-commerce product photography.
• Game asset photogrammetry.
• Digital twins.
• Art / marketing creative.

9.2 Single-image object delighting

Recover albedo + normals + roughness + metallic from a single product photo. Then re-render under new lighting.

• IntrinsicAnything: diffusion-based intrinsic decomposition for general objects.
• NeuralPBR: regress full PBR maps.
• AlbedoX, \(\mathbf{RGB}{\to}\mathbf{X}\): Adobe's research line.

9.3 Multi-image SVBRDF capture

2–16 photos with flash from different angles. Recover full SVBRDF.

• Single-image SVBRDF (Deschaintre): with flash photo only.
• Two-shot SVBRDF: flash + ambient.
• MaterialGAN: GAN-based SVBRDF generation conditioned on photo.

9.4 Diffusion-based object relighting

• IC-Light: fg + bg conditioning \(\to\) relit composite.
• DiLightNet: diffusion conditioned on direction map.
• Generative Photography: full pipeline for ad/product.
• Imagine Flash: real-time relighting via consistency models.

9.5 Photogrammetry to PBR pipeline

1. Capture 50–200 photos (turntable / handheld).
2. Reconstruct mesh + textures (Reality Capture, Metashape, NeRFstudio).
3. Bake into PBR maps via baking process or estimation.
4. Delight: remove baked lighting from albedo (cross-polarized capture or AI).
5. Re-render in PBR engine.

9.6 Cross-polarized object photogrammetry

• Polarizing filter on light source.
• Cross-polarizing filter on camera.
• Captures only diffuse \(\to\) direct albedo.
• Repeat without cross-polarization for full reflectance; subtract.

Standard for high-fidelity asset capture (Quixel Megascans).

10. Scene Relighting

10.1 NeRF-based relighting

• NeRD (Neural Reflectance Decomposition): per-point BRDF + lighting.
• NeRF-OSR: outdoor scene relighting from photos.
• PhySG: BRDF + spherical-Gaussian env light.
• NeRO: better specular-handling for shiny objects.
• NeRFactor: factor scene into geometry + reflectance + lighting.

10.2 3D Gaussian Splatting relighting

• Relightable 3D Gaussians (Gao et al. 2024): per-Gaussian BRDF + scene lighting.
• GS-IR (Inverse Rendering): BRDF + indirect lighting via path tracing on Gaussians.
• R3DG (Relightable 3D Gaussians): estimates per-Gaussian normal, albedo, roughness, metallic.
• GShader: differentiable shader for 3DGS.

10.3 Inverse rendering challenges

• Albedo / shading ambiguity.
• Indirect illumination usually ignored or approximated.
• Specular vs diffuse split unstable.
• Anisotropic materials hard.
• Light sources within scene (emissive) tricky.

10.4 Scene-level workflows

1. Capture scene (photos or video).
2. Reconstruct 3D rep (NeRF, 3DGS).
3. Decompose into geometry + materials + lighting (inverse rendering).
4. Edit lighting (replace env map, move light sources).
5. Render with new lighting.

10.5 Hybrid: reconstruct + delight + relight via diffusion

1. Reconstruct scene 3D.
2. Render canonical views.
3. Apply diffusion-based relighting (IC-Light) per view.
4. Bake back into 3D representation.

Production-quality but per-view consistency is a challenge.

11. Generative / Diffusion-Based Relighting

11.1 Why diffusion for relighting?

• Strong priors: diffusion learns natural lighting from billions of images.
• Sidesteps explicit inverse rendering's ill-posedness.
• Single-image input; no capture rig needed.
• Plausible (if not physically exact) results.

11.2 IC-Light (lllyasviel / Lvmin Zhang)

Open-source 2024 hit. Two variants:

• FC (Foreground-Conditioned): input = subject; condition on text describing target lighting; output = relit subject.
• FBC (Foreground+Background): input = subject + bg; output = subject relit to match bg.

ControlNet-style architecture; trained on \(\sim 50\mathrm{M}\) images with synthetic lighting augmentations.

11.3 SwitchLight (Beeble.ai)

Foundation-model-scale portrait relighting. Trained on huge light-stage corpus + diffusion prior. Photoreal portrait relighting at production quality. Commercial.

11.4 Relightful Harmonization

Google research. Composite portrait + bg + relight to match bg's implicit lighting. End-to-end without explicit lighting estimation.

11.5 DiLightNet

Diffusion conditioned on direction maps. More controllable than text-only.

11.6 Generative Photography (commercial tools)

Photoshop's Generative Fill, Adobe Firefly, Topaz Photo AI: relighting baked into general image-edit features.

"Make it sunset" / "Add window light" work via diffusion fine-tunes.

11.7 Trade-offs vs physically-based

• Pro: easy, single image, no capture rig.
• Pro: photoreal results without material estimation.
• Con: not physically consistent (shadows / GI may be wrong).
• Con: less controllable than env-map-based.
• Con: per-view inconsistency in 3D / video.

11.8 Multi-view consistent diffusion relighting

2025 research direction: condition diffusion on multiple views simultaneously (MVDream-style) for consistent relighting across viewpoints. Early but promising.

12. Video Relighting

12.1 Per-frame challenges

• Temporal flicker from per-frame inconsistency.
• Motion-blur handling.
• Long-clip identity preservation.
• Fast-changing real lighting (camera moving through space).

12.2 Methods

• Optical flow + warping: relight one frame, warp to others.
• Temporal diffusion: video diffusion conditioned on lighting.
• 4D Gaussian Splatting + relight: capture as 4D, edit lighting.
• Per-clip light-stage capture: only feasible in studio.

12.3 Live relighting (streaming)

• AR avatars: real-time relight from device sensor.
• Virtual production: LED volumes + actor; real-time relight via reflectance field.
• Game cinematics: relight pre-rendered cutscenes.

12.4 Temporal regularization tricks

• Frame-to-frame consistency loss in training.
• Latent interpolation across video diffusion.
• Cached intermediate features (cross-frame attention).

13. AR / Mixed Reality Lighting

13.1 The integration problem

Inserting virtual objects into real environments: must match lighting, color, shadows, reflections, of the real scene to be believable.

13.2 Lighting estimation from device

• ARKit Environment Probe / ARCore Lighting Estimation: low-resolution env map from camera over time.
• Spherical Harmonics estimate: low-frequency directional lighting.
• Mean intensity + tint: simplest; cheap.
• Deep methods: predict env map from single phone photo (Gardner et al., DeepLight).

13.3 Real-time IBL on mobile

• Pre-filtered cubemap from estimated env map.
• Split-sum approximation.
• Diffuse irradiance map (9-coef SH usually).
• Update at \(\sim 10\text{–}30\) Hz.

13.4 Shadow casting

Virtual objects cast shadows on real surfaces. Approaches:

• Plane-based shadows (assumed flat ground).
• Real geometry estimation (LiDAR on Pro iPhones / Vision Pro).
• Rendered shadow plane composited via multiply.

13.5 Apple ARKit / Vision Pro

• RealityKit: built-in IBL + shadow casting.
• Vision Pro: full per-frame env capture; high-fidelity virtual integration.
• LiDAR on Pro iPhones / iPads: real-geometry shadows.

13.6 Niantic Lightship

Outdoor lighting estimation from sun position + sky model + camera analysis. Persistent VPS-tied lighting.

13.7 Quest / Meta SDK

Lightweight env probe; real-time IBL for hand and pass-through MR.

14. VFX and Virtual Production

14.1 Traditional VFX relighting

Per-shot lighting decisions made in post:

• Plate (live-action footage) graded for canonical lighting.
• CG elements lit to match.
• Composited via Nuke / Fusion.
• HDR env map captured on set used for IBL.

14.2 LED Volumes / Virtual Production (Mandalorian, Stagecraft)

Real-time game-engine background on huge LED wall:

• Actor lit by content on screen \(\to\) realistic IBL captured in-camera.
• No greenscreen; final pixels in-camera (mostly).
• Reduces post-production time dramatically.
• ILM Stagecraft, Sony Volume, ARWall, others.

14.3 Light-stage scanning for digital humans

• Capture actor under polarized + gradient lighting.
• Recover full PBR maps.
• Rebuild in CG (Houdini / Maya).
• Relight to fit any scene.
• Used for de-aging, digital doubles, deceased-actor replication.

14.4 Set extension and re-lighting

Add CG environment behind real actors; relight actors / partial set to match. Common in modern blockbusters.

14.5 Color science in VFX

• Capture: log encoding (S-Log3, Log C4, Log3G10).
• Working: ACEScg.
• Composite & grade in Nuke / Resolve.
• Output: PQ HDR / DCI-P3 cinema or Rec.709 SDR.

15. Capture Rigs and Methodologies

15.1 Multi-flash capture

Smartphone with multiple computer-controlled lights or external flash positions. 4–8 captures suffice for SVBRDF.

15.2 Polarization rigs

• Linear polarizer on flash.
• Linear polarizer on camera (cross-aligned).
• Captures diffuse only.
• Industry standard for albedo capture (Quixel, X-Rite).

15.3 Structured light

Project known patterns to estimate depth / normals (Microsoft Kinect generation 1/2; Apple Face ID). Combined with photometric for per-pixel material.

15.4 HDR + bracketing

Required for any inverse rendering: linear sensor data with full dynamic range.

15.5 Flash photography for SVBRDF

Single-image SVBRDF from flash photo (Deschaintre et al. 2018): a single photo with on-axis flash provides controlled lighting. CNN regresses albedo + normal + roughness + specular.

15.6 Turntable photogrammetry

Object on turntable; camera fixed; 50–200 images. Reconstruct geometry via SfM/MVS; bake albedo via crosspolarization.

15.7 Smartphone capture apps (consumer)

• Polycam, Luma AI, KIRI Engine: photogrammetry / NeRF / 3DGS.
• Scaniverse: LiDAR-aided.
• RealityScan (Epic): photogrammetry.

16. Material Capture and Datasets

16.1 Measured BRDF databases

• MERL BRDF Database: 100 isotropic measured materials.
• UTIA BRDF: 150 anisotropic.
• Disney MERL Anisotropic: extension.

16.2 Material libraries (PBR)

• Quixel Megascans (Epic): 15k+ scanned materials + objects.
• Substance 3D Source (Adobe): procedural + scanned.
• Poly Haven: open materials.
• ambientCG: open CC0 materials.
• Texture Haven, HDRI Haven.

16.3 HDR environment libraries

• Poly Haven (HDRI Haven): open HDR panos.
• HDRI Haven Pro: commercial.
• Maxime Roz, sIBL Archive.

16.4 Synthetic datasets for training

• Hypersim: 461 scenes with physically-correct intrinsic data.
• InteriorNet: indoor synthetic.
• OpenIllumination: lighting / inverse-rendering benchmark.
• Light Stage data (controlled academic releases).

17. Neural BRDF and Material Models

17.1 Neural BRDF (NeuMIP, Neural BTF)

Replace analytical BRDF with a small MLP. Captures complex anisotropic / structured surfaces (woven cloth, brushed metal). Memory + render trade-off vs traditional textures.

17.2 MaterialGAN, Differentiable BRDF

GAN generates SVBRDF maps from noise / photo. Differentiable BRDF allows joint material + lighting optimization.

17.3 NeRO, GS-IR, Relightable 3DGS

Per-point neural material parameters embedded in 3D representation. Renders + edits jointly with geometry.

17.4 Latent material space

Embed material library in a latent space; interpolate / mix. Used in tools like Adobe Substance Sampler.

17.5 Diffusion-generated PBR maps

TexGen, Material Anything, IntrinsicAnything: text-to-PBR or image-to-PBR via diffusion. Generate full material set (albedo + normal + roughness + metallic) from text or photo.

18. Evaluation

18.1 Quantitative metrics

• Reflectance MSE / PSNR / SSIM against ground truth (intrinsic decomposition).
• LMSE (Local Mean Squared Error): invariant to global scaling.
• WHDR (Weighted Human Disagreement Rate): agreement with human pairwise.
• Re-rendering error: re-render with estimated \((M, L)\); compare to image.

18.2 Perceptual metrics

• LPIPS: learned perceptual.
• User studies: side-by-side.
• HDR-VDP-3: for HDR relit content.

18.3 Multi-view consistency (for 3D / video relighting)

• Re-project relit images across views; compare consistency.
• Optical-flow-based temporal consistency for video.

18.4 Benchmarks

• IIW (Intrinsic Images in the Wild): pairwise human ordering.
• SAW (Shading Annotations in the Wild): shading discontinuity annotations.
• MIT Intrinsic Images: 16 objects with GT.
• OpenIllumination: full inverse-rendering benchmark.

19. Production Workflows (2026)

19.1 E-commerce product relighting

1. Capture: cross-polarized turntable or 4–8 flash.
2. Decompose into albedo + normal + roughness + metallic (\(\mathbf{RGB}{\to}\mathbf{X}\) / IntrinsicAnything).
3. Re-render in PBR engine (Blender Cycles, V-Ray) or use IC-Light for direct.
4. Output: relit product photos for any campaign / season.

19.2 Game asset pipeline

1. Photogrammetry capture (Reality Capture / Metashape).
2. Decimate mesh; bake to game-ready topology.
3. Recover albedo via cross-polarization or AI delighting.
4. Generate normal / roughness / metallic.
5. Author shader in engine (Unreal, Unity).

19.3 Portrait relighting (consumer / creative)

1. Single photo input.
2. SwitchLight or IC-Light for relighting.
3. Optional harmonization with new background (Relightful Harmonization).
4. Output: relit portrait.

19.4 Digital human (premium)

1. Light stage capture (Beeble, Disney, ICT).
2. Recover full reflectance field + PBR maps.
3. Build CG asset.
4. Relight to fit any scene via reflectance-field synthesis or PBR rendering.

19.5 AR insertion

1. Estimate env map from device camera (ARKit / ARCore / Vision Pro).
2. Pre-filter for IBL.
3. Render virtual object with estimated lighting.
4. Composite with shadow.

19.6 VFX composite

1. HDR env map captured on set (chrome ball + bracketed photos).
2. CG element rendered with IBL from env map.
3. Plate graded to canonical color space (ACEScg).
4. Composite in Nuke / Fusion with grade match.

20. Frontier 2025–2026 and Open Problems

20.1 Diffusion-based relighting mainstream

IC-Light, SwitchLight, Relightful Harmonization productionized. Generative photography in commercial tools (Adobe Firefly, Photoshop Generative Fill, Topaz).

20.2 Multi-view consistent relighting

Open challenge. 2025–26 research: condition diffusion on multiple views, or relight in 3DGS / NeRF directly.

20.3 Physical fidelity vs plausibility

Diffusion gives plausible; physics-based gives correct. Hybrid systems (physics-based base + diffusion polish) emerging.

20.4 Real-time relighting

• Distilled IC-Light / consistency-model variants.
• Mobile GPU inference.
• AR glasses (Vision Pro 2, Quest 3+).

20.5 4D relighting (video)

Per-frame relighting with temporal coherence. Active research direction.

20.6 Inverse rendering with full GI

Most methods ignore indirect illumination. NeRO + path tracing extensions handle some; full-quality indirect remains open.

20.7 Open research questions

• Physically-correct yet diffusion-grade results.
• Long-horizon temporal consistency.
• User-controllable lighting beyond text / direction.
• Material editing via text / sketch.
• Generalizing to unusual materials (subsurface, anisotropic).

21. Production Stack 2026

★ 2026 SOTA update — Diffusion-based relighting (portrait / object / scene)

• IC-Light v2 (FLUX-based): FLUX-backbone successor to IC-Light. 16ch VAE + native high resolution yield far better detail preservation than the SD1.5 v1; text-conditioned and background-conditioned relighting. Underlying method scored a rare 10/10/10/10 at ICLR 2025.

★ 2026 SOTA update — Neural inverse + forward rendering (video), relighting / material editing / object insertion

• DiffusionRenderer: Unifies inverse rendering (G-buffer estimation from real video) and forward rendering (photoreal images from G-buffers) in one video-diffusion framework, enabling de-lighting and relighting from a single video. Later integrated with Cosmos Predict-1 for sharper temporally-consistent results.

★ 2026 SOTA update — Portrait relighting (diffusion)

• SynthLight: Frames portrait relighting as a re-rendering problem; trained on a PBR-synthesized dataset of 3D head assets under varying HDR lighting, with multi-task training on unlabeled real portraits plus CFG-based inference to preserve identity. Produces specular highlights and cast shadows.

★ 2026 SOTA update — Portrait performance (video) relighting

• Lux Post Facto: Conditional video-diffusion portrait relighting with a lighting-injection mechanism, trained on a hybrid dataset of static-expression OLAT captures plus in-the-wild performance videos. State-of-the-art photorealism and temporal consistency without paired video data.

★ 2026 SOTA update — Video relighting (diffusion)

• RelightVid: Flexible video relighting accepting background video, text, or environment-map conditions. Trained on in-the-wild videos with illumination augmentations plus its LightAtlas rendered dataset; achieves high temporal consistency without intrinsic decomposition.

★ 2026 SOTA update — Monocular human relighting + harmonization (image & video)

• Comprehensive Relighting: First all-in-one model to both control and harmonize lighting for humans with arbitrary body parts (portrait, full-body, multi-person) from any scene. Repurposes a pretrained diffusion prior in a coarse-to-fine framework with an unsupervised temporal lighting model for video coherence.

★ 2026 SOTA update — Relightable 3D Gaussian inverse rendering

• IRGS (Inter-Reflective Gaussian Splatting): Applies the full rendering equation (no simplification) on 2D Gaussians and computes incident radiance on-the-fly via differentiable 2D Gaussian ray tracing, correctly modeling visibility and indirect/inter-reflected light for more accurate material and lighting decomposition than prior 3DGS inverse-rendering methods.

Appendix A: Twenty-Five Things to Know

1. Rendering equation (Kajiya 1986).
2. BRDF properties: non-negative, reciprocity, energy conservation.
3. Microfacet form: \(f_r = FGD / (4\cos\theta_i\cos\theta_o)\).
4. GGX is the industry-standard \(D\).
5. Schlick Fresnel: \(F = F_0 + (1 - F_0)(1 - \cos\theta)^5\).
6. Disney Principled BRDF is the artist standard.
7. PBR maps: albedo, normal, roughness, metallic, +AO.
8. Intrinsic image: \(I = R \cdot S\) (reflectance × shading).
9. Retinex: sharp log \(I\) edges = reflectance; smooth = shading.
10. IIW + CGIntrinsics are the standard datasets.
11. Photometric stereo (Woodham): \(\mathbf{I} = L \cdot (\rho\, \mathbf{n})\).
12. Cross-polarized capture removes specular for clean albedo.
13. Light stage = hundreds of LEDs + reflectance field.
14. Reflectance field: relight is matrix-vector multiply in lighting basis.
15. Spherical Harmonics for low-frequency env lighting (9 coefficients standard).
16. Spherical Gaussians for higher-frequency / specular.
17. Pre-filtered cubemap + split-sum for real-time IBL (UE4-style).
18. Sun et al. (2019): single-image deep portrait relighting baseline.
19. Total Relighting: foreground matting + relighting end-to-end.
20. SwitchLight / IC-Light: 2024 diffusion-based relighting wave.
21. Relightable 3D Gaussians / GS-IR: 3DGS + relighting.
22. ARKit / ARCore env probes for mobile AR lighting.
23. LED volumes (Stagecraft): real-time IBL on virtual production.
24. Photogrammetry \(\to\) delight \(\to\) PBR maps (Quixel pattern).
25. Diffusion gives plausible; physics gives correct (hybrid emerging).

Appendix B: Decision Tree — "Which Relighting Method?"

1. Single-image portrait, fast result? \(\to\) SwitchLight (commercial) or IC-Light (open).
2. Composite portrait into a new background? \(\to\) Relightful Harmonization + IC-Light FBC.
3. Single-image object, e-commerce? \(\to\) IC-Light text-conditioned or PBR re-render via IntrinsicAnything.
4. Single-image SVBRDF estimation? \(\to\) Deschaintre / \(\mathbf{RGB}{\to}\mathbf{X}\) / IntrinsicAnything.
5. Multi-view scene with arbitrary lighting? \(\to\) Relightable 3D Gaussians / NeRO / GS-IR.
6. Premium digital human (film)? \(\to\) Light stage scan + PBR engine.
7. Photogrammetry asset for game? \(\to\) Cross-polarized capture + bake to PBR maps (Quixel pattern).
8. AR insertion of virtual object? \(\to\) ARKit / ARCore env probe + IBL in RealityKit / Unity.
9. VFX virtual production? \(\to\) LED volume (Stagecraft) + Unreal.
10. Live video relighting? \(\to\) Distilled IC-Light or temporal-consistent diffusion (research).

Appendix C: Year-by-Year Milestones

• 1971: Land's Retinex.
• 1980: Woodham's photometric stereo.
• 1986: Kajiya's rendering equation.
• 1990s: Cook-Torrance, Marschner hair BRDF; HDR / Radiance from Greg Ward.
• 2000: Debevec light stage + reflectance fields; Ramamoorthi-Hanrahan SH irradiance.
• 2003–2007: Pre-filtered IBL, GGX standardized; Disney Principled BRDF (Brent Burley) 2012.
• 2012–2015: Quixel Megascans launches; UE4 split-sum IBL.
• 2014–2018: Intrinsic Images CNN era (Bell, Li & Snavely, Direct Intrinsics, IIW).
• 2018: Single-image SVBRDF from flash (Deschaintre).
• 2019: Sun et al. deep portrait relighting (SIGGRAPH).
• 2019–2022: NeRD / NeRF-OSR / PhySG / NeRO — NeRF + inverse rendering.
• 2021: Total Relighting; Mandalorian's Stagecraft popularizes virtual production.
• 2023–2024: IC-Light (Lvmin Zhang); SwitchLight (Beeble.ai); Relightful Harmonization (Google); \(\mathbf{RGB}{\to}\mathbf{X}\) (Adobe); Relightable 3D Gaussians; GS-IR; IntrinsicAnything.
• 2025: Diffusion-based relighting mainstream in commercial tools (Adobe Firefly, Photoshop Generative Fill); 4D-GS + relight; Vision Pro / Quest 3 high-fidelity AR insertion.
• 2026: Real-time consistency-distilled relighting on mobile; multi-view consistent diffusion relighting; integrated diffusion + physical hybrid pipelines for VFX.