Surface Light Fieldsfor 3D Photography
Daniel N. WoodUniversity of Washington
SIGGRAPH 2001 Course
Collaborators(and co-authors on SIGGRAPH 2000
paper)Daniel Azuma Wyvern Aldinger
Brian Curless Tom DuchampDavid Salesin Werner Stuetzle
University of Washington
Outline
1. Surface light field representation
2. Compression primer
3. Surface light fields for 3D photography• With details of compression• And a preliminary look at a new
compression algorithm
Surface light fields
Lumisphere-valued “texture” maps
Lumisphere
Surface light fields in flatland
s
Surface light fields in flatland
s
Surface light fields in flatland
s
Compression primer
• Singular value decomposition(or principal components analysis)
– Handling color
– Using regions
– Reflection parameterization
• Vector quantization
• Others… (Wavelets, DCT)
Singular value decomposition
SLF
U VT
SVD in two matrices
EigenTextures
EigenLumispheres
SLF
First eigenvectors
EigenTextures
EigenLumispheres
SLF First eigentexture and corresponding first eigenlumisphere
Truncated SVD
Outer product of first eigen-texture and first eigen-lumisphere is closest rank 1 (separable) matrix.
=
Eigen-lumisphere
Eig
en
-textu
re
~~
Handling color
Separatematrices
Colors insurface texture
(columns)
Colors indirections
(rows)
Handling color
Separatematrices
Nishino et al.Wood et al.Chen et al.
Colors insurface texture
(columns)
Colors indirections
(rows)
SVD in color
EigenTextures
EigenLumispheres
SLF
Zooming in…
EigenTextures
EigenLumispheres
SLF
Zoom into important vectors
... ...SLF
Reconstruction using SVD
Original Rank 1 Rank 7 (1:20)
Surface light field structure
Rows are points on surface
What are the columns? And, can they be made more coherent?
Surface light field structure
Rows are points on surface
Increasing column coherence:
1. Break into regions, and / or
2. Use reflection parameterization
Separate SVD for regions
EigenTextures
EigenLumispheres
EigenTextures
EigenLumispheres
Reconstruction using regions
Original One region(Rank 5)
Two regions(Rank 5)
Reflection reparameterization
Reflection reparameterization
Reflection reparameterization
Reflection in flatland*
*sort of
Un-reflected Reflected
Reflection doesn’t happenin the plane
Un-reflected Reflected
Reflection in 3 space
Un-reflected
Reflected
Reflection in “flatland”
Un-reflected Reflected
Original Rank 5 Original Rank 5
Other compression strategies
• Discrete cosine transform[ Miller et al. 1999
– Eurographics Workshop on Rendering]
• Vector quantization[ Wood et al. 2000 - SIGGRAPH ]
• Wavelet decomposition[ Magnor and Girod 2000 - SPIE VCIP]
Vector quantization (unreflected)
Uncompressed Codebook Vector-quantized
Vector quantization (reflected)
Uncompressed Codebook Vector-quantized
Surface light fields for 3D photography
(SIGGRAPH 2000)Goals
Rendering and editing
InputsPhotographs and geometry
RequirementsEstimation and compression
Overview
Dataacquisition
Estimationand
compressionRendering
Editing
Overview
Dataacquisition
Estimationand
compressionRendering
Editing
Scan and reconstruct geometry
Reconstructed geometryRange scans(only a few shown . . .)
Take photographs
Camera positions Photographs
Register photographs to geometry
GeometryPhotographs
Register photographs to geometry
User selected correspondences (rays)
Parameterizing the geometry
Base mesh Scanned geometry
Map
Sample base mesh faces
Base mesh Detailed geometry
Assembling data lumispheres
Data lumisphere
Overview
Dataacquisition
Estimationand
compressionRendering
Editing
Pointwise fairing
Faired lumisphereData lumisphere
Pointwise fairing results
Input photograph Pointwise faired(177 MB)
Pointwise fairing
Many input data lumispheres Many faired lumispheres
Compression
Small set of prototypes
Compression / Estimation
Small set of prototypesMany input data lumispheres
Median removal
+
Reflected
Median(“diffuse”)
Median-removed(“specular”)
+
Median removal
Median values Specular Result
Function quantization
Codebook of lumispheres
Input data lumisphere
Lloyd iteration
Input data lumispheres
Lloyd iteration
Codeword
Lloyd iteration
Perturb codewords to create larger codebook
Lloyd iteration
Form clusters around each codeword
Lloyd iteration
Optimize codewords based on clusters
Lloyd iteration
Create new clusters
Function quantization results
Input photograph Function quantized(1010 codewords, 2.6 MB)
Principal function analysis
Subspace of lumispheres
Input data lumisphere
Prototype lumisphere
Principal function analysis
Approximating subspace
Prototype lumisphere
Principal function analysis
Principal function analysis
Principal function analysis
++
+
Median
PFA decomposition
Principal function analysis
+ + …
Median
PFA decomposition
=Final
approximation
Principal function analysis results
Input photograph PFA compressed(Order 5 - 2.5 MB)
Compression comparison
Pointwise fairing(177 MB)
Function quantization(2.6 MB)
Principal functionanalysis (2.5 MB)
Rewind…what we didn’t want to do
Instead, a middle ground
Regularly sampled directions,but not all there.
New method:Principal components with missing
data
SLF with missing data
Hallucinatedata
Use either fairing or pair-wise present covariance matrix to fill holes
Hole-filled SLF
New method:Principal components with missing
data
SLF with missing data Low rankapproximation
Hallucinatedata
Find best low rank
approximation
Hole-filled SLF
New method:Principal components with missing
data
SLF with missing data Low rankapproximation
Hallucinatedata
Improvehallucination
Preliminary comparison
Principal function analysisOrder 3, RMS error 26.9
SVD with missing dataOrder 3, RMS error 26.1
Overview
Dataacquisition
Estimationand
compressionRendering
Editing
Interactive rendererscreen capture
Overview
Dataacquisition
Estimationand
compressionRendering
Editing
Lumisphere filtering
Original surface light field Glossier coat
Lumisphere filtering
Rotating the environment
Original surface light field Rotated environment
Deformation
Original Deformed
Deformation
Summary
1. Estimation and compression• Function quantization• Principal function analysis
2. Rendering• From compressed representation• With view-dependent level-of-detail
3. Editing• Lumisphere filtering• Geometric deformations and transformations
Future work
• Better geometry-to-image registration
• Derive geometry from images
• More complex surfaces (mirrored, refractive, fuzzy…) under more complex illumination
Acknowledgements
• Marc Levoy and Pat Hanrahan– (Thanks for the use of the Stanford
Spherical Gantry)
• Michael Cohen and Richard Szeliski
• National Science Foundation
Nearly the end
For more information
http://graphics.cs.washington.edu/projects/slf
Talks, papers, … and raw data.