1

Online Construction of Surface Light Fields

By Greg Coombe, Chad Hantak, Anselmo Lastra, and Radek Grzeszczuk

2

* *

Image-Based Modeling

Benefits: Photorealistic content Modeling by acquisition

Difficulties: Sampling issues Lack of feedback

3

Motivation

Casual Capture

Pick up a video camera, wave it around the model, and capture reflectance model

4

Surface Light Field

A surface light field is a representation of the appearance of a model with known geometry and static lighting:

( , , , )f u v viewing direction

surface position

(u,v)

(θ,Φ)

5

Our system

Incremental system for online capture, construction, and rendering of surface light fields

Each image is processed as it is captured Interactive feedback

Guide user to undersampled regions

Data-driven heuristic Structured method for handling missing data

6

Previous Work Regular parameterizations light field data [Levoy96, Gortler96] Sparse, scattered data [Debevec96, Debevec98, Buehler01] BRDF capture systems [Dana99, Lafortune97, Marschner99,

Debevec01, Gardner03] Function fitting

Lafortune BRDF [McAllister02], Torrance-Sparrow BRDF [Sato97], Clusters of BRDFs [Lensh01], Homomorphic Factorization [McCool?], Bi-quadratic polynomials [Malzbender01]

Online Methods Fixed viewpoint, progressive refinement [Matusik04] Adaptive Meshing of light field [Schirmacher99] Streaming non-linear optimization [Hillesland04]

Data Mining [Brand02, Roweis97]

7

SLFs using PCA

Represent 4D function as a matrix Use Principal Component Analysis [Noshino01,

Chen02] For each surface patch, break the 4D function into two

2D functions Store the principal components in texture maps

),(*),(),,,( vuhgvuf rank

8

SLFs using PCA

Problems: Requires that the entire data set be available

at once. Difficult to locate undersampled regions Requires recomputation when new images are

added

SVD*

9

Online SVD

Online SVD [Brand02] is a incremental PCA Update output matrices one sample at a time Advantages:

Never store the entire data matrix at once Stream images

Online SVD*...

10

Online SVD Algorithm

( )

0

diag s m

p

[ ', ', ']diagonalize U S V

6. The rotations are computed by re-diagonalizing this matrix

-p a Um3. Compute the orthogonal component

pTm U a

2. Project new image samples onto eigenspace

j 4. If ||p|| is less than a threshold, then we just compute the rotations and update the eigenspace.

U’ = URu V’ = VRv 5. Otherwise, the current rank r is insufficient, so we

append a new eigenvector.

U’ = [U; m]Ru V’ = VRv

1. Existing rank-r PCATA USV

11

Error

Reference PCA Online SVD

Rank

12

Imputation

Missing data in red

Imputed

Matrix factorization approach requires fully resampled data matrices

Missing data is a common problem Occlusion, meshing errors

Imputation is the process of filling in these holes with “reasonable” guesses Better than zeros or mean

Use the current Online SVD approximation to generate these missing values In practice, need about 5-10 initial

images, and 50%+ coverage Further details in [Brand03]

13

Automatic Feedback

Aid the user in capturing surface light fields Direct attention towards

undersampled areas Use a data-driven

quality heuristicE = sqrt( e0^2 + e1^2)

e0 – variation over hemisphere

e1 – variation over surface

14

System

Real Time PoseEstimation

Resampling* Online SVDVisibility*

QualityHeuristic

VideoCamera

Rendering

Capture PC Render PC

* From Intel’s OpenLF system

15

Results

Captured image SLF before incorporating new image

SLF after incorporating new image

16

Results

17

Timing

18

Conclusion

We present a method with Incremental construction Data-driven quality heuristic

Online SVD is well-suited for streaming model Combine distribution and solution Requires only one pass over data

19

Future Work

We are still a long ways from “Casual Capture” Geometry must be known a priori Fixed illumination conditions

Implementation issues Real-time pose estimation errors

20

21

Online SVD

Problem: Multiple small rotations can accumulate error

Brand proposes splitting output matrices to avoid accumulating error

Advantages: Due to small working set, most data in cache (fast)

TSVD

TSVD

VSVUUA

USVA

''

22

Higher-Dimensional Problems

We would like to extend Online SVD to higher-order factorizations Moveable light source

Introduces another dimension of data

Requires tensor product expansion

),(*),(*),(),,,,,( 22112211 kvuhgvuf

View maps Surface maps Light maps

23

2. Convergence

Averaged over twenty random vertices

24

3. Sensitivity

25

Imputation

Incomplete sample: (-1, ??)

Existing SVD approximation

Impute using SVD