Optimal Illumination for Image and Video Relighting
Francesc Moreno-Noguer Shree K. Nayar Peter N. Belhumeur
Department of Computer Science – Columbia University
European Conference on CVMP November 2005, London, UK
Relighting Static Objects
Relighting Static Objects
Relighting Static Objects
Relighting Static Objects
Relighting Static Objects
Relighting Static Objects
Relighting Static Objects
Relighting Static Objects
Relighting Static Objects
Relighting Static Objects
Relighting Static Objects
Relighting Static Objects
Relighting Static Objects
Relighting Static Objects
Relighting Static Objects
Relighting Static Objects
Relighting Static Objects
Relighting Static Objects
Relighting Static Objects
Relighting Static Objects
Relighting Static Objects
Relighting Static Objects
Relighting Static Objects
Relighting Static Objects
Relighting Static Objects
?
Relighting Static Objects
?
x x xx x x
x x x x x
x x x x x
x x x x x
x
x
x
Relighting Static Objects
x x xx x x
x x x x x
x x x x x
x x x x x
x
x
x
Relighting Static Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
?
Relighting Moving Objects
?
x x xx x x
x x x x x
x x x x x
x x x x x
x
x
x
Relighting Moving Objects
x x xx x x
x x x x x
x x x x x
x x x x x
x
x
x
Relighting Moving Objects
x x xx x x
x x x x x
x x x x x
x x x x x
x
x
x
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
Relighting Moving Objects
= I
= L
Relighting Moving Objects
= Ialign
= L
Relighting Moving Objects
Lnew
Relighting Moving Objects
= Ialign
= L
Lnew
?Inew
Relighting Moving Objects
= Ialign
= L
Relighting Moving Objects
Inew= Ialign pinv(L) Lnew
1 + 2 3 4+ +=
Inew= Ialign pinv(L) Lnew
Relighting Moving Objects
=
Relighting Moving Objects
Inew= Ialign pinv(L) Lnew
1 + 2 3 4+ +=
What is the optimal lighting?
What is the optimal lighting?
Lopt ?
Sources of error in video relighting
Sub-basis error
Ground Truth
Sources of error in video relighting
Sub-basis error
Ground Truth Rendered 3 basis
Error 3 basis
Sources of error in video relighting
Sub-basis error
Ground Truth Rendered 3 basis
Error 3 basis
Sources of error in video relighting
Sub-basis error
Ground Truth Rendered 3 basis
Error 3 basis
Rendered 6 basis
Error 6 basis
Sources of error in video relighting
Sub-basis error
Ground Truth Rendered 3 basis
Error 3 basis
Rendered 6 basis
Error 6 basis
Sources of error in video relighting
Alignment errorReference
frame
Sources of error in video relighting
Alignment errorReference
frameFrame
t+1
Sources of error in video relighting
Alignment errorReference
frame
Alignment errors
Frame t+1
Sources of error in video relighting
Alignment errorReference
frame
Alignment errors
Frame t+1
Frame t+2
Sources of error in video relighting
Alignment errorReference
frame
Alignment errors
Frame t+1
Frame t+2
Sources of error in video relighting
Alignment errorReference
frame
Alignment errors
Frame t+1
Frame t+2
Frame t+3
Sources of error in video relighting
Alignment errorReference
frame
Alignment errors
Frame t+1
Frame t+2
Frame t+3
Sources of error in video relighting
Alignment errorReference
frame
Alignment errors
Frame t+1
Frame t+2
Frame t+3
Frame t+4
Sources of error in video relighting
Alignment errorReference
frame
Alignment errors
Frame t+1
Frame t+2
Frame t+3
Frame t+4
Sources of error in video relighting
Alignment errorReference
frame
Alignment errors
Frame t+1
Frame t+2
Frame t+3
Frame t+4
Frame t+5
Sources of error in video relighting
Alignment errorReference
frame
Alignment errors
Frame t+1
Frame t+2
Frame t+3
Frame t+4
Frame t+5
Sources of error in video relighting
Orientation error
Sources of error in video relighting
Orientation error
Sources of error in video relighting
Reference frame
Orientation error
Sources of error in video relighting
Reference frame
Perfectly aligned frame
Orientation error
Sources of error in video relighting
Reference frame
Error afterperfect
alignment
Perfectly aligned frame
Orientation error
Sources of error in video relighting
Sub-basis error
Alignment error
Orientation error
Criterion for selecting the OLB
Minimize the sub-basis error
BUT
Keep the number of reference images to a minimum
Object independent lighting bases
Spherical Harmonic basis (SHLB)
Object independent lighting bases
Spherical Harmonic basis (SHLB)
Fourier basis (FLB)
Object independent lighting bases
Fourier basis (FLB)
Haar basis (HaLB)
Spherical Harmonic basis (SHLB)
Object dependent lighting basis
Optimal Lighting Basis (OLB)
• Computed as a simple calibration procedure before acquisition
• Initial acquisition of the images of the still object under single light sources
• Compute SVD over the matrix of images
• Computed as a simple calibration procedure before acquisition
• Initial acquisition of the images of the still object under single light sources
• Compute SVD over the matrix of images
...A=
A=UDWT
Object dependent lighting basis
A=UDWT
...A=
Object dependent lighting basis
A=UDWT
...A=
...U=
Object dependent lighting basis
A=UDWT
...A=
...
...
U=
W = T
Object dependent lighting basis
A=UDWT
...A=
...
...
...
U=
W = T
Lopt=
Object dependent lighting basis
Synthetic results
Optimal Lighting Basis (OLB)
Spherical Hamonic (SHLB)
Synthetic results
Optimal Lighting Basis (OLB)
Spherical Hamonic (SHLB)
Optimal Lighting Basis (OLB)
Fourier (FLB)
Haar (HaLB)
Synthetic results – Static objects
Ground Truth
Synthetic results – Static objects
Fourier 16 basis
Error
Fourier 16 basis
Ground Truth
Synthetic results – Static objects
Fourier 16 basis OLB 16 basis
Error
Fourier 16 basis
Error
OLB 16 basis
Ground Truth
Synthetic results – Static objects
Ground Truth
Synthetic results – Static objects
Haar 3 basis
Error
Haar 3 basis
Ground Truth
Synthetic results – Static objects
Haar 3 basis OLB 3 basis
Error
Haar 3 basis
Error
OLB 3 basis
Ground Truth
Synthetic results – Static objects
Ground Truth
Synthetic results – Static objects
Sph. Harm.
7 basis
Error
Sph. Harm. 7 basis
Ground Truth
Synthetic results – Static objects
Sph. Harm.
7 basis
OLB 7 basis
Error
Sph. Harm. 7 basis
Error
OLB 7 basis
Ground Truth
Synthetic results – Static objects
Ground Truth
Synthetic results – Static objects
Sph. Harm. 3 basis
Error
Sph. Harm. 3 basis
Ground Truth
Synthetic results – Static objects
Sph. Harm. 3 basis OLB 3 basis
Error
Sph. Harm. 3 basis
Error
OLB 3 basis
Ground Truth
Synthetic results – Static objects
Synthetic results – Static objects
Dragon Example:
Synthetic results – Static objects
8 OLB = 8 x 1.6 SHLB 13 SHLBDragon Example:
Synthetic results – Static objects
8 OLB = 8 x 1.6 SHLB 13 SHLB8 OLB = 8 x 1.9 SHLB 15 FLB
Dragon Example:
Synthetic results – Static objects
Dragon Example: 8 OLB = 8 x 1.6 SHLB 13 SHLB8 OLB = 8 x 1.9 SHLB 15 FLB8 OLB = 8 x 3.2 SHLB 25 HaLB
Frame #50
Synthetic results – Moving objects
Frame #1
OLB vs. SHLB
Frame #50
Synthetic results – Moving objects
Frame #1
9 OLB
OLB vs. SHLB
Frame #50
Synthetic results – Moving objects
Frame #1
9 OLB 16 SHLB
OLB vs. SHLB
Frame #50
Synthetic results – Moving objects
Frame #1
OLB vs. Fourier & Haar
Frame #50
Synthetic results – Moving objects
Frame #1
3 OLB
OLB vs. Fourier & Haar
Frame #50
Synthetic results – Moving objects
Frame #1
3 OLB 5 FLB
OLB vs. Fourier & Haar
Frame #50
Synthetic results – Moving objects
Frame #1
3 OLB 5 FLB
OLB vs. Fourier & Haar
16 HaLB
Real Experiments - Setup
Real Experiments - Procedure
OLB computation
1.1. Static scene, single lights1.2. SVD OLB
Video adquisition
2. Moving scene, OLB
Postprocessing 3.1. Alignment3.2. Relighting
Real Experiments – Tennis Ball
Real Experiments – Tennis Ball
Real Experiments – Face
Real Experiments – Face
Real Experiments – Face
Real Experiments – Room corner
Real Experiments – Room corner
Real Experiments – Room corner
Real Experiments – Room corner
Conclusions
• Proposed an “object-dependent” lighting basis appropriate for relighting tasks.
• Minimizes the number of reference images needed for relighting.
• Effective method for video relighting.
• Proposed an “object-dependent” lighting basis appropriate for relighting tasks.
• Minimizes the number of reference images needed for relighting.
• Effective method for video relighting.
Future work
• Use faster cameras.
• Use “motion-dependent” relighting bases.
• Use faster cameras.
• Use “motion-dependent” relighting bases.
Frame t+1
Frame t+2
Frame t+3
Frame t+4
Frame t+5
Frame t
Slow motion
More accurate results
Future work
• Use faster cameras.
• Use “motion-dependent” relighting bases.
• Use faster cameras.
• Use “motion-dependent” relighting bases.
Fast motion
Frame t+1
Frame t+2
Frame t+3
Frame t+4
Frame t+5
Frame t
Slow motion
Reduce alignment & orientation
errors
More accurate results
Acknowledgments
• Ko Nishino for his useful comments.
• Anne Fleming for her help in the edition of the videos.
• All the people from the “Laboratory for the Study of Visual Appearance” and “Columbia Automated Vision Environment's” at Columbia University.
• Cyberware for the 3D models.
• Funding Sponsors: • Spanish Ministry of Science & Technology• National Science Foundation
• Ko Nishino for his useful comments.
• Anne Fleming for her help in the edition of the videos.
• All the people from the “Laboratory for the Study of Visual Appearance” and “Columbia Automated Vision Environment's” at Columbia University.
• Cyberware for the 3D models.
• Funding Sponsors: • Spanish Ministry of Science & Technology• National Science Foundation