Page 1
Optimal Illumination for Image and Video Relighting
Francesc Moreno-Noguer Shree K. Nayar Peter N. Belhumeur
Department of Computer Science – Columbia University
European Conference on CVMP November 2005, London, UK
Page 6
Relighting Static Objects
Page 7
Relighting Static Objects
Page 8
Relighting Static Objects
Page 9
Relighting Static Objects
Page 10
Relighting Static Objects
Page 11
Relighting Static Objects
Page 12
Relighting Static Objects
Page 13
Relighting Static Objects
Page 14
Relighting Static Objects
Page 15
Relighting Static Objects
Page 16
Relighting Static Objects
Page 17
Relighting Static Objects
Page 18
Relighting Static Objects
Page 19
Relighting Static Objects
Page 20
Relighting Static Objects
Page 21
Relighting Static Objects
Page 22
Relighting Static Objects
Page 23
Relighting Static Objects
Page 24
Relighting Static Objects
Page 25
Relighting Static Objects
Page 26
Relighting Static Objects
Page 27
Relighting Static Objects
Page 28
Relighting Static Objects
Page 29
Relighting Static Objects
Page 30
Relighting Static Objects
Page 31
?
Relighting Static Objects
Page 32
?
x x xx x x
x x x x x
x x x x x
x x x x x
x
x
x
Relighting Static Objects
Page 33
x x xx x x
x x x x x
x x x x x
x x x x x
x
x
x
Relighting Static Objects
Page 34
Relighting Moving Objects
Page 35
Relighting Moving Objects
Page 36
Relighting Moving Objects
Page 37
Relighting Moving Objects
Page 38
Relighting Moving Objects
Page 39
Relighting Moving Objects
Page 40
Relighting Moving Objects
Page 41
Relighting Moving Objects
Page 42
Relighting Moving Objects
Page 43
Relighting Moving Objects
Page 44
Relighting Moving Objects
Page 45
Relighting Moving Objects
Page 46
Relighting Moving Objects
Page 47
Relighting Moving Objects
Page 48
Relighting Moving Objects
Page 49
Relighting Moving Objects
Page 50
Relighting Moving Objects
Page 51
Relighting Moving Objects
Page 52
Relighting Moving Objects
Page 53
Relighting Moving Objects
Page 54
Relighting Moving Objects
Page 55
Relighting Moving Objects
Page 56
Relighting Moving Objects
Page 57
Relighting Moving Objects
Page 58
Relighting Moving Objects
Page 59
?
Relighting Moving Objects
Page 60
?
x x xx x x
x x x x x
x x x x x
x x x x x
x
x
x
Relighting Moving Objects
Page 61
x x xx x x
x x x x x
x x x x x
x x x x x
x
x
x
Relighting Moving Objects
Page 62
x x xx x x
x x x x x
x x x x x
x x x x x
x
x
x
Relighting Moving Objects
Page 63
Relighting Moving Objects
Page 64
Relighting Moving Objects
Page 65
Relighting Moving Objects
Page 66
Relighting Moving Objects
Page 67
Relighting Moving Objects
Page 68
Relighting Moving Objects
Page 69
Relighting Moving Objects
Page 70
Relighting Moving Objects
Page 71
Relighting Moving Objects
Page 72
Relighting Moving Objects
Page 73
Relighting Moving Objects
Page 74
Relighting Moving Objects
Page 75
Relighting Moving Objects
Page 76
Relighting Moving Objects
Page 77
Relighting Moving Objects
Page 78
Relighting Moving Objects
Page 79
Relighting Moving Objects
Page 80
Relighting Moving Objects
Page 81
Relighting Moving Objects
Page 82
Relighting Moving Objects
Page 83
= I
= L
Relighting Moving Objects
Page 84
= Ialign
= L
Relighting Moving Objects
Page 85
Lnew
Relighting Moving Objects
= Ialign
= L
Page 86
Lnew
?Inew
Relighting Moving Objects
= Ialign
= L
Page 87
Relighting Moving Objects
Inew= Ialign pinv(L) Lnew
Page 88
1 + 2 3 4+ +=
Inew= Ialign pinv(L) Lnew
Relighting Moving Objects
Page 89
=
Relighting Moving Objects
Inew= Ialign pinv(L) Lnew
1 + 2 3 4+ +=
Page 90
What is the optimal lighting?
Page 91
What is the optimal lighting?
Lopt ?
Page 92
Sources of error in video relighting
Sub-basis error
Ground Truth
Page 93
Sources of error in video relighting
Sub-basis error
Ground Truth Rendered 3 basis
Error 3 basis
Page 94
Sources of error in video relighting
Sub-basis error
Ground Truth Rendered 3 basis
Error 3 basis
Page 95
Sources of error in video relighting
Sub-basis error
Ground Truth Rendered 3 basis
Error 3 basis
Rendered 6 basis
Error 6 basis
Page 96
Sources of error in video relighting
Sub-basis error
Ground Truth Rendered 3 basis
Error 3 basis
Rendered 6 basis
Error 6 basis
Page 97
Sources of error in video relighting
Alignment errorReference
frame
Page 98
Sources of error in video relighting
Alignment errorReference
frameFrame
t+1
Page 99
Sources of error in video relighting
Alignment errorReference
frame
Alignment errors
Frame t+1
Page 100
Sources of error in video relighting
Alignment errorReference
frame
Alignment errors
Frame t+1
Frame t+2
Page 101
Sources of error in video relighting
Alignment errorReference
frame
Alignment errors
Frame t+1
Frame t+2
Page 102
Sources of error in video relighting
Alignment errorReference
frame
Alignment errors
Frame t+1
Frame t+2
Frame t+3
Page 103
Sources of error in video relighting
Alignment errorReference
frame
Alignment errors
Frame t+1
Frame t+2
Frame t+3
Page 104
Sources of error in video relighting
Alignment errorReference
frame
Alignment errors
Frame t+1
Frame t+2
Frame t+3
Frame t+4
Page 105
Sources of error in video relighting
Alignment errorReference
frame
Alignment errors
Frame t+1
Frame t+2
Frame t+3
Frame t+4
Page 106
Sources of error in video relighting
Alignment errorReference
frame
Alignment errors
Frame t+1
Frame t+2
Frame t+3
Frame t+4
Frame t+5
Page 107
Sources of error in video relighting
Alignment errorReference
frame
Alignment errors
Frame t+1
Frame t+2
Frame t+3
Frame t+4
Frame t+5
Page 108
Sources of error in video relighting
Orientation error
Page 109
Sources of error in video relighting
Orientation error
Page 110
Sources of error in video relighting
Reference frame
Orientation error
Page 111
Sources of error in video relighting
Reference frame
Perfectly aligned frame
Orientation error
Page 112
Sources of error in video relighting
Reference frame
Error afterperfect
alignment
Perfectly aligned frame
Orientation error
Page 113
Sources of error in video relighting
Sub-basis error
Alignment error
Orientation error
Page 114
Criterion for selecting the OLB
Minimize the sub-basis error
BUT
Keep the number of reference images to a minimum
Page 115
Object independent lighting bases
Spherical Harmonic basis (SHLB)
Page 116
Object independent lighting bases
Spherical Harmonic basis (SHLB)
Fourier basis (FLB)
Page 117
Object independent lighting bases
Fourier basis (FLB)
Haar basis (HaLB)
Spherical Harmonic basis (SHLB)
Page 118
Object dependent lighting basis
Optimal Lighting Basis (OLB)
• Computed as a simple calibration procedure before acquisition
• Initial acquisition of the images of the still object under single light sources
• Compute SVD over the matrix of images
• Computed as a simple calibration procedure before acquisition
• Initial acquisition of the images of the still object under single light sources
• Compute SVD over the matrix of images
...A=
Page 119
A=UDWT
Object dependent lighting basis
Page 120
A=UDWT
...A=
Object dependent lighting basis
Page 121
A=UDWT
...A=
...U=
Object dependent lighting basis
Page 122
A=UDWT
...A=
...
...
U=
W = T
Object dependent lighting basis
Page 123
A=UDWT
...A=
...
...
...
U=
W = T
Lopt=
Object dependent lighting basis
Page 124
Synthetic results
Optimal Lighting Basis (OLB)
Spherical Hamonic (SHLB)
Page 125
Synthetic results
Optimal Lighting Basis (OLB)
Spherical Hamonic (SHLB)
Optimal Lighting Basis (OLB)
Fourier (FLB)
Haar (HaLB)
Page 126
Synthetic results – Static objects
Ground Truth
Page 127
Synthetic results – Static objects
Fourier 16 basis
Error
Fourier 16 basis
Ground Truth
Page 128
Synthetic results – Static objects
Fourier 16 basis OLB 16 basis
Error
Fourier 16 basis
Error
OLB 16 basis
Ground Truth
Page 129
Synthetic results – Static objects
Ground Truth
Page 130
Synthetic results – Static objects
Haar 3 basis
Error
Haar 3 basis
Ground Truth
Page 131
Synthetic results – Static objects
Haar 3 basis OLB 3 basis
Error
Haar 3 basis
Error
OLB 3 basis
Ground Truth
Page 132
Synthetic results – Static objects
Ground Truth
Page 133
Synthetic results – Static objects
Sph. Harm.
7 basis
Error
Sph. Harm. 7 basis
Ground Truth
Page 134
Synthetic results – Static objects
Sph. Harm.
7 basis
OLB 7 basis
Error
Sph. Harm. 7 basis
Error
OLB 7 basis
Ground Truth
Page 135
Synthetic results – Static objects
Ground Truth
Page 136
Synthetic results – Static objects
Sph. Harm. 3 basis
Error
Sph. Harm. 3 basis
Ground Truth
Page 137
Synthetic results – Static objects
Sph. Harm. 3 basis OLB 3 basis
Error
Sph. Harm. 3 basis
Error
OLB 3 basis
Ground Truth
Page 138
Synthetic results – Static objects
Page 139
Synthetic results – Static objects
Dragon Example:
Page 140
Synthetic results – Static objects
8 OLB = 8 x 1.6 SHLB 13 SHLBDragon Example:
Page 141
Synthetic results – Static objects
8 OLB = 8 x 1.6 SHLB 13 SHLB8 OLB = 8 x 1.9 SHLB 15 FLB
Dragon Example:
Page 142
Synthetic results – Static objects
Dragon Example: 8 OLB = 8 x 1.6 SHLB 13 SHLB8 OLB = 8 x 1.9 SHLB 15 FLB8 OLB = 8 x 3.2 SHLB 25 HaLB
Page 143
Frame #50
Synthetic results – Moving objects
Frame #1
OLB vs. SHLB
Page 144
Frame #50
Synthetic results – Moving objects
Frame #1
9 OLB
OLB vs. SHLB
Page 145
Frame #50
Synthetic results – Moving objects
Frame #1
9 OLB 16 SHLB
OLB vs. SHLB
Page 146
Frame #50
Synthetic results – Moving objects
Frame #1
OLB vs. Fourier & Haar
Page 147
Frame #50
Synthetic results – Moving objects
Frame #1
3 OLB
OLB vs. Fourier & Haar
Page 148
Frame #50
Synthetic results – Moving objects
Frame #1
3 OLB 5 FLB
OLB vs. Fourier & Haar
Page 149
Frame #50
Synthetic results – Moving objects
Frame #1
3 OLB 5 FLB
OLB vs. Fourier & Haar
16 HaLB
Page 150
Real Experiments - Setup
Page 151
Real Experiments - Procedure
OLB computation
1.1. Static scene, single lights1.2. SVD OLB
Video adquisition
2. Moving scene, OLB
Postprocessing 3.1. Alignment3.2. Relighting
Page 152
Real Experiments – Tennis Ball
Page 153
Real Experiments – Tennis Ball
Page 154
Real Experiments – Face
Page 155
Real Experiments – Face
Page 156
Real Experiments – Face
Page 157
Real Experiments – Room corner
Page 158
Real Experiments – Room corner
Page 159
Real Experiments – Room corner
Page 160
Real Experiments – Room corner
Page 161
Conclusions
• Proposed an “object-dependent” lighting basis appropriate for relighting tasks.
• Minimizes the number of reference images needed for relighting.
• Effective method for video relighting.
• Proposed an “object-dependent” lighting basis appropriate for relighting tasks.
• Minimizes the number of reference images needed for relighting.
• Effective method for video relighting.
Page 162
Future work
• Use faster cameras.
• Use “motion-dependent” relighting bases.
• Use faster cameras.
• Use “motion-dependent” relighting bases.
Frame t+1
Frame t+2
Frame t+3
Frame t+4
Frame t+5
Frame t
Slow motion
More accurate results
Page 163
Future work
• Use faster cameras.
• Use “motion-dependent” relighting bases.
• Use faster cameras.
• Use “motion-dependent” relighting bases.
Fast motion
Frame t+1
Frame t+2
Frame t+3
Frame t+4
Frame t+5
Frame t
Slow motion
Reduce alignment & orientation
errors
More accurate results
Page 164
Acknowledgments
• Ko Nishino for his useful comments.
• Anne Fleming for her help in the edition of the videos.
• All the people from the “Laboratory for the Study of Visual Appearance” and “Columbia Automated Vision Environment's” at Columbia University.
• Cyberware for the 3D models.
• Funding Sponsors: • Spanish Ministry of Science & Technology• National Science Foundation
• Ko Nishino for his useful comments.
• Anne Fleming for her help in the edition of the videos.
• All the people from the “Laboratory for the Study of Visual Appearance” and “Columbia Automated Vision Environment's” at Columbia University.
• Cyberware for the 3D models.
• Funding Sponsors: • Spanish Ministry of Science & Technology• National Science Foundation