Departamento de Departamento de ÓÓptica, Facultad de Ciencias, ptica, Facultad de Ciencias, Universidad de Granada, 18071Universidad de Granada, 18071--Granada (SPAIN)Granada (SPAIN)
Photometric -stereo estimation from spectral systemsJuan Luis Nieves et al.
Colour Imaging Lab (www.ugr.es/local/colorimg)
3rd Thematic Network MeetingFebruary 5, 2009, Toledo (Spain)
Colour Imaging Lab
Universidad de Granadawww.ugr.es/~colorimg
Javier Romero
Juan Luis Nieves
Eva M. Valero
Clara PlataJavier Hernández
Andrés
Raul Luzón
At present our research revolves around the development of techniques for acquiring and processing multispectral images and applying them to…
� Photometric-stereo techniques,� Design of optimum sensors in the identification and measurement of the
spectral composition of daylight,� Color constancy in the development of color-descriptors invariant to
changes in illumination applicable to color imaging,� … etc.
The problem
Reflected light from objects depends on reflectance(albedo) and illumination…
Scene400 500 600 7000
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
0.1
Wavelength (nm)
Spectral image acquisition
� Better resolution� Easy and cheaper
The problem
� Different approaches
Liquid Crystal Tunable Filtre (LCTF)
A CCD camera with a narrow -band filtre set
coupled
The problemSpectral image acquisition
Filtre wheel
n colour filtres
� Different approaches
A CCD camera throughbroad-band colour filtres
some “a priori” information+
The problemSpectral image acquisition
Spectral approach vs. colorimetric approach• Not limited to the human visual range: advantages to include
the infrared (IR) and UV (ultraviolet);• Metamerism is avoided;• Illuminant changes can be simulated; • Applications from remote sensing, astronomy, medicine, art
restoration, cosmetics, printing, computer graphics, biology, agriculture, etc.
The problem
The problem
… and on surface relief.
Reflected light from objects depends on reflectance (albedo) and illumination…
��PhotometricPhotometric--stereo as a way of obtaining stereo as a way of obtaining 3D and albedo from surfaces.3D and albedo from surfaces.
��An RGB digital camera to simultaneously An RGB digital camera to simultaneously recover spectral reflectance and surface recover spectral reflectance and surface relief from images.relief from images.
Outline
Directmethods
�A technique in computer vision for estimating the surface normals of objects by observing that object under different lighting conditions.
Photometric Stereo
�A technique in computer vision for estimating the surface normals of objects by observing that object under different lighting conditions.
Photometric Stereo
Indirect methods
ρNL1 L2
L3
Photometric Stereo
z y
x
e
g
i
N
R LN
( )yxSz ,=
( )x
Syxp
∂∂=, ( )
y
Syxq
∂∂=,
( )Tqp
qp
1,,1
1
22
−++
=N
Indirect methods
ρN
L1
L2
L3
Photometric Stereo
( )TIII321
,,=I
[ ] ( )TL321
,, LLL=
[ ]NI Lρ=
( )NL ·kk
I ρ=
[ ] NI-1 ρ=L
4-source photometric stereo
� It’s faster than laser scanning;
� It can be made surface rotation invariant.
� The colour maps of 3D surface textures may be separated and analysedindependently.
Why albedo is important in computer graphics?
4-source photometric stereoFigura con el ilumiante descontado
utilizando la media de las normales recuperadas
Figura con el iluminante descontadoutilizando la normal recuperada para cada canal
� It’s faster than laser scanning;
� It can be made surface rotation invariant.
� The colour maps of 3D surface textures may be separated and analysedindependently.
Why albedo is important in computer graphics?
Spectral analysis and synthesis
sensors’ responses EC=ρ illuminant spectrum
sensors’ spectral sensitivities
kx1 Nx1kxN
Pseudoinverse: ( ) TT ρρρρ 1−+ =Training set:
+= ρSD
S ρ
Retiga 1300
+
Test set:1
ρ11
ρDS =⇒
400 700
0
0.1
0.2
0.3
0.4
0.5
400 700
0
0.1
0.2
0.3
0.4
0.5
400 700
0
0.1
0.2
0.3
0.4
0.5
400 700
0
0.1
0.2
0.3
0.4
0.5
400 700
0
0.1
0.2
0.3
0.4
0.5
400 700
0
0.1
0.2
0.3
0.4
0.5
400 700
0
0.1
0.2
0.3
0.4
0.5
400 700
0
0.1
0.2
0.3
0.4
0.5
400 700
0
0.1
0.2
0.3
0.4
0.5
400 700
0
0.1
0.2
0.3
0.4
0.5
400 700
0
0.1
0.2
0.3
0.4
0.5
400 700
0
0.1
0.2
0.3
0.4
0.5
400 700
0
0.1
0.2
0.3
0.4
0.5
400 700
0
0.1
0.2
0.3
0.4
0.5
400 700
0
0.1
0.2
0.3
0.4
0.5
400 700
0
0.1
0.2
0.3
0.4
0.5
400 700
0
0.1
0.2
0.3
0.4
0.5
400 700
0
0.1
0.2
0.3
0.4
0.5
400 700
0
0.1
0.2
0.3
0.4
0.5
400 700
0
0.1
0.2
0.3
0.4
0.5
400 700
0
0.1
0.2
0.3
0.4
0.5
400 700
0
0.1
0.2
0.3
0.4
0.5
400 700
0
0.1
0.2
0.3
0.4
0.5
400 700
0
0.1
0.2
0.3
0.4
0.5
Spectral analysis and synthesis
Retiga 1300
+
400 450 500 550 600 650 7000
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Wavelength (nm)
Re
lativ
e S
PD
Illuminant: TRILITEGFC= 0.9268Colour Difference= 6.0RMSE= 0.049IIE= 0.39
400 450 500 550 600 650 7000
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Wavelength (nm)
Re
lativ
e S
PD
Illuminant: TRILITEGFC= 0.8361Colour Difference= 9.8RMSE= 0.073IIE= 0.501
Results for a 6-band camera
Outline
Spectral imageacquisition
Photometric - stereo+
400 450 500 550 600 650 700
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Wavelenght (nm) N
Original, 0º
Original, 90º
Original, 180º
Original, 270º Albedo
Spectral analysis and synthesis
( )NL·ρ=IρNL1 L2
L3
400 450 500 550 600 650 7000
0.2
0.4
0.6
0.8
1
Wavelength (nm)
Nor
mal
ized
SP
D
Digilite 600IncandescentTri-Lite
( ) ( ) ( )∑= λλλρkk
QSE
Mul
tidim
ensi
onal
pro
blem
Mul
tidim
ensi
onal
pro
blem
� Root-Mean-Square-Error (RMSE)
Spectral and colorimetric evaluation metrics
�Goodness-of-Fit-Coefficient (GFC): colorimetric accurate fit >0.995good spectral fit >0.999almost exact fit >0.9999
[ ] [ ]21
2
21
2
)()(
)()(
GFC
∑∑
∑=
j
jR
j
j
j
jRj
EE
EE
λλ
λλ
� CIELab colour difference: colorimetricallyacceptable ~3 CIELab( ) ( ) ( )[ ] 2/12*2*2**
baLEab
∆+∆+∆=∆
� RGB-error: ( )222
3
1xx
BGRRGBerrorxx
∆+∆+∆=
( )1cos
j o eAE
−= ⋅ρ ρ� Angular error:
Results : color reproduction
SAME illuminant for captured and recovered images;DIFFERENT geometry;
1.520.750.75AE
6.02.63.1CIELAB
1718082RGBerror
Averaged results
1.530.750.76AE
6.12.83.2CIELAB
1778987RGBerror
Test set of samples
1.510.740.74AE
5.82.32.9CIELAB
1657078RGBerror
Training set of samples
90 prctile
Std DevMean
�The results show good color recovery (around2% of total error);
20 40 60 80 100
20
40
60
80
100
50
100
150
200
250
300
0 50 100 150 200 250 3000
500
1000
1500
RGBE differences
Mean: 73Std Dev: 3190 perc: 113
Results : color reproduction
Original Simulated
0 5 10 15 20 25 30 35 40 45 50 55 600
200
400
600
800
1000
RMS Diferences
Gris 1390
1 2 3 4 5 6 7 8 9 10 11 12 13 14 150
1000
2000
3000
4000
5000
6000Gris 1390
SCIELab differences
SAME illuminant for captured and recovered images;DIFFERENT geometry;
6.206.204.004.001.021.022.072.070.710.7112.5812.58SCIELabSCIELab333321217788006060RMSRMS
PercPerc 9999PercPerc 9595StdStdDevDevMeanMeanMinMinMaxMaxGris Gris
13901390
270º 270º
Original Simulated0 10 20 30 40 50
0
500
1000
1500
2000
2500
3000
SCIELab differences
Rosa Valencia 1518
24.9824.9813.6913.694.764.764.904.900.250.2554.9354.93SCIELabSCIELab1601601051053333393900264264RMSRMS
PercPerc 9999PercPerc 9595StdStdDevDevMeanMeanMinMinMaxMaxRosa Rosa
ValenciaValencia
0 50 100 150 200 2500
200
400
600
800
1000
1200
RMS differences
Rosa Valencia 1518
Results : color reproduction
0º 0º
SAME illuminant for captured and recovered images;DIFFERENT geometry;
Original Simulated
29.4829.4814.0614.065.275.274.934.930.080.0875.0375.03SCIELabSCIELab11611684842323404022220220RMSRMS
PercPerc 9999PercPerc 9595StdStdDevDevMeanMeanMinMinMaxMaxAlabastro Alabastro
13841384
0 20 40 600
500
1000
1500
2000
2500
SCIELab differences
Alabastro 1384, 90º, Digilite
0 50 100 150 2000
200
400
600
800
1000
1200
RMS differences
Alabastro 1384, 90º, Digilite
Results : color image synthesis
SAME illuminant for captured and recovered images;DIFFERENT geometry;
0º 0º
Original Simulated
16.6016.6011.4811.482.922.925.865.861.801.8057.9557.95SCIELabSCIELab13091309942942246246466466313122292229RMSRMS
PercPerc 9999PercPerc 9595StdStdDevDevMeanMeanMinMinMaxMaxAlabastro Alabastro
13841384
0 500 1000 1500 20000
200
400
600
800
1000
1200
RMS differences
Alabastro 1384, 90º, Incandescent
0 10 20 30 40 50 600
500
1000
1500
2000
2500
3000
SCIELab differences
Alabastro 1384, 0º, Incand
Results : color image synthesis
0º 0º
DIFFERENT illuminant for captured and recovered images;DIFFERENT geometry;
Original Simulated
50.4750.4718.0818.088.768.768.758.751.931.93119.29119.29SCIELabSCIELab33233226226257571561561919556556RMSRMS
PercPerc 9999PercPerc 9595StdStd DevDevMeanMeanMinMinMaxMaxHueso Hueso 13821382
0 20 40 60 80 100 1200
500
1000
1500
2000
SCIELab differences
Hueso 1382, 90º, Tri-Lite
0 100 200 300 4000
100
200
300
400
500
600
RMS differences
Albero 1391, 180º, Digilite
Results : color image synthesis
DIFFERENT illuminant for captured and recovered images;DIFFERENT geometry;
90º 90º
• A digital RGB camera and a 4-source photometric-stereo can be used to model surface properties under Lambertian assumption.
• A spectral-based photometric-stereo algorithm have been developed to synthesize real-world textured objects under different illumination geometries.
• It is possible to use albedo in spectral analysis to recover spectral reflectances from linear pseudo-inverse.
Conclusions
• What are the deviations from Lambertian behaviors in real-world textured surfaces?
• …and what do other computer graphics and vision models predict?
Future
Thanks!Departamento de Departamento de ÓÓptica, Facultad de Ciencias, ptica, Facultad de Ciencias, Universidad de Granada, 18071Universidad de Granada, 18071--Granada (SPAIN)Granada (SPAIN)
MASTER Erasmus Mundus
"Color in Informatics and MEdia
Technology“ (CIMET)
University of Granada, SpainUniversity of Joensuu, Finland
University College of Gjovik, NorwayUniversity of Saint-Etienne, France
www.master-erasmusmundus-color.eu
120 ECTS (two years, in English)
The study programme of this Master course is broadly interdisciplinary, encompassing photonics, computer vision and imaging science, computer science and multimedia technology as a mix of relevant theoretical and practical knowledge.
Official fees:Non-EU students: 10.060 € per year (with 18 grants of 21.000 € to the best non-EU students)
EU students: 3.350€ per year (with 8 grants to reduce the fees to 350€ per year to the best EU students). Deadline for application: 22nd of May 2009 .
www.master-erasmusmundus-color.eu