Illumination Estimation Illumination Estimation via Thin Plate Splinevia Thin Plate Spline

Weihua XiongWeihua Xiong((OmniVision Technology,USAOmniVision Technology,USA))

Lilong Shi, Brian FuntLilong Shi, Brian Funt ((Simon Fraser University, Canada)Simon Fraser University, Canada)

Sung-Su Kim, Byoung-Ho Kang, Sung-Sung-Su Kim, Byoung-Ho Kang, Sung-Duk Lee, Duk Lee,

and Chang-Yeong Kimand Chang-Yeong Kim ((Samsung Advanced Institute of Technology, Samsung Advanced Institute of Technology,

KoreaKorea))

Automatic White Automatic White BalancingBalancing

Image

Estimate the Illumination

Original Surface Color

Illumination Estimation Illumination Estimation MethodsMethods

• Non-statistical SolutionNon-statistical Solution– Gray World (1980 Buchsbaum)Gray World (1980 Buchsbaum)– Max-RGB Max-RGB – Shades of Gray (2004 Finlayson)Shades of Gray (2004 Finlayson)– Gray-Edge Hypothesis (Weijei & Gevers 2005)Gray-Edge Hypothesis (Weijei & Gevers 2005)

• Statistics Based SolutionStatistics Based Solution– Color by Correlation (Finlayson at. el. 2001)Color by Correlation (Finlayson at. el. 2001)– Neural Network (Funt. 2002)Neural Network (Funt. 2002)– Support Vector Regression (Xiong, 2004)Support Vector Regression (Xiong, 2004)– KL-Divergence (Rosenberg 2001)KL-Divergence (Rosenberg 2001)

Limitations of previous Limitations of previous statistical solutionsstatistical solutions

• Color by Correlation Illuminations pre-fixed

• Neural Network Local Minima

• Support Vector Regression Many parameters to be determinedby user

Thin Plate Spline (TPS) Thin Plate Spline (TPS) InterpolationInterpolation

• Thin Plate Spline (TPS) interpolates between Thin Plate Spline (TPS) interpolates between control pointscontrol points– Minimizes the bending energy function of a thin Minimizes the bending energy function of a thin

metal plate.metal plate.– Originally designed for deformable matching Originally designed for deformable matching

between imagesbetween images

• AdvantageAdvantage– Output is always uniqueOutput is always unique– Fits to all the training data smoothly Fits to all the training data smoothly – No extrapolationNo extrapolation– Solution can never be singularSolution can never be singular

• Has been extended into 3D for mapping from Has been extended into 3D for mapping from RGB to XYZ (CIC 2005)RGB to XYZ (CIC 2005)

Source Image Target Image

Thumbnail InputThumbnail Input

• The method uses thumbnails as The method uses thumbnails as inputinput– 8-by-8 input RGB images8-by-8 input RGB images– Each of the 64 pixels is the average of Each of the 64 pixels is the average of

pixels from original input imagepixels from original input imageOriginal Image

Change into chromaticity space [R/(R+G+B) G/(R+G+B)]

So we extend TPS to 128 Dimensions here

Thumbnail Image

TPS for AWB: IntuitionTPS for AWB: Intuition

• Basic ideaBasic idea– Similar images require similar white Similar images require similar white

balancingbalancing– Illumination rg-chromaticity interpolatedIllumination rg-chromaticity interpolated

• Training set contains images and their Training set contains images and their respective white points (illumination respective white points (illumination color)color)

• For input imageFor input image– Interpolate white point based on the distance Interpolate white point based on the distance

between the input and training imagesbetween the input and training images

• Training set consists of •N images•Corresponding illumination chromaticity values

•(for case of 8x8x2=128 input image)•{(Ii,1,Ii,2,…Ii,128), (ri,gi)}. (i/o pair)

• TPS determines parameters •wi and (a0, a1, a2, …, a128) •Controls two non-rigid mapping functions fr, fg, • such that (ri,gi) = (fr(Ii,1,Ii,2,…Ii,128), fg(Ii,1,Ii,2,…Ii,128)).

• TPS is defined by a non-linear function with an additional linear term. • Without loss of generality, consider only fr definition in which wi and ai are coefficients to be determined:

TPS Details

Weighted distance to every image in the training set

X is either r or g chromaticity

TPS Details (Continued)

rrrUwhere

Iaa

IIIIIIUwIIIf

jjj

N

iiiiiX

log)(

'

||)),...,()',...','((||)',...','(

2

128

10

1128,,1,1282112821

Linear Term

Nonlinear Term

TPS Details (Continued)

Smoothness constraint

12821128... 12821

128

12821

......!!...!

!128)(

12821

12821dIdIdI

III

ffJ Xr

where is the total bending energy described in terms of the curvature of the energy is minimized when

0

....0

0

0

128,

2,

1,

ii

ii

ii

i

wI

wI

wI

w

Direct Solution (once per training set)

N + 129 unknowns and N + 129 equations

Define L:

OQ

QUL

T

0......................

..

.............0

.........0

1,

,21,2

,12,1

N

N

N

U

UU

UU

U

Uij = U(||(Ii1,Ii2,…Ii128 )- (Ij1,Ij2,…Ij128)||,

U(r) = r2*logr

128,128,128,

128,22,21,2

128,12,11,1

...1

...

...1

...1

NNN III

III

III

Q

Additionally define W= (w1, w2,…, wN, a0, a1, a2, …, a128)T, andK = (r1, r2, r3… rN,0,0,0,….0)T.

We have K =LW and solution W = L-1K

0 is ZERO matrix with size of 129x129

Original Training Data Pairs

X

X

Graphical Example of TPS Interpolation

Nonlinear TermX

Linear Term

X

Error

TPS ResultTPS Result

Input Image(Bluish)

TPS OutputGround Truth Image

TPS on Large Image TPS on Large Image DatabaseDatabase

Average of bright part on the grayball is assumed to the true illumination value

Performance ComparisonPerformance Comparison

MethodMethodAngularAngular Distance Distance (x10(x1022))

MaxMax RMSRMS MaxMax RMSRMSSVR (3D)SVR (3D) 24.5524.55 6.766.76 18.6218.62 5.035.03

TPSTPS 34.8134.81 7.027.02 25.7825.78 5.195.19

SOGSOG 37.0137.01 8.938.93 27.9927.99 6.596.59

Gray WorldGray World 43.8443.84 9.669.66 45.0945.09 7.827.82

Max RGBMax RGB 27.4227.42 12.8112.81 21.7221.72 9.149.14

• The database is divided into two non-overlapped subsets• Test set size 4080. Training data set size 3581

Algorithm Comparison Using Algorithm Comparison Using WilcoxonWilcoxon

MethodMethod TPSTPS SVR(3D)SVR(3D) SoGSoG MAXMAX GWGW

TPSTPS == ++ ++ ++SVR(3D) SVR(3D) == ++ ++ ++SoGSoG -- -- -- --MAXMAX -- -- -- --GWGW -- -- -- ++

TPS is equal to SVR, but better than GW, SoG, and Max

Conclusion Conclusion

• Thin Plate Interpolation is applied in Thin Plate Interpolation is applied in illumination estimationillumination estimation

• Performs non-uniform interpolation Performs non-uniform interpolation – Assumption that similar images require similar Assumption that similar images require similar

color correctioncolor correction

• Method is parameter independentMethod is parameter independent• Both training and testing are fastBoth training and testing are fast• The performance on natural images The performance on natural images shows shows

the accuracy of TPS illumination the accuracy of TPS illumination estimation to be good estimation to be good

Funded by Samsung Funded by Samsung Advanced Technology Advanced Technology

Institute Institute

Thanks & Thanks & QuestionsQuestions