! !#"$ &%'"(#'#" )*+ & !,- !(#%

./10321054.60708489.;:</>=?/A@CB+D2E4F0

G60H@I432KJMLN<NPORQS@T48/>=VUW<WU

XZY[]\_^+`badce]c fhgiaAjc;kmlnao[pcaq[ rcstjcgu+vwxy>z|v~z]twn~zvyKv<v ! 5 588' ¢¡88¤£;¥PH¦§¨ Hi©>p ª3¬«5­®

¯ & !#"6 !%'"(#'#" )°*+ ! !, !(#%

.;/±0²21084M.³0705489.:</=?/A@B'D2E4F0

G60H@I432KJMLN<NPORQ_S#@T47/=´UW<WU

µ¶!·R¸7¹bº>»i¶H¼ºV½¾¿À½8»#·RÁ¤º>¶!¹Â²ÃHÄA¶!¼¤Ã!¶7ÅÇÆ;ÈÊÉ-ÈÊË<¶!Á¤Ì7¶!¼

ÍVÎRÏEÐbÑÒmÓ1ÐÔoÕÖ¨×mØÚÙ~Û¨ÜbÝÞHÛßÖàÜÝÛáÜbÙ¨ÜEÕ!Ö~ânÕÜEà'ÛáÜãqäÊØÚåæ×!ÖáØÊÕåÀÖáÜbçè×Õجé!êÜÖá×mâ±Ö>ëìíÞæÛPâ

Ù¨ØÊÕmåæäÚÜî&ØÚÜEà<Ý7ÞæØÚÕ!ÖbëHâHçEçEêÛáâ1Ö¨ÜbäÊï]çbâ1ÝðÖáêÛáÜbÙ¤Ö¨×mÜ<Û¨ÜEñmÜbçÖáâ1ÕçÜòmÜEä¬óÀÞ1ì8Û¨Ü>â1äâæÕmóî&ØÚÛßÖáêmâ1äÞæôðõdÜ>ç¢ÖáÙbë!à<ØöÖá×ÞæêðÖÛ¨Ü>ÙdÖáۨجç¢ÖáØÊÞHÕmÙÇÞHÕnÖ¨×ÜbØÊÛåæÜbÞæ÷nÜÖáۨجçEâæäçEÞæ÷nÝäÚÜøðØöÖdïÞHÛ÷nâ1Ö¨Übۨجâ1äùÝÛáÞæÝ7ÜEÛ¨Ö¨ØÚÜbÙbúVû~Ù]âÛ¨Ü>ÙßêäÊÖÖ¨×mÜ´ÞæôðõdÜ>ç¢ÖáÙ]çEâ1Õô7ÜVÛ¨ÜEãAäÚØÊÖêmÕmóðÜEÛâæÛ¨ôØÊÖ¨Ûèâ1ÛáïäÊØÚåæ×!Ö¨ØÚÕåçÞHÕmóðØÊÖ¨ØÚÞæÕmÙbú#ü²Þçbâ1ÝðÖáêÛ¨ÜÖ¨×mÜtÛáÜñÜbç¢Öèâ1ÕmçEÜVòmÜbäÚóëàÜÖèâ1ýæÜÀÝ×ÞæÖ¨ÞHåæÛèâ1Ý×mÙÞæìâ1ÕÞæôõdÜbç¢ÖäÚØöÖô&ï_Ù¨ÜEîHÜEÛèâ1äRäÊØÚåæ×!ÖáØÊÕåÝmâ1ÖßÖáÜEÛáÕmÙÞæÕâÙ¨êÛáÛ¨ÞHêÕmóðØÚÕå-óØÚÙáçÛáÜÖáØÊþbÜbó×mÜE÷nØÚçEêô7Üæú6üp×ÜØÊäÚäÊêm÷ÀØÚÕmâ1Ö¨ØÚÞæÕ#ÞæìPÜ>âæçè×#ÝmØöøðÜEäóðêmÜ]ÖáÞÜb÷ÀØÊÖßÖáÜbó-ÛáâHóðجâ1ÕmçEÜìíÛáÞæ÷ÿÜbâHçè×óðج٨çEÛ¨ÜEÖ¨ÜÀÝmâ±Öèçè×ÞHÕ_Ö¨×mÜT×Üb÷nØÚçEêôÜجٴâ1ÝÝmÛ¨ÞøðØÚ÷nâ1Ö¨Ü>ó;ô&ï#âÛ¨ÜEñmÜbçÖ¨ØÚÞæÕiçEÞ&ÜVçØÚÜEÕ!ÖVâ1Õmó³âÛ¨Ü>ç¢Öèâ1ÕåHêäÚâæÛÀÙ¨êÝðãÝ7ÞæÛ¨ÖPà<×جçè×_â1ÛáÜìíÞHêÕmó¯Öá×ÛáÞæêåH×_â1Õ_ÞHÝðÖ¨ØÚ÷nØÊþ>â±Ö¨ØÚÞæÕ¯ÝmÛ¨ÞðçÜ>óðêÛáÜæú~üp×mÜTóðجÙdãçEÛ¨ÜEÖ¨ØÚþbâ±ÖáØÊÞHÕÞ1ì<Öá×Üt×ÜE÷nجçêô7ÜÜbÕmÙ¨êÛ¨Ü>ÙTÙ¨êVçØÚÜEÕ!Ö´â1ÕåHêä¬â1ÛTÙáâ1÷nÝäÚØÚÕå_ÖáÞçbâ1ÝðÖáêÛ¨ÜóðØ8êmÙßÜ]÷nâ1Ö¨Übۨجâ1ä5ÝÛáÞæÝ7ÜEÛ¨Ö¨ØÚÜbÙbúÇüp×ÜêÙßÜÞæìRâÀÝ7ÜEÛÝmâ1Öáçèׯ٨êÝÝ7ÞæÛ¨ÖâHçEçÞH÷n÷ÀÞðóâ1Ö¨Ü>Ù~Öá×ÜVÙ¨÷nâæäÊä ÙßÞHäÊجó-âæÕåæäÚÜnÞ1ì<âtØÊÕmçEÞæ÷nØÚÕå¯äÊØÚåæ×!ÖØÚ÷ÀÝ7ÞæÛ¨ÖáâæÕ!ÖìíÞHÛ]ÙßÝ7ÜbçEêä¬â1Û÷´â±ÖáÜEÛáØÚâæäÚÙbúü²Þ¯Û¨ÜEãAäÚØÊåH×!ÖÖá×ÜVÞHôðõdÜbçÖbë²Ö¨×Ü´Öèâ1ÛáåæÜEÖØÚäÚäÊê÷nØÚÕmâ±ãÖáØÊÞHÕëâ×mØÊåH×ðãqóï!Õâ1÷nØÚçTÛèâ1ÕåHÜ]ÜEÕ&î&ØÚÛ¨ÞHÕ÷nÜEÕ!ÖP÷´â1Ý3ë5ØÚÙâîæÜbÛáâæåæÜ>óÞ±îæÜEÛPÖ¨×mÜÙ¨êÝÝ7ÞæÛ¨Ö¤âæÕmó÷TêäöÖáØÊÝmäÊØÚÜbó]ô!ïÖá×Ü ÛáÜñÜbç¢ÖáØÊÞHÕ]çÞ&Ü´çEØÊÜbÕ!Öbë±Ý7ÜEÛRÝØÊø&Übäðâ1ÕmóÝ7ÜEÛóðجÙáçÛáÜÖ¨Ü<Ýâ±Öáçè×nÞHÕnÖ¨×Ü×mÜE÷nØÚçEêô7Üæúùüp×ÜÛáÜbÙ¨êäÊÖáÙùÞHôðÖáâæØÊÕÜ>ónÙß×Þ±à6âæçbçêÛèâ±ÖáÜÛáÜãqäÊØÚåæ×!ÖáØÊÕåÀÞ1ìùóðØ7êÙßÜâHÙ àÜEäÚä3âHÙ<ÙßÝ7ÜbçEêäÚâæÛ<ÞæôðõdÜ>ç¢ÖèÙEú

Ñ¤Ï Ôo÷´â1åHÜãqômâæÙ¨Übó<ÜãqäÚØÊåH×HÖáØÊÕmåmëØÚåæ×!Ö!Öèâ1åæÜHëÇÕ&î&ØÊÛáÞæÕm÷ÀÜbÕ!Öâ±ÖßÖáØÊÕmåmú "!3Î# Ó1Ð$%ÒÏ>Ï'&)(¤ÓHÒðÐ'& +* Ô¢ú ,ú.-&ë&Ô¢ú /ú0Í$12435126879%ÒmÏbÏ:&;(RÓæÒðÐ:& <* >= Û¨ØÚ÷´â1Ûáï?ðÔ¢ú ,mú@-ðëA&ÜbçEÞæÕmóâæÛ¨ï?mÔ¢ú /ú0Hú

AccurateImageBasedRe-lightingthroughOptimization

PieterPeers Philip DutréDepartmentof ComputerScience,K.U.Leuven,Belgium

April 11,2002

Abstract

In this reportwe presenta new re-lighting techniquethat, for a singleview-point, accuratelycapturesthereflectancefield of realandvirtual objects,withoutrestrictionson their geometricalcomplexity or materialproperties. As a resultthe objectscan be re-lit underarbitrary lighting conditions. To capturethe re-flectancefield, we take photographsof anobjectlit by several lighting patternsonasurroundingdiscretizedhemicube.Theilluminationof eachpixel dueto emittedradiancefrom eachdiscretepatchon the hemicubeis approximatedby a reflec-tion coefficientandarectangularsupportwhicharefoundthroughanoptimizationprocedure.Thediscretizationof thehemicubeensuressufficient angularsamplingto capturediffusematerialproperties.The useof a per patchsupportaccommo-datesthe small solid angleof a incoming light importantfor specularmaterials.To re-light theobject,the target illumination, a high-dynamicrangeenvironmentmap,is averagedover thesupportandmultiplied by thereflectioncoefficient, perpixel andperdiscretepatchon thehemicube.Theresultsobtainedshow accuratere-lightingof diffuseaswell asspecularobjects.

1

1 Intr oduction

Image-basedre-lighting representsa classof techniquesthatapplynew lighting con-ditions to a scene,givena setof baseimages.Possibleapplicationsrangefrom light-ing designto augmentedreality. Lighting designcan be a tedioustask, sincemov-ing objectsandlight sourcesis not alwaysfeasible.Objectscanbe very fragile (e.g.archaeologicalartifacts),the illumination is not controllable(e.g. the sunthroughasky-window) or an objectis too massive to move around(e.g. a statue).Augmentedreality applicationsareconstantlyevolving andpushingthe limits of widely acceptedtechniques.Placingreal objectsin virtual environmentsinfluencesthe appearanceofthe objects. It is not alwayspossibleto approximatethe desiredillumination at themomentanobjectis captured(e.g.specialeffectsin amovie or computergameswhichcombinerealandvirtual environments).

In thisreportwedevelopare-lightingtechniquethatcombinesstrengthsfrom tech-niquesastheLight Stage[2, 5] andenvironmentmatting[1, 14] intoasingleframework.This allows us to capturecomplex geometricalobjectswithout placing restrictionson materialpropertiesor losing practicalusability. To capturethe reflectancefield ahemicubearoundtheobjectis decretizedin several light patches.Eachpatchemitsaseriesof illuminationpatternsandfor eachpatternahigh-dynamicrangephotograph[3]is recorded.Currentlywe usea calibratedCRT monitorperhemicubesideto emit theillumination patterns.For eachpixel andfor eachlight patchthe emittedradianceisthenapproximatedby a reflectioncoefficient anda supportareaon the light patch.Aleastsquareminimizationprocedureon thepixel radianceover differentlighting pat-ternsis usedto determinethebestapproximationfor thesupport.To re-light thescenewith anarbitraryhigh-dynamicrangelight-map,theilluminationon thesupportis cal-culatedandmultiplied by the reflectancefactor. This is repeatedfor eachpixel andeachlight patch.

Our reportis structuredasfollows: In thenext sectionwe discusspreviouswork.In section3 a mathematicalframework is developed.Thedevelopedtechniqueis pre-sentedasathreestepalgorithmin sections5,6 and7. Therelationwith theLight Stageandenvironmentmattingis furtherexploredin section8. We concludethis reportwithresultsin section9 andfuturework in section10.

2 Previous Work

In this sectionwe give an overview of work relatedto imagebasedre-lighting. Anumberof techniquesrequiresor reconstructageometricalmodelof thescene.Loscoset. al.[8, 9] reconstructsasimplifiedgeometricalmodel,approximatestheilluminationandcalculatesun-occludedillumination textureswith anadaptedradiosityalgorithm.Re-lightingis doneby modifying real andvirtual light intensities.Yu et. al.[12, 13]focusmainlyon recoveringthereflectancepropertiesof materialsby usinganiterativeoptimizationprocedureto find parametersto representthe reflectancepropertiesofsurfaces. The obtainedresultsarere-renderedusinga global illumination renderingsystem.Reconstructionof geometryis advantageouswhenvisualizationfrom multipleviewpoints is desired,but is a disadvantagewhena sceneconsistout of geometrical

2

complex objects.Anotherclassof techniquescircumventstheproblemof complex geometricalmod-

elsby directlymanipulatingimages.Nimeroff et. al.[10] introducedtheconceptof lin-earlyweightingandcombiningbaseimagesof ascene.Thesebaseimagesarerenderedwith a numberof lighting conditionsfrom which thedesiredfinal lighting canbecon-structed.Wonget. al.[11] basedtheir re-lightingmethodon light field rendering[4, 6].By samplingthesceneunderdifferentviewpointsandillumination anapparentBRDFfor eachsurfacepixel is found.Lin. et. al.[7] defineareflectedirradiancefield, thedualof a light field, andderive uppersamplingboundsperBRDF to reproducethemtruth-fully. TheLight Stage[2, 5] usesbaseimagesof a scenelit by lights regularly spacedon a spheresurroundingthescene.A light-mapis appliedby linearly combiningandweightingthesebaseimagesinto a singleresultingimage.Environmentmatting,orig-inally introducedby Zongkeret. al.[14] andextendedby Chuanget. al.[1], capturethereflectancepropertiesof specularandrefractive materialsby illuminating themwithdifferentlighting patterns.Thedirectionandsolidanglecanbederivedfrom this infor-mationin theform of a reflectioncoefficientanda supporton thebackor side-drops.

3 Mathematical Framework

In this sectionwe develop an unifying mathematicalframework for re-lighting. Wederive the re-lighting equationin subsection3.1. In subsections3.2, 3.3 and3.4 weexpressour andotherre-lightingtechniqueswith this re-lightingequation.

We derivea formula,basedon reflectionfields,which wecall there-lightingequa-tion throughoutthis report. With this mathematicalframework, we have a correctmathematicalbasisin which we cansolidly develop our techniqueandcancompareourmethodon amathematicalbasiswith othertechniques.

3.1 Re-lighting equation

Weuseanotationsimularto Debevecet. al.[2]. Thereflectancefield ataclosedsurfaceA is definedasa 8 dimensionalfunction:

R B R C Ri D Rr E B R C ui D vi D θi D φi;ur D vr D θr D φr E (1)

whereRi representsthe incidentlight field at A andRr representsthe radiant(ex-itant) light-field at A. C u D v E is a parameterizationof A and C θ D φ E is usedto denotedirections.

Wemakethefollowingassumption:Theincidentlight field Li C θ D φ E originatesfroma point at infinity. This allows us to expressthe incident light field independentofC ui D vi E on A.

Ri C ui D vi D θi D φi E B Ri C θi D φi EThis assumptionreduces(1) to a6 dimensionalfunction:

3

R B R FGC θi D φi;ur D vr D θr D φr E (2)

Thereflectancefield for afixedcamerapositioncanbeexpressedasthereflectancefunction Rp througha pixel p of a photograph.The total radianceL seenthroughapixel p:

L C p E BIHΩ4π

Rp C θi D φi E Li C θi D φi E dω (3)

whereΩ4π representsthe entiresphere.We approximateRp C θi D φi E by a discretesumof reflectioncoefficientsRp J j anda setof normalizedbasisfunctionsS j C θi D φi E ,eachdefinedoverasetof discretizedsolid angles.

Rp C θ D φ E"K ∑j

Rp J jS j C θ D φ E (4)

and L j : M Ω jS j C θ D φ E dω B 1. Combining(3) and(4) we arrive at the re-lighting

equation:

L C p E"K ∑j

Rp J j HΩ j

S j C θi D φi E Li C θi D φi E dω B Ip (5)

where N j Ω j B Ω4π.Thepurposeof this reportis to computefor eachpixel thereflectancecoefficients

Rp J j.3.2 Choiceof the reflectioncoefficientRp O j and support S j P θ Q φ RThe choiceof the reflectioncoefficient Rp J j andsupportS j C θ D φ E dependson the im-plementationof a particulartechnique. The choiceof S j C θ D φ E is of great influenceon which kind of re-lighting is possible.Whena narrow supportfunction is chosen,emphasisis put on spacialsampling. Choosinga variablebut detailedsupport,localsamplingcanbedonein greatdetail.

For Ω4π we usea hemicubeand divided eachside in a numberof patchesΩ j.We usefor S j C θ D φ E a box-filter on a axis-alignedsupportwithin a patch. A morecomplex filter is possible,but theresultsobtainedwith a box-filter weregood.If Ω F j isa rectangularsupportin Ω j then:

S j C θ D φ E B 1Ω F j i f C θ D φ E>S Ω F j

B 0 otherwise

4

andRp J j canbe calculatedby definingLw C i E a constantillumination function forpatchΩi:

Ipi B ∑j

Rp J j HΩ j

S j C θ D φ E Lw C i E dω

B ∑j

Rp J jLw C i EB Rp J iLw C i E

thus:

Rp J i B Ipi

Lw C i E (6)

3.3 The Light Stage

We show that the Light Stagecanbe expressedwith the re-lightingequation(5).De-bevecet. al. usedthefollowing definition:

L C x D y E B ∑θ J φ Rx J y C θ D φ E Li C θ D φ E δA C θ D φ E

whereL C x D y E is a recordedimage,Rx J y is the reflectancefunction througha pixelC x D y E with incomingdirection C θ D φ E . Li C θ D φ E the incomingradianceandδA C θ D φ E BsinC φ E .

Set j andS j C θ D φ E asfollows:

j B TUC θ j D φ j E4V f ixed positions on the sphere WS j C θ D φ E B δ C θ B θ j X φ B φ j E

DefineΩ j accordinglyto j andthenthere-lightingequation(5)resultsin:

Ip B ∑j

Rp J j HYHΩ j

S j C θi D φi E Li C θi D φi E sinC φi E dθidφi

B ∑θ j J φ j

Rp J θ j J φ j Li C θ j D φ j E sinC φ j EBy usingaDiracfunctionasasupport,emphasisis putondiscretespatialsampling.

Reflectiondetailsof materialsarelimited by thediscretizationof Ω4π in C θ D φ E .3.4 Envir onment matting

We show thatenvironmentmattingcanbeexpressedwith the re-lightingequation(5).Zongkeret. al. usedthefollowing definition:

5

C B F Z[C 1 \ α E B Z m

∑i ] 1Ri ^ C Ti D Ai E

whereC is therecordedpixel color, F theforegroundcolor that theobjectreflectswithout applying controlledillumination. α the matteand determinesthe coverageof a pixel by the object. B the backdropcolor seenthroughthe pixel. The number(m) of illumination texturesTi which are usedto display controlledillumination ontheobjectis 3 in theoriginal paper(backandside-drops).Ri thereflectioncoefficientand ^ C Ti D Ai E is a box filtered supportof an areaAi in Ti which determinethe maindirectionandsolid angleof visually importantlight for apixel.

We ignoretheforegroundcolorF , sinceweassumethattheilluminationonly orig-inatesfrom infinity andis completelycontrolled.SetΩi B Ti andΩm _ 1 B B. Usingthere-lightingequation(5)we obtain:

Ip B m _ 1∑

jRp J j H

Ω j

S C θi D φi E Li C θi D φi E dω

B Rp Jm _ 1 HB

S C θi D φi E Li C θi D φi E dω

Z m

∑j

Rp J j HTi

S C θi D φi E Li C θi D φi E dω

By choosingS C θ D φ E asfollows:

HTi

S C θ D φ E Li C θ D φ E dω B ^ C Ti D Ai EandsettingAm _ 1 equalto B andα B 1 \ Rp Jm _ 1, we obtain:

Ip B`C 1 \ α E B Z m

∑j

Rp J j ^ C Ti D Ai Ewhich wastheequationwe werelooking for. Sincem is low (e.g. 3), emphasisis

put on detailedlocal illumination supports,which areproficientfor capturingspecularandtransparentmaterials.

4 Thr eestepalgorithm

Thegoal of our re-lighting techniqueis to captureobjectswithout placingrestrictiononmaterialpropertiesandto faithfully reproducetheseobjectsunderarbitrarylightingconditions.In the following threesectionswe discussthe threestagesof our methodbuild on themathematicalframework developedin theprevioussection.

6

Firstly, enoughinput datahasto be acquiredto extract the reflectancepropertiesfor eachpixel for a fixed viewpoint (section5). Oncethe input datais gathered,thereflectancepropertiesarecalculatedandstoredfor fastre-lightingafterwards(section6). This processingstepis only doneoncefor eachscene.Whenall datais processed,an arbitrarylight-mapcanbe appliedto the illumination of a scene(section7). Thisre-lighting can be donean arbitrary numberof times with different light-mapsandrequireslittle processingtime.

5 Data Acquisition

To capturethe reflectancefield, the object is lit by several light patcheslocatedon asurroundinghemicube.Eachlight patchemitsaseriesof illuminationpatterns,andforeachpatterna highdynamicrangephotograph[3] is recorded.

It is importantthat therecordedhigh-dynamicrangephotographsarenot underorover-saturated,becausethis might resultin lossof accuracy. High intensityaswell aslow intensityradiancefrom theilluminationpatternsareimportantfor findingacorrectsupportfor specularmaterials.Every emittedradiancevalueresultsin extra informa-tion regardingthe reflectancepropertiesof a pixel. Sincea low numberof patternsis used,this information must be maximized. Low intensity radiancein the result-ing high-dynamicrangephotographis neededto accuratelycalculatethe reflectancecoefficient for diffusematerials. Low intensity radiancein the input data,doesnotnecessarilymapto low intensity radiancein the re-lit resultandlarge relative errors(associatedwith theselow intensityradiancedata)cancausevisualartifacts.

5.1 Illumination Patterns

The illumination patternsarea key elementin our method. From the differentradi-ancevaluesin the high-dynamicrangephotographs,dueto the illumination patterns,a reflectioncoefficient is calculatedanda supportis searchedthrougha optimizationprocedure(section6). Theideaof usingillumination patternsto find a supporton thelight patchwasintroducedby Zongkeret. al.[14] andextendedby Chuanget. al.[1].

Zongker et. al. usedhorizontalandverticalstripesasillumination patterns,whileChuanget. al. usedGaussianstimuli. Thesepatternsarenot directly usablefor ourmethodbecausethey requirealargeamountof photographsto berecorded(thisnumbergrowsevenmorewhentakenin accountthatweusehigh-dynamicrangephotographs).Chuanget. al. also introducean approachfor materialswith neutralcolor. A slicethroughaRGB-cube(agradient)is usedandis sufficient for findingthesupporton thebackdrop.

We wantto limit thenumberof photographsthathave to berecorded,thusamaxi-mumof informationmustbecontainedin a singleillumination pattern.Chuanget. alhave shown thata singlegradientis sufficient to capturewhite materialwith just onephotograph,we extentthis ideato usea numberof differentlyorientedgradients.Thequestionnow remainingis: which gradientsandwhich orientations?

Fromformula (6) follows that the reflectioncoefficient Rp J j canbeeasilyderivedwhena solid illumination patternLw C j E is used.Anotherway to extract thereflection

7

coefficientis to useinverseilluminationpatternsfor oneor for all gradientilluminationpatterns.We definea inverseillumination patternby statingthat thesumof a patternandits inverseequalsa solid constantpattern.This sumcanthenbeusedto estimateRp J j. Theuseof inversepatternshasadvantagesover a solid pattern,becauseit is notmeaningfulto optimizefor asupportoverasolidpattern,while thisstill is possibleforinverseillumination patterns.In practicewe usedboth,but for anotherreason.If forsomereasonsomethinggoeswrong during the recordingof the high-dynamicrangephotographs,it is possibleto reconstructthe lost photographfrom thesolid patternorits inversepattern.

To maketheoptimizationproceduremorerobust,a few otherconstraintsareputonthechoiceof gradients.First of all, smoothgradientsareeasierto optimizefor. Sec-ondly, thedifferencebetweentheaverageillumination of areaswith thesamepositionandsizein differentilluminationpatternsmustbeaslargeaspossible.Theideais thatif thedifferencein appliedillumination is maximized,thedifferencein resultingradi-anceis alsomaximized.Furthermoretheaverageillumination betweendifferentareason thesameillumination patternmustalsobeaslargeaspossible,makingit easiertouniquelyidentify thesupportsfrom eachother.

Figure1: The9 illuminationpatternused.Thesolidpatternis displayedin thecenter. Inversepatternsaredisplayedonoppositesides.

A wide rangeof gradientpatternsarepossible.A choicehasto bemadebetweenaccuracy andthenumberof illuminationpatterns(andthusthenumberof photographsthathave to berecorded).We have chosenfor 4 quadratic(gray-scale)gradientseachdiffering 90 degreesin orientationandstartingat 45 degrees.We alsouse4 inverseillumination patternsand1 solid white pattern(figure 1). Empirical testshow goodresultswith thesepatterns,but aslightblurring occursfor specularmaterials.

5.2 Practical Setup

For thepracticalsetup,we have chosento usea calibratedCRT monitor for eachsideof the hemicube. This choiceis motivatedby the generalavailability and the largeview angleof CRT monitors.We usea 4 a 4 grid of light patchesperhemicubeside,which givesgoodresultsin reproducingvariousmaterialpropertiesandrequiresonlyamoderatenumberof high-dynamicrangephotographs(figure2).

8

bcbcbbcbcbbcbcbbcbcbbcbcbbcbcbbcbcbbcbcbdcdcddcdcddcdcddcdcddcdcddcdcddcdcddcdcd

Figure2: For eachpixel andeachlight patcha reflectionfactorandsupportareais calculated.Our currentsetuplimits monitor placementto 4 sides,excluding the sidewherethe cameraislocated.Eachhemicubesideconsistof a 4 e 4 grid of patchesandwe use9 differentgradientpatternsperpatch.

A calibratedCRT monitoris used,becausewewantto optimizeonradiancevalues.An approachsimularto Chuanget. al.[1] is usedfor themonitorcalibration.A calibra-tion pattern(figure3) containingall possiblegray-scalepixel-valuesis displayedanda high-dynamicrangephotographis recorded.Theradiancevaluesperpixel-valueareextractedandtheinversemappingfrom radiancevalueto pixel-valueis appliedon theilluminationpatterns.Theresultingemittedradianceis now accordingto ourquadraticgradientpattern.

Figure 3: Monitor calibrationpattern. A high-dynamicrangephotographis recordedandsamplesper gray-scalepixel-value are taken at threedifferent location on the screento limitthe effects of noise. The radiancevalue per gray-scalevalue is extractedfrom the recordedphotograph.

The useof a CRT monitor limits monitor placementto 4 sidesof a hemicube,excludingthesidewherethecamerais located.We usea singleCRT monitor, whichis movedaroundfor eachhemicubeside.Theplacementof theCRT monitorhasto becarefullymeasured.Gapsbetweenlight patchesat thebordersresultin visibleartifactsin theresults.

9

6 Processingof the input data

Oncethe high-dynamicrangephotographsarerecorded,the reflectancecoefficient iscalculatedandsupportis searchedthroughanoptimizationprocedureperpixel andperlight-patch.

The reflectancecoefficient Rp J j canbedirectly calculatedfrom therecordedradi-ancevaluesof theillumination from asolid patternor thecombinationof illuminationpatternsandtheir inversepatterns.

A least-squaredminimizationperpixel andper light patchΩ j is usedto optimizethesupport.Theleast-squarederroris definedby (usingthenotationfrom section3):

ErrorΩ j C p E B ∑k

C Rp J j HΩ j

S j C θi D φi E Lik C θi D φi E dω \ Lk C p EfE 2wherek is thenumberof patternsweoptimizeonandLk C p E is therecordedradiance

at pixel p illuminatedwith Lik C θ D φ E , the incomingradiancefrom patternk. We useasteepestdescentoptimizationalgorithmperpixel andperpatch:

Algorithm 1:

1) Select an initial support2) Calculate the least-squared error3) while not minimized

a) Test all (80) expansion possibilities of the supportb) Calculate least-squared errors and select minimum errorc) if minimum error < error then update support

else minimized = true

The80possibleexpansionareobtainedby movingasideof theaxisalignedsupport1 stepto bothdirections.Combiningthis for 2, 3 and4 sidesresultsin 80possibilities.

Thealgorithmcanbeoptimizedby usinga Levenburg-Marquardtoptimizationoralternatively, insteadof on expandingthe support1 pixel in the illumination pattern,alsoexpandx pixels(e.g.5%of thetotalsizeof thepattern),thishelpsspeedinguptheoptimizationprocessandavoidingsmalllocalminima.

Thechoiceof the initial supportΩ F j is doneasfollows: if thedifferencein pixel-valuewith thepreviouspixel is lessthenacertainthreshold,thesupportof thepreviouspixel is used,otherwisetheentirelight patchis setasinitial support.The ideais thatsimularneighboringpixelsprobablyhavethesamematerialpropertiesandthushaveasimularsupport.

10

Figure4: We plottedthe4 sidesof thehemicubefor 4 differentpixels.At eachhemicubesidethedifferentoptimizedsupportsareshown. (theplot perhemicubeis organizedasfollows: up;left, back,right)

ThereflectancecoefficientRp J j andthesupportΩ F j (figure4) arestoredondisk forre-lighting afterwards. A per pixel or per scanlinecompressionis possibleto reducestoragerequirements.

7 Re-lighting

To re-light thesceneusinganarbitraryhigh-dynamicrangeenvironmentmap,theillu-minationis averagedover the supportandmultiplied by the reflectioncoefficient, foreachpixel andfor eachpatch.

8 Discussion

Dueto thesubdivision of thehemicubein light patches,diffusematerialsarecapturedcorrectly. However, thissubdivisionis toocoarsefor correctlycapturingspecularhigh-lights. The useof of illumination patternsmakesit possibleto find sub-patchareaswhich are neededto simulatethe small solid anglerequiredfor specularmaterials.Thisapproachcombinesthestrengthsof theLight Stageandenvironmentmatting.

In section3.3and3.4weshowedthatboththeLight Stageandenvironmentmattingcanbederivedfrom there-lightingequation(formula(5)).

Thesubdivision of thehemicubematchesthefixedpositionof light sourceon thespherein the Light Stage(the sumin formula(5)). This limited angularresolutionissufficient to capturediffusematerialsandreproducethemfaithfully. Theuseof a sup-port (the integral in formula(5))directly mapsto environmentmattingandenablesusto find small areason the environmentthat matchesthe small solid angleneededforspecularmaterials.While eachtechnique,the Light Stageandenvironmentmatting,fail to reproducerespectively specularanddiffusematerials,ourmethodcleverly com-binesthestrengthsof eachtechniqueandis ableto re-light specularaswell asdiffusematerials.

11

9 Results

Theresultsshow two scenes.Onecontaininga Venetianmaskanda decorative glasssphereanda diffusecoloredsurface. The otherscenedisplaysa glossywoodencaron a diffuse surface. Eachsceneis re-lit with 4 different environmentmaps. Thescenewith theVenetianmaskshowstheaccuratecapturingandreproductionof diffusematerials(surface),specularandtransparentmaterials(glasssphereandpearlon topof themask)andcaustics(in thebaseof theglasssphere).Thescenewith thewoodencarshowsglossyhighlights(theroundcurves)andsoft shadows(on thesurfacebelowthewoodencar).

We recordedthesescenesusinga 19” Iiyama CRT monitor for eachsideof thehemicube.Thephotographswererecordedusinga CanonEOSD30 camera.We au-tomatedthe processof capturingandonly the CRT monitor had to be moved. Therecordingtime wasabout10 hours(of which at least6 hourswereoverheadduetotheslow speedof thecamera).Recordingcouldbedonein a matterof minuteswhenusinga synchronizeddigital videocameraanddifferentF-stopfilters to acquirehigh-dynamicrangeimages.

Theprocessinglastedfor 20 hourson a Athlon 1.2Ghrzsystemwith 512meg ram.This processingtime canbedividedamongdifferentcomputersystemsbecauseeachpatchandeachpixel canbeprocessedindependentlyof eachother.

We verifiedthe resultson a virtual sceneby renderingthe sceneilluminatedwiththe target illumination andcomparingit with the re-lit result. The resultsshowed thesamevisualfeatures.A slightblurringoccurswhenhighly specularmaterialsareused,but this is mainly becauseof the choiceof illumination patterns.A choicehasto bemadebetweenrecordingtimeandreproductionquality.

When high-dynamicrangeenvironmentmapsare usedas light map that are ofmuchlargerdynamicrangethentherecordedinput data,visualartifactsappear. Thisis causedby noisein the recordedhigh-dynamicrangephotographsandoptimizationerrorsaugmentedto visual levels.

10 Conclusions

We developeda mathematicalframework in which we derived the re-lighting equa-tion. Using this mathematicalframework we developeda new re-lighting technique.This techniqueis suitedfor capturinga reflectancefield of objects,from a singleviewpoint, consistingof any materialpropertyandwith complex geometry. This methodsis practicallyusableandis ableto capturediffuseandspecularmaterial,causticsandtransparentmaterials.

A whole new rangeof applicationsis possible:the digital captureanddisplayofobjectswith a mix of diffuseandspecularappearance(e.g. jewelry or sensitive ar-chaeologicalobjects),augmentedreality applications,(interactive) re-lightingof fullyvirtual scenes,...

Futureresearchwill focusonfasterandeasiercapturingof thereflectancefield, theuseof otherprojectiondevicessuchasVR-cavesthatcanaccommodatelargerobjects

12

and reducingthe numbernecessaryphotographsby using betterpatternsand betteroptimizationprocedures.

Acknowledgments

We would like to thankFrank SuykensandVincentMasselusfor proof-readingthereportandmany constructive discussions.Furthermorewe would like to thankPaulDebevecfor his highquality environmentmaps(www.debevec.org/Probes).

References

[1] Yung-Yu Chuang,DouglasE. Zongker, JoelHindorff, Brian Curless,David H.Salesin,andRichardSzeliski. Environmentmattingextensions:Towardshigheraccuracy andreal-timecapture.In Kurt Akeley, editor, Siggraph 2000, ComputerGraphics Proceedings, AnnualConferenceSeries,LosAngeles,2000.ACM Sig-graph,AddisonWesley Longman.

[2] Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-PieterDuiker, WestleySarokin,andMark Sagar. Acquiring the reflectancefield of a humanface. InKurt Akeley, editor, Siggraph 2000, Computer Graphics Proceedings, AnnualConferenceSeries,pages145–156,LosAngeles,2000.ACM Siggraph,AddisonWesley Longman.

[3] Paul E. Debevec andJitendraMalik. Recovering high dynamicrangeradiancemapsfrom photographs.In TurnerWhitted,editor, SIGGRAPH 97 ConferenceProceedings, AnnualConferenceSeries,pages369–378.ACM SIGGRAPH,Ad-disonWesley, August1997.

[4] Steven J. Gortler, RadekGrzeszczuk,RichardSzeliski,andMichael F. Cohen.The lumigraph. In Holly Rushmeier, editor, SIGGRAPH 96 Conference Pro-ceedings, AnnualConferenceSeries,pages43–54.ACM SIGGRAPH,AddisonWesley, August1996.heldin New Orleans,Louisiana,04-09August1996.

[5] Tim Hawkins, JonathanCohen,andPaul Debevec. A photometricapproachtodigitizing culturalartifacts.In In 2nd International Symposium on Virtual Reality,Archaeology, and Cultural Heritage, Glyfada, Greece, November 2001., 2001.

[6] MarcLevoy andPatHanrahan.Light field rendering.In Holly Rushmeier, editor,SIGGRAPH 96 Conference Proceedings, Annual ConferenceSeries,pages31–42. ACM SIGGRAPH,Addison Wesley, August 1996. held in New Orleans,Louisiana,04-09August1996.

[7] ZhouchenLin, Tien-TsinWong, andHeungYeungShum. Relightingwith thereflectedirradiancefield: Representation,samplingandreconstruction.In Pro-ceedings of IEEE Computer Vision and Pattern Recognition, December2001.

13

[8] CélineLoscosandGeorgeDrettakis.Low-costphotometriccalibrationfor inter-active relighting. In Proceedings of the First French-British International Work-shop on Virtual Reality, Brest,France,July2000.

[9] CélineLoscos,GeorgeDrettakis,andLuc Robert. Interactive virtual relightingof real scenes.In IEEE Transactions on Visualization and Computer Graphics,volume6(4),pages289–305.IEEE ComputerSociety, 2000.

[10] Jeffry S.Nimeroff, EeroSimoncelli,andJulieDorsey. Efficient Re-renderingofNaturallyIlluminatedEnvironments.In Fifth Eurographics Workshop on Render-ing, pages359–373,Darmstadt,Germany, June1994.

[11] Tien-TsinWong,Pheng-AnnHeng,Siu-HangOr, andWai-Yin Ng. Image-basedrenderingwith controllableillumination. In JulieDorsey andPhilipp Slusallek,editors,Eurographics Rendering Workshop 1997, pages13–22,New York City,NY, June1997.Eurographics,SpringerWien.

[12] YizhouYu, Paul Debevec,JitendraMalik, andTim Hawkins. Inverseglobalillu-mination: Recoveringreflectancemodelsof realscenesfrom photographsfrom.In Alyn Rockwood,editor, Siggraph99, Annual Conference Series, AnnualCon-ferenceSeries,pages215–224,Los Angeles,1999. ACM Siggraph,AddisonWesley Longman.

[13] YizhouYu andJitendraMalik. Recoveringphotometricpropertiesof architecturalscenesfrom photographs.In MichaelCohen,editor, SIGGRAPH 98 ConferenceProceedings, AnnualConferenceSeries,pages207–218.ACM SIGGRAPH,Ad-disonWesley, July 1998.

[14] DouglasE. Zongker, Dawn M. Werner, Brian Curless,and David H. Salesin.Environmentmatting andcompositing. In Alyn Rockwood, editor, Sigggraph1999, Computer Graphics Proceedings, Annual ConferenceSeries,pages205–214,LosAngeles,1999.ACM Siggraph,AddisonWesley Longman.

14

Figure5: A Venetianmaskwith glasssphereandwoodencarlit by avirtual environmentmap.

Figure6: A Venetianmaskwith glasssphereandwoodencarlit by a environmentmapof a beach.

Figure7: A Venetianmaskwith glasssphereandwoodencarlit by a environmentmapof theuffizi galleria.

Figure8: A Venetianmaskwith glassspherelit by an environmentmapof a buildingwith a glassceiling anda woodencar lit by anenvironmentmaprecordedattheparkingspaceoutsideour office.

15