INFORMATIK Design of a Tone Mapping Operator for High Dynamic Range Images based upon Psychophysical Evaluation and Preference Mapping Design of a Tone

INFORMATIKINFORMATIK

Design of a Tone Mapping Operator Design of a Tone Mapping Operator for High Dynamic Range Images based for High Dynamic Range Images based upon Psychophysical Evaluation and upon Psychophysical Evaluation and

Preference MappingPreference Mapping

F. DragoF. Drago11, W. Martens, W. Martens22, K. Myszkowski, K. Myszkowski33, , and N. Chibaand N. Chiba11

11Iwate University and Iwate University and 22Aizu University, JapanAizu University, Japan33Max-Planck-Institut fMax-Planck-Institut füür Informatik, Germanyr Informatik, Germany

F. DragoF. Drago11, W. Martens, W. Martens22, K. Myszkowski, K. Myszkowski33, , and N. Chibaand N. Chiba11

11Iwate University and Iwate University and 22Aizu University, JapanAizu University, Japan33Max-Planck-Institut fMax-Planck-Institut füür Informatik, Germanyr Informatik, Germany

INFORMATIKINFORMATIKOverviewOverview

• MotivationMotivation• Previous workPrevious work• Psychophysical experimentPsychophysical experiment• Enhancements of Retinex for HDR imagesEnhancements of Retinex for HDR images• ConclusionsConclusions

• MotivationMotivation• Previous workPrevious work• Psychophysical experimentPsychophysical experiment• Enhancements of Retinex for HDR imagesEnhancements of Retinex for HDR images• ConclusionsConclusions

INFORMATIKINFORMATIKMotivationMotivation

Many applicationsMany applications• Lighting simulation and realistic renderingLighting simulation and realistic rendering• High Dynamic Range photographyHigh Dynamic Range photography• Multimedia: distributing HDR video streamsMultimedia: distributing HDR video streams

Many applicationsMany applications• Lighting simulation and realistic renderingLighting simulation and realistic rendering• High Dynamic Range photographyHigh Dynamic Range photography• Multimedia: distributing HDR video streamsMultimedia: distributing HDR video streams


HDR Photographs + Rendering: HDR Photographs + Rendering: Real World LightingReal World Lighting

1) Photographs of mirror sphere at varying exposure times

2) High-dynamicrange environment map

3) Use as light source in Monte Carlo radiosity algorithm

Philippe Bekaert

INFORMATIKINFORMATIKGoalsGoals• Technical requirement Technical requirement

– Match the dynamic range of image to the range Match the dynamic range of image to the range available on a given display deviceavailable on a given display device

• Various objectivesVarious objectives– Get good perceptual match between the real-world Get good perceptual match between the real-world

and corresponding imagesand corresponding images– Reproducing detailsReproducing details– Maximize reproducible contrastMaximize reproducible contrast– Just to get “nice-looking” imagesJust to get “nice-looking” images

• Technical requirement Technical requirement – Match the dynamic range of image to the range Match the dynamic range of image to the range

available on a given display deviceavailable on a given display device

• Various objectivesVarious objectives– Get good perceptual match between the real-world Get good perceptual match between the real-world

and corresponding imagesand corresponding images– Reproducing detailsReproducing details– Maximize reproducible contrastMaximize reproducible contrast– Just to get “nice-looking” imagesJust to get “nice-looking” images

INFORMATIKINFORMATIKVarious ClassificationsVarious Classifications

• Theoretical foundationsTheoretical foundations– Perception-based Perception-based – Pure image processing techniquesPure image processing techniques

• Mapping functionMapping function– Global – the same for all pixelsGlobal – the same for all pixels– Local – depends on local image contentsLocal – depends on local image contents

• Temporal processingTemporal processing– StaticStatic– DynamicDynamic

• Theoretical foundationsTheoretical foundations– Perception-based Perception-based – Pure image processing techniquesPure image processing techniques

• Mapping functionMapping function– Global – the same for all pixelsGlobal – the same for all pixels– Local – depends on local image contentsLocal – depends on local image contents

• Temporal processingTemporal processing– StaticStatic– DynamicDynamic


Previous Work:Previous Work:Global MethodsGlobal Methods

Perception-basedPerception-based• Tumblin and Rushmeier (1993,1999)Tumblin and Rushmeier (1993,1999)

– Brightness matchingBrightness matching• Ward (1994), Ferwerda et al. (1996)Ward (1994), Ferwerda et al. (1996)

– Contrast matching (a linear function is used)Contrast matching (a linear function is used)• Ward et al. (1997)Ward et al. (1997)

– Adjusting image histogram to avoid exceeding Adjusting image histogram to avoid exceeding display contrast in respect to the real-world scenedisplay contrast in respect to the real-world scene

Efficiency-drivenEfficiency-driven• Schlick (1994)Schlick (1994)

– Rational functionsRational functions

Perception-basedPerception-based• Tumblin and Rushmeier (1993,1999)Tumblin and Rushmeier (1993,1999)

– Brightness matchingBrightness matching• Ward (1994), Ferwerda et al. (1996)Ward (1994), Ferwerda et al. (1996)

– Contrast matching (a linear function is used)Contrast matching (a linear function is used)• Ward et al. (1997)Ward et al. (1997)

– Adjusting image histogram to avoid exceeding Adjusting image histogram to avoid exceeding display contrast in respect to the real-world scenedisplay contrast in respect to the real-world scene

Efficiency-drivenEfficiency-driven• Schlick (1994)Schlick (1994)

– Rational functionsRational functions

INFORMATIKINFORMATIKExamplesExamples

Ferwerda et al. Tumblin (1999) Ward et al. SchlickFerwerda et al. Tumblin (1999) Ward et al. Schlick


Previous Work:Previous Work:Local MethodsLocal Methods

• Early methods – prone to halo artifactsEarly methods – prone to halo artifacts– Chiu et al. (1993), Schlick (1994), Chiu et al. (1993), Schlick (1994), – Land (1971), Jobson et al. (1997): RetinexLand (1971), Jobson et al. (1997): Retinex– Pattanaik et al. (1998): The most comprehensive Pattanaik et al. (1998): The most comprehensive

model of HVS used in CGmodel of HVS used in CG• LCIS: Tumblin and Turk (1999)LCIS: Tumblin and Turk (1999)

– Based on an anisotropic diffusion procedureBased on an anisotropic diffusion procedure– Emphasize on details but compress excessively Emphasize on details but compress excessively

contrastcontrast• New wave: Fattal et al., Reinhard et al., Durand and New wave: Fattal et al., Reinhard et al., Durand and

Dorsey, Ashikhmin (2002)Dorsey, Ashikhmin (2002)

• Early methods – prone to halo artifactsEarly methods – prone to halo artifacts– Chiu et al. (1993), Schlick (1994), Chiu et al. (1993), Schlick (1994), – Land (1971), Jobson et al. (1997): RetinexLand (1971), Jobson et al. (1997): Retinex– Pattanaik et al. (1998): The most comprehensive Pattanaik et al. (1998): The most comprehensive

model of HVS used in CGmodel of HVS used in CG• LCIS: Tumblin and Turk (1999)LCIS: Tumblin and Turk (1999)

– Based on an anisotropic diffusion procedureBased on an anisotropic diffusion procedure– Emphasize on details but compress excessively Emphasize on details but compress excessively

contrastcontrast• New wave: Fattal et al., Reinhard et al., Durand and New wave: Fattal et al., Reinhard et al., Durand and

Dorsey, Ashikhmin (2002)Dorsey, Ashikhmin (2002)


Tumblin and Turk Retinex Ashikhmin

ExamplesExamples


Durand and Dorsey Reinhard et al. Fattal et al.

ExamplesExamples


Durand and Dorsey

Reinhard et al. Fattal et al.

Ashikhmin

INFORMATIKINFORMATIKPsychophysical ExperimentPsychophysical Experiment• Perceptual evaluation of subject preference by pairwise Perceptual evaluation of subject preference by pairwise

comparison of tone mapped imagescomparison of tone mapped images

• Seven tone mapping algorithms examined: Seven tone mapping algorithms examined:

– Tumblin and Rushmeier (1993), Tumblin and Rushmeier (1993),

– Ferwerda et al. (1996), Ferwerda et al. (1996),

– Ward et al. (1997), Ward et al. (1997),

– Schlick (1994), Schlick (1994),

– Retinex - based on Funt and Ciurea (2001) implementation Retinex - based on Funt and Ciurea (2001) implementation but with our extensions toward suppressing halobut with our extensions toward suppressing halo

– Reinhard et al. (2002) – photographic methodReinhard et al. (2002) – photographic method

– Tumblin and Turk (1999) - LCISTumblin and Turk (1999) - LCIS

• Four scenes consideredFour scenes considered

• Perceptual evaluation of subject preference by pairwise Perceptual evaluation of subject preference by pairwise comparison of tone mapped imagescomparison of tone mapped images

• Seven tone mapping algorithms examined: Seven tone mapping algorithms examined:

– Tumblin and Rushmeier (1993), Tumblin and Rushmeier (1993),

– Ferwerda et al. (1996), Ferwerda et al. (1996),

– Ward et al. (1997), Ward et al. (1997),

– Schlick (1994), Schlick (1994),

– Retinex - based on Funt and Ciurea (2001) implementation Retinex - based on Funt and Ciurea (2001) implementation but with our extensions toward suppressing halobut with our extensions toward suppressing halo

– Reinhard et al. (2002) – photographic methodReinhard et al. (2002) – photographic method

– Tumblin and Turk (1999) - LCISTumblin and Turk (1999) - LCIS

• Four scenes consideredFour scenes considered


INFORMATIKINFORMATIKStatistical Data ProcessingStatistical Data Processing

• 11 subjects participated11 subjects participated• Dissimilarity ratings for pairwise comparisons of Dissimilarity ratings for pairwise comparisons of

images submitted to Individual Differences Scaling images submitted to Individual Differences Scaling (INDSCAL) analysis(INDSCAL) analysis

• Stimulus Space configures the stimuli such that Stimulus Space configures the stimuli such that Euclidian distances between the stimuli match the Euclidian distances between the stimuli match the obtained dissimilarity judgmentsobtained dissimilarity judgments

• Axes labeled based upon correlation of the Axes labeled based upon correlation of the dimensional coordinates with independently dimensional coordinates with independently generated attribute ratings (naturalness, detail and generated attribute ratings (naturalness, detail and contrast reproduction)contrast reproduction)

• ““Ideal” preference point obtained through PREFMAP Ideal” preference point obtained through PREFMAP analysisanalysis

• 11 subjects participated11 subjects participated• Dissimilarity ratings for pairwise comparisons of Dissimilarity ratings for pairwise comparisons of

images submitted to Individual Differences Scaling images submitted to Individual Differences Scaling (INDSCAL) analysis(INDSCAL) analysis

• Stimulus Space configures the stimuli such that Stimulus Space configures the stimuli such that Euclidian distances between the stimuli match the Euclidian distances between the stimuli match the obtained dissimilarity judgmentsobtained dissimilarity judgments

• Axes labeled based upon correlation of the Axes labeled based upon correlation of the dimensional coordinates with independently dimensional coordinates with independently generated attribute ratings (naturalness, detail and generated attribute ratings (naturalness, detail and contrast reproduction)contrast reproduction)

• ““Ideal” preference point obtained through PREFMAP Ideal” preference point obtained through PREFMAP analysisanalysis

INFORMATIKINFORMATIKSubject PreferencesSubject Preferences– TT: Tumblin & R.: Tumblin & R.

– VV: Ferwerda et al. : Ferwerda et al.

– HH: Ward et al. : Ward et al.

– QQ: Schlick : Schlick

– XX: Retinex: Retinex

– PP: Reinhard et al. : Reinhard et al.

– TT: Tumblin & R.: Tumblin & R.

– VV: Ferwerda et al. : Ferwerda et al.

– HH: Ward et al. : Ward et al.

– QQ: Schlick : Schlick

– XX: Retinex: Retinex

– PP: Reinhard et al. : Reinhard et al.

INFORMATIKINFORMATIKRetinexRetinexWe use the “Frankle-McCann Retinex” algorithmWe use the “Frankle-McCann Retinex” algorithm• ratio-product-reset-averageratio-product-reset-average

– NP(x,y) new pixel value is obtained from the original image R() and previous iteration image OP() as follows: NP(x,y) new pixel value is obtained from the original image R() and previous iteration image OP() as follows:

– Reset testReset test

• In each iteration (the number of iterations predefined by the user)In each iteration (the number of iterations predefined by the user)

– the distance D between pixels (x,y) and (xs,ys) is halved the distance D between pixels (x,y) and (xs,ys) is halved

– the direction for pixel comparison is rotated 90the direction for pixel comparison is rotated 90oo clockwise clockwise

• Main problem: Suppressing halo effectsMain problem: Suppressing halo effects

We use the “Frankle-McCann Retinex” algorithmWe use the “Frankle-McCann Retinex” algorithm• ratio-product-reset-averageratio-product-reset-average

– NP(x,y) new pixel value is obtained from the original image R() and previous iteration image OP() as follows: NP(x,y) new pixel value is obtained from the original image R() and previous iteration image OP() as follows:

– Reset testReset test

• In each iteration (the number of iterations predefined by the user)In each iteration (the number of iterations predefined by the user)

– the distance D between pixels (x,y) and (xs,ys) is halved the distance D between pixels (x,y) and (xs,ys) is halved

– the direction for pixel comparison is rotated 90the direction for pixel comparison is rotated 90oo clockwise clockwise


2

),(log)),(log(),(log),((log),(log

yxOPysxsRyxRysxsOPyxNP

sceneLysxsRyxRysxsOP maxlog)),(log(),(log),((log

INFORMATIKINFORMATIKRetinex Extensions: for HDRRetinex Extensions: for HDR


– Adding counterclockwise rotation of the pathAdding counterclockwise rotation of the path

suggested by Cooperssuggested by Coopers

– Spatially varying levels of pixel interaction based Spatially varying levels of pixel interaction based contrast informationcontrast information

Suggested by Sobol, but we use a smooth Suggested by Sobol, but we use a smooth function for clippingfunction for clipping

– Adjusting a reset ratio to the maximum luminance Adjusting a reset ratio to the maximum luminance of the display device instead of the maximum of the display device instead of the maximum luminance of the scene luminance of the scene


– Adding counterclockwise rotation of the pathAdding counterclockwise rotation of the path

suggested by Cooperssuggested by Coopers

– Spatially varying levels of pixel interaction based Spatially varying levels of pixel interaction based contrast informationcontrast information

Suggested by Sobol, but we use a smooth Suggested by Sobol, but we use a smooth function for clippingfunction for clipping

– Adjusting a reset ratio to the maximum luminance Adjusting a reset ratio to the maximum luminance of the display device instead of the maximum of the display device instead of the maximum luminance of the scene luminance of the scene


Halo Reduction: Halo Reduction: Retinex RotationRetinex Rotation

CounterClockwiseClockwise Both Ways

All images for 40 iterations


Halo Reduction:Halo Reduction:Retinex Contrast Crop with BiasRetinex Contrast Crop with Bias

5.0log

log

)(functionBiasa

xxf

7.05.0log

log

a

DipContrastCl

;

)(

;

)(

),(log),(log

ipContrastClContrast

ipContrastClContrastifelse

ipContrastClContrast

ipContrastClContrastif

ysxsRyxRContrast

2

),(log)),(log(),(log),((log),(log

yxOPysxsRyxRysxsOPyxNP



Standard Retinex Standard Retinex 33 iterations cw and ccw33 iterations cw and ccw

The same settingsThe same settingsbut crop with bias addedbut crop with bias added



95.08.0

7.05.0log

log

aa

DipContrastCla

33 Retinex iterations33 Retinex iterations 33 Retinex iterations33 Retinex iterations



95.0

7.05.0log

log

a

DipContrastCla

4 Retinex iterations4 Retinex iterations 30 Retinex iterations30 Retinex iterations

INFORMATIKINFORMATIKRetinex Maximum Reset Retinex Maximum Reset

Maximum = 226.5 cd/m^2 Maximum = 100 cd/m^2

displayLysxsRyxRysxsOP maxlog)),(log(),(log),((log

display

scene

L

L

max

max

to

fromchange


LinearLinearmappingmapping

RetinexRetinex4 iterations4 iterations

ExtendedExtendedRetinexRetinex

4 iterations4 iterations

ExtendedExtendedRetinexRetinex

4 iterations4 iterations

sceneLmaxdisplayLmax

INFORMATIKINFORMATIKRetinex + Tone Mapping Op.Retinex + Tone Mapping Op.

Ferwerda et al. (1996)Ferwerda et al. (1996) Logmap - newLogmap - new

INFORMATIKINFORMATIKLogmap EquationLogmap Equation


Performance:Performance:• SoftwareSoftware

– 30 fps on 30 fps on PentiumIV, PentiumIV, 2.2GHz2.2GHz

• HardwareHardware– ??

Performance:Performance:• SoftwareSoftware

– 30 fps on 30 fps on PentiumIV, PentiumIV, 2.2GHz2.2GHz

• HardwareHardware– ??

Adaptive Adaptive Logarithmic Logarithmic Mapping Mapping

INFORMATIKINFORMATIKConclusionsConclusions

• We performed psychophysical of seven existing tone We performed psychophysical of seven existing tone mapping operators. More details in our TechRep:mapping operators. More details in our TechRep:

http://data.mpi-sb.mpg.de/internet/reports.nsf/AG4NumberView?OpenView

• Good performance of Retinex in the experiment Good performance of Retinex in the experiment encouraged us extend it toward reducing hallo encouraged us extend it toward reducing hallo artifactsartifacts

• Addind a regular tone mapping processing atop of Addind a regular tone mapping processing atop of Retinex results make the resulting images more Retinex results make the resulting images more independent on the number of Retinex iterations and independent on the number of Retinex iterations and improve the image naturalnessimprove the image naturalness

• Future work: repeating psychophysical with all recent Future work: repeating psychophysical with all recent local tone mapping operators and our extended local tone mapping operators and our extended Retinex Retinex

• We performed psychophysical of seven existing tone We performed psychophysical of seven existing tone mapping operators. More details in our TechRep:mapping operators. More details in our TechRep:

http://data.mpi-sb.mpg.de/internet/reports.nsf/AG4NumberView?OpenView

• Good performance of Retinex in the experiment Good performance of Retinex in the experiment encouraged us extend it toward reducing hallo encouraged us extend it toward reducing hallo artifactsartifacts

• Addind a regular tone mapping processing atop of Addind a regular tone mapping processing atop of Retinex results make the resulting images more Retinex results make the resulting images more independent on the number of Retinex iterations and independent on the number of Retinex iterations and improve the image naturalnessimprove the image naturalness

• Future work: repeating psychophysical with all recent Future work: repeating psychophysical with all recent local tone mapping operators and our extended local tone mapping operators and our extended Retinex Retinex

INFORMATIKINFORMATIKColor Balance CorrectionColor Balance Correction

Retinex Applied to All Channels Retinex Applied to All Channels in LMS Color Spacein LMS Color Space


Stanford Memorial Church Stanford Memorial Church PhotographPhotograph


Stanford Memorial Church Stanford Memorial Church PhotographPhotograph

INFORMATIKINFORMATIKAcknowledgments Acknowledgments

We would like to thank Michael Ashikhmin, We would like to thank Michael Ashikhmin, Paul Debevec, Fredo Durand, Dani Paul Debevec, Fredo Durand, Dani Lischinski, Eric Reinhard, and Greg Ward for Lischinski, Eric Reinhard, and Greg Ward for providing us with some images used in this providing us with some images used in this presentation.presentation.

We would like also to thank Greg Ward for We would like also to thank Greg Ward for his precious comments concerning our his precious comments concerning our work.work.

We would like to thank Michael Ashikhmin, We would like to thank Michael Ashikhmin, Paul Debevec, Fredo Durand, Dani Paul Debevec, Fredo Durand, Dani Lischinski, Eric Reinhard, and Greg Ward for Lischinski, Eric Reinhard, and Greg Ward for providing us with some images used in this providing us with some images used in this presentation.presentation.

We would like also to thank Greg Ward for We would like also to thank Greg Ward for his precious comments concerning our his precious comments concerning our work.work.

Documents

INFORMATIK Design of a Tone Mapping Operator for High Dynamic Range Images based upon Psychophysical Evaluation and Preference Mapping Design of a Tone