Applied Perception in Applied Perception in GraphicsGraphics

Erik ReinhardErik Reinhard

University of Central FloridaUniversity of Central Florida

reinhard@cs.ucf.edureinhard@cs.ucf.edu

Computer GraphicsComputer Graphics

• Produce computer Produce computer generated imagery that generated imagery that cannot be cannot be distinguished from real distinguished from real scenesscenes

• Do this in real-timeDo this in real-time

Trends in Computer GraphicsTrends in Computer Graphics

• Greater realismGreater realism– Scene complexityScene complexity– Lighting simulations Lighting simulations

• Faster renderingFaster rendering– Faster hardwareFaster hardware– Better algorithmsBetter algorithms

• Together: still too slow and unrealisticTogether: still too slow and unrealistic

Algorithm designAlgorithm design

• Largely opportunisticLargely opportunistic

• Computer graphics is a maturing fieldComputer graphics is a maturing field

• Hence, a more directed approach is neededHence, a more directed approach is needed

Long Term StrategyLong Term Strategy

• Understand the differences between natural Understand the differences between natural and computer generated scenesand computer generated scenes

• Understand the Human Visual System and Understand the Human Visual System and how it perceives imageshow it perceives images

• Apply this knowledge to motivate graphics Apply this knowledge to motivate graphics algorithmsalgorithms

This Presentation (1)This Presentation (1)

Reinhard et. al., “Color Transfer between Images”, IEEE CG&A, Sept. 2001.

This Presentation(2)This Presentation(2)

Reinhard et. al., “Photographic Tone Reproduction for Digital Images, SIGGRAPH 2002.

IntroductionIntroduction

The Human Visual System is evolved to look at natural images

Natural Random

Human Visual SystemHuman Visual System

RetinaRetina

Color ProcessingColor Processing

Rod and Cone pigments

Color ProcessingColor Processing

Cone output is logarithmic

Color opponent space

Image StatisticsImage Statistics

• Ruderman’s work on color statistics:Ruderman’s work on color statistics:

– Principal Components Analysis (PCA) on colors of Principal Components Analysis (PCA) on colors of natural image ensemblesnatural image ensembles

– Axes have meaning: color opponents Axes have meaning: color opponents (luminance, red-green and yellow-blue)(luminance, red-green and yellow-blue)

Color Processing SummaryColor Processing Summary

• Human Visual System expects images with natural Human Visual System expects images with natural characteristics (not just color)characteristics (not just color)

• Color opponent space has decorrelated axes (but Color opponent space has decorrelated axes (but in practice close to independent)in practice close to independent)

• Color space is logarithmic (compact and symmetric Color space is logarithmic (compact and symmetric data representation)data representation)

• Independent processing along each axis should be Independent processing along each axis should be possible possible Application Application

LL Color Space Color Space

Convert RGB triplets Convert RGB triplets to LMS cone spaceto LMS cone space

Take logarithmTake logarithm

Rotate axesRotate axes

Color TransferColor Transfer

• Make one image look like anotherMake one image look like another

• For both images:For both images:– Transfer to new color spaceTransfer to new color space– Compute mean and standard deviation along Compute mean and standard deviation along

each color axiseach color axis

• Shift and scale the target image to have the Shift and scale the target image to have the same statistics as the source imagesame statistics as the source image

Why not use RGB space?Why not use RGB space?

Input images Output images

RGB

L

Color Transfer ExampleColor Transfer Example

Color Transfer ExampleColor Transfer Example

Color Transfer ExampleColor Transfer Example

Color Processing SummaryColor Processing Summary

• Changing the statistics along each axis Changing the statistics along each axis independently allows one image to resemble independently allows one image to resemble a second imagea second image

• If the composition of the images is very If the composition of the images is very unequal, an approach using small swatches unequal, an approach using small swatches may be used successfullymay be used successfully

Tone ReproductionTone Reproduction

Tone ReproductionTone Reproduction

Global vs. LocalGlobal vs. Local

• GlobalGlobal– Scale each pixel according to a fixed curveScale each pixel according to a fixed curve– Key issue: shape of curveKey issue: shape of curve

• LocalLocal– Scale each pixel by a curve that is modulated by Scale each pixel by a curve that is modulated by

a local averagea local average– Key issue: size of local neighborhoodKey issue: size of local neighborhood

Global OperatorsGlobal Operators

TumblinWard

Ferwerda

Global OperatorsGlobal Operators

TumblinWard

Ferwerda

Local OperatorLocal Operator

Pattanaik

Spatial ProcessingSpatial Processing

• Circularly symmetric Circularly symmetric receptive fieldsreceptive fields

• Centre-surround mechanismsCentre-surround mechanisms– Laplacian of GaussianLaplacian of Gaussian– Difference of GaussiansDifference of Gaussians– BlommaertBlommaert

• Scale space modelScale space model

Scale Space Scale Space (Histogram Equalized Images)(Histogram Equalized Images)

Tone Reproduction IdeaTone Reproduction Idea

• Modify existing global Modify existing global operator to be a local operator to be a local operator, e.g. Greg Ward’soperator, e.g. Greg Ward’s

• Use spatial processing to Use spatial processing to determine a local determine a local adaptation level for each adaptation level for each pixelpixel

5.2

4.0

4.0

max

max

5.2

4.0

4.0

max

max

),(219.1

2219.1),(

219.1

2219.1),(

yxL

L

L

yxLL

L

L

L

yxLL

a

d

doutput

w

d

doutput

Blommaert Brightness ModelBlommaert Brightness Model

22

2

22

1 sk

r

ii

iesk

R

ii RvuLsyxV ),(),,(

),,(2

),,(),,()(),,(

12

21

syxVs

syxVsyxVsWsyxV

ns

s

syxVyxB0

),,(),(

Gaussian filter

Center/surround

Neural response

Brightness

BrightnessBrightness

ns

s

syxVyxB0

),,(),(

Scale Selection AlternativesScale Selection Alternatives

0

)),(,,(),(

ss

yxsyxVyxB

n

m

Mean value

Thresholded )),(,,(: yxsyxVs mm

How large should a local neighborhood be?

Mean ValueMean Value

0

)),(,,(),(

ss

yxsyxVyxB

n

m

ThresholdedThresholded

),,(: mm syxVs

Tone-mappingTone-mapping

5.2

4.0

4.0

max

max

1

),(219.1

2219.1),(

)),(,,(),(

yxL

L

L

yxLL

yxsyxVyxL

a

d

doutput

maLocal adaptation

Greg Ward’s tone-mapping with local adaptation

ResultsResults

• Good results, but something odd about scale Good results, but something odd about scale selection:selection:

• For most pixels, a large scale was selectedFor most pixels, a large scale was selected

• Implication: a simpler algorithm should be Implication: a simpler algorithm should be possiblepossible

Simplify AlgorithmSimplify Algorithm

),(),(

),(1

),(

),(219.1

2219.1),(

5.2

4.0

4.0

max

max

yxLL

ayxL

yxL

yxLL

yxL

L

L

yxLL

ww

output

a

d

doutput

Greg Ward’s tone-mapping with local adaptation

Simplify

Fix overall lightness of image

Global Operator ResultsGlobal Operator Results

WardOur method

Global Operator ResultsGlobal Operator Results

WardOur method

Global Global Local Local

)),(,,(1

),(

),(1

),(

1 yxsyxV

yxLL

yxL

yxLL

moutput

output

Global operator

Local operator

Local Operator ResultsLocal Operator Results

Global

Local

Local Operator ResultsLocal Operator Results

Global Local Pattanaik

SummarySummary

• Knowledge of the Human Visual System can Knowledge of the Human Visual System can help solve engineering problemshelp solve engineering problems

• Color and spatial processing investigatedColor and spatial processing investigated

• Direct applications shownDirect applications shown

Future WorkFuture Work

This presentation

AcknowledgmentsAcknowledgments

• Thanks to my collaborators: Peter Shirley, Thanks to my collaborators: Peter Shirley, Jim Ferwerda, Mike Stark, Mikhael Jim Ferwerda, Mike Stark, Mikhael Ashikhmin, Bruce Gooch, Tom TrosciankoAshikhmin, Bruce Gooch, Tom Troscianko

• This work sponsored by NSF grants 97-This work sponsored by NSF grants 97-96136, 97-31859, 98-18344, 99-78099 and 96136, 97-31859, 98-18344, 99-78099 and by the DOE AVTC/VIEWSby the DOE AVTC/VIEWS