Visual Perception in Realistic Image Synthesis Ann McNamara

Visual Perception in Realistic Image Synthesis

Ann McNamara

Outline

• Introduction• Modeling important characteristics

of the human visual system (HVS)• Perception based rendering• Image quality metrics• Tone reproduction operators• Summary

Realism

• Architecture• Stage lighting• Entertainment• Safety systems• Archaeology

Human Visual System

• Physical structure well established

• Perceptual behaviour is a complex process

Modeling Important Characteristics

of the Human Visual System

Visual Acuity

• How well we can see fine detail • Adaptation level• Rods and cones

Cones

• Number of grating that fall on one degree of the retina

• Dependent on distance

Spatial Frequency

Spatial Frequency

•Spatial mechanisms (channels) which are used to represent the visual information at various scales and orientations as it is believed that primary visual cortex does.

Contrast Sensitivity Function

•Contrast sensitivity function which specifies the detection threshold for a stimulus as a function of its spatial frequencies.

Campbell-Robson contrast sensitivity chartCampbell-Robson contrast sensitivity chart

Contrast Sensitivity

Masking

•Visual masking affecting the detection threshold of a stimulus as a function of the interfering background stimulus which is closely coupled in space and time.

Masking

Colour Appearance

Perceptually Based Rendering

[Mitchell 1987]]

Low Sampling Densities

•Non-uniform sampling is less conspicuous

•Optimise using how the eye perceives noise as a function of contrast and colour

Raytracing -> Point Samples-> Aliasing

Uniform

Non-Uniform

Adaptive

Sampling Schemes[Mitchell

1987]]

[Mitchell 1987]]

Low Sampling Densities

• Contrast

• Colour

IIIIC

minmax

minmax

R 0.4 G 0.3 B 0.6

[Mitchell 1987]]

Low Sampling Densities

Frequency Based Raytracing[Bolin &Meyer 1992]]

• Synthesise directly into frequency domain

• Simple vision model to control •Where to cast rays•How to spawn rays

Frequency Based Raytracing[Bolin &Meyer 1992]]

• Vision model•Contrast sensitivity•Spatial frequency•Masking

Frequency Based Raytracing[Bolin &Meyer 1992]]

• Specific luminance difference at low intensity more important than same luminance difference at high intensity

• Colour spatial frequency variations given fewer samples

• Decrease rays spawned in high frequency regions

Limited Color Acuity[Meyer & Liu1998]]

• Colour Abberation• Limited sampling of receptor• Spatial acuity of opponent

channels

[Meyer & Liu1998]]

Application

• How much computation is enough?• How much reduction is too much?• An objective metric of image quality

which takes into account basic characteristics of the HVS could help to answer these questions without human assistance.

Questions of Appearance Preservation

The Concern Is Not Whether Images Are the Same

Rather the Concern Is Whether Images Appear the Same

Perceptually Based Adaptive Sampling Algorithm

[Bolin &Meyer 1998]]

• Image quality model embedded into image synthesis

• Use statistical information about spatial frequency to determine where to estimate values where samples were yet to be taken

Perceptually Based Adaptive Sampling Algorithm

[Bolin &Meyer 1998]]

JND’s

VDM

= 200s= 200s = 400s= 400s = 800s= 800s = 1600s= 1600s

Deterministic radiosityDeterministic radiosity

Monte Carlo radiosityMonte Carlo radiosity

Convergence Evaluation[Myszkowski 1997]]

vs. referencevs. reference0.50.5 vs. vs.

[Myszkowski 1997]]

Termination Criterion

Physical Based Perceptual Metric[Ramasubramanian et

al1999]]• Threshold model defines a physical error metric

• Handles luminance-dependent and spatially dependent processing independently•Allowing pre-computation of

spatially-dependent component

Physical Based Perceptual Metric[Ramasubramanian et

al1999]]

Image Quality Metrics

Image Quality

• Compare and validate lighting simulations

• Use comparisons to guide rendering more efficiently•Compute less without altering

perception•Pixel by pixel comparison might be >

0, human might not see any difference

RMSE 9.5 RMSE 5.2

Pixel by Pixel ComparisonPrikryl, 1999

Visible Differences PredictorVDPVisible Differences PredictorVDP

Image

2

Image

1

Ps

ych

om

etr

icF

un

cti

on

Pro

bab

ility

Su

mm

atio

n

Vis

ua

lisa

tio

n o

fD

iffe

ren

ce

s

AmplitudeNonlinear.

AmplitudeNonlinear.

ContrastSensitivityFunction

ContrastSensitivityFunction

+

CortexTransform

CortexTransform

MaskingFunction

MaskingFunction

Unidirectionalor MutualMasking

[Daly ‘93, Myszkowski ‘98]

VDP: Results

Standard

Comparison

Pixel differences:Standard - Comparison

Pixel differences

The VDP response:probability ofperceivingthe differences

VDP response

Daly’s VDP: Features

•Predicts local differences between images •Takes into account important visual

characteristics:• Amplitude compression• Advanced CSF model• Masking

•Uses the cortex transform, which is a pyramid-style, invertible & computationally efficient image representation

Daly, 1993

Visible Discrimination Model

• Map of Just Noticeable Differences• Point sample function to model optics• Resample the image according to

foveal eccentricity• Band pass response

•Contrast pyramid•steerable filters

Lubin, 1997

Visible Discrimination Model

• Both images subjected to Identical processing• Distance measure

•Difference in responses for Each channel and summing Them to obtain a JND Map of the two images

input images

optics

sampling

contrastpyramid

transducer

masking

distance

Qnorm

JNDValue

Lubin, 1997

An Experimental Evaluation of Computer Graphics Imagery

• Comparing image to real-world scene

• An approach to image synthesis consisting of•A physical module•A perceptual module

[Meyer et al, 1986]]

An Experimental Evaluation of Computer Graphics Imagery

[Meyer et al, 1986]]

DifferenceSimulated Measured

An Experimental Evaluation of Computer Graphics Imagery

[Meyer et al, 1986]]

An Experimental Evaluation of Computer Graphics Imagery

[Meyer et al, 1986]]

Image Quality Metrics[Rushmeier et al, 1995]]

• Components of perceptually based metrics adapted from image compression•Gervais et al 1984•Mannos et al 1974•Daly 1993

Image Quality Metrics[Rushmeier et al, 1995]]

• Daly tested very well

Real Room Simulated Model of Room

Visual Psychophysics

• Determine the relationship between the physical world and human’s subjective experience of that world

• Measure the response (“psycho”) to a known stimulus (“physics”)

Why Lightness ?[Gilchrist 1977]]

[McNamara et al 1998, 2000]]

A Psychophysical Investigation

• Painted 5-sided cube

• Objects painted with different grey paints

• Complex illumination, with secondary reflections

Graphic Reconstructions[McNamara et al 1998, 2000]]

Experiment

Rendered

Real Scene

[McNamara et al 1998, 2000]]

Results[McNamara et al 1998, 2000]]

Tone Reproduction Operators

~105cd/m2~10-5

cd/m2

Tone Reproduction

~100 cd/m2~1 cd/m2

Same VisualResponse ?

Tone Reproduction for Realistic Images

• Mapping between radiances computed and light energy emitted from CRT

• Psychophysical model of brightness perception

• Observer model• Display model

Tumblin & Rushmeier, 1993

Tone Reproduction

Tone Reproduction for Realistic Images

http://graphics.cs.uni-sb.de/~slusallek/Doc/html/node14.html

Low Medium High

Tumblin & Rushmeier, 1993

A Contrast-based Scalefactor for Luminance Display

http://graphics.cs.uni-sb.de/~slusallek/Doc/html/node14.html

• Linear transformLd = mLW

• Matching contrast between real and image

Ward, 1994

A Contrast-based Scalefactor for Luminance Display

http://graphics.cs.uni-sb.de/~slusallek/Doc/html/node14.html

Min-Max Ward

Ward, 1994

A Model of Visual Adaptation for Realistic Image Synthesis

• Threshold visibility• Changes in colour appearance• Visual acuity• Temporal Sensitivity

Ferwerda et al, 1996

A Model of Visual Adaptation for Realistic Image Synthesis

Ferwerda et al, 1996

Spatially Nonuniform Scaling for High Contrast Images

• Incorrect to apply the same mapping to each pixel

• Spatial position

Chiu et al, 1993

Quantization Techniques for Visualization of High Dynamic

Range Pictures

• Similar to Chiu et al• Rational rather than logarithmic

• Accounts for the non-linearities of both the display device and human perception

• The biggest advantages is speed

Schlick, 1994

A Visibility Matching Tone Reproduction Operator for High

Dynamic Range Scenes

• Preserve visibility of objects• Histogram - adjusted to minimise

the visible contrast distortions• Also includes glare, colour

sensitivity, and acuity

Larson et al, 1997

A Visibility Matching Tone Reproduction Operator for High

Dynamic Range ScenesLarson et al, 1997

Perceptually Driven Radiosity[Gibson & Hubbold. 1997]]

• Steer computation to areas in need of most refinement

• A-priori estimate adaptation luminance•Tone-mapping to transform

luminance to display•Distance between two colors in

uniform colour space = numerical measure of perceived difference

Perceptually Driven Radiosity[Gibson & Hubbold. 1997]]

• Stop patch refinement once the difference between successive levels becomes perceptually unnoticeable

• Determine the perceived importance of any shadow

• Optimise the mesh for faster interactive display and minimise storage

Standard shadowtesting

(19.33 hours)

Perceptually-driven shadow testing

(3.10 hours)

[Gibson & Hubbold. 1997]]

Shadow Testing

Output meshOutput mesh Optimised meshOptimised mesh

[Hedley et al. 1997]]

Discontinuity Meshing

• Throw out discontinuities that are deemed visually unimportant

• Tone mapping• Compare colour differences along

the discontinuity line

Culled discontinuitiesCulled discontinuitiesOriginal sceneOriginal scene

Summary

• Applications of visual perception in computer graphics•Efficient software• Image quality evaluations•Tone reproduction operators

• Knowledge of HVS can be used to greatly benefit the synthesis of realistic images at various stages of production

Conclusion

• Great deal of potential• Perceptually accurate as well as

physically correct• Allow high level of confidence in

computer imagery allowing us to demonstrate to the world that our images are faithful representations !

Thank You

Ann McNamara

Ann.McNamara@tcd.ie

http://www.cs.tcd.ie/ann.mcnamara

Extra…

Spatial and Orientation Mechanisms

• The following filter banks are commonly used:

• Gabor functions (Marcelja80), • Steerable pyramid transform

(Simoncelli92), • Discrete Cosine Transform (DCT), • Difference of Gaussians (Laplacian)

pyramids (Burt83,Wilson91), • Cortex transform (Watson87, Daly93).

Cortex Transform: Organization of the Filter Bank

Cortex Transform: Orientation Orientation BandsBands

Input image Input image

Spatiovelocity CSF

• Contrast sensitivity data for traveling gratings of various spatial frequencies were derived in Kelly’s psychophysical experiments (1960).

• Daly (1998) extended Kelly’s model to account for target tracking by the eye movements.

log visual sensitivity

log velocity [deg/sec]

log spatial frequency [cycles/deg]

Temporal frequency [Hz]

Visual Masking•Masking is strongest between stimuli

located in the same perceptual channel, and many vision models are limited to this intra-channel masking.

•The following threshold elevation model is commonly used: