Visual PerceptionVisual Perception

Rich Clarke

Q: Why should we care about humans in the loop?

A: Imagine building VR system...

• Asked to specify the ultimate, fully immersive visual display device.

Want the Virtual World to be utterly indistinguishable from reality• Want the Virtual World to be utterly indistinguishable from reality

• I.e. what are the hard limits on our perception of reality?

• What’s the best that we need to do?• What s the best that we need to do?

How to start to answer this question?

• Basic visual hardware: Visual anatomy & physiologyBasic visual hardware: Visual anatomy & physiology

• The fundamental unit of visual coding: Receptive fields

• Visual pathways: The split of information streamsp y p

• Brief aside: How do photoreceptors change EM radiation into neural activity?

• How well can you really see? Resolution and acuity

• Seeing the sun and the stars: Brightness adaption

• Colour perception

W ’ i t k i i d Th S Wh t? I li ti f C tWe’re going to keep in mind: The So What? Implications for Computer

Graphics?

The eye…Basic visual hardware: Visual anatomy & physiology

POSTERIOR: ~mesh of

ANTERIOR: Modelled as camera optics…

POSTERIOR: ~mesh of photoreceptors…

Photoreceptors in the retina: Rods & ConesBasic visual hardware: Visual anatomy & physiology

Pigmented layer

Start of the ‘brain’…

LIGHTTo the optic nerve

LIGHT

Basic visual hardware: Visual anatomy & physiology

Photoreceptors in the retina: Rods & Cones (2)

• Extremely sensitive to light• Provide achromatic visionProvide achromatic vision• Work at low level (scotopic)

illumination• Large receptive fieldsg p• Peak absorbance (sensitivity)

at ~500nm

• Less sensitive to lightLess sensitive to light• Provide colour vision• Work at high level (photopic)

illumination• Three types:

‘B’ peak at 437nm, ‘G’ peak at 533nm, ‘R’ peak: 564nm

Much smaller receptive fields

Basic visual hardware: Visual anatomy & physiology

SO WHAT?

U d t di f “h d ” i i i ht i t ki d f i f ti th t• Understanding of “hardware” gives insight into kinds of information that can be coded

• Real VR system: focus resource on right areasy g

The fundamental unit of visual coding: Receptive fields

A Cell’s Receptive Field‘On centre’ cells

A Cell s Receptive Field

• Defined by spatially localised group of photoreceptors serving some ganglion cell

‘Off centre’ cells

Higher up in the brain (in V1): integrate simple cells:

photoreceptors serving some ganglion cell

• Location and quality of stimulus to which the ganglion cell is responsive cellssimple cells:

complex cellsthe ganglion cell is responsive

• Opponency:On centre and off centre

Eye

On centre and off centreSpectral as well as spatial: Red/GreenYellow BlueAbsolute physical values lost:Absolute physical values lost:

Visual system encodes the “empirical significance” of the stimulus

Maybe it’s an artefact of the centre surround receptive fields of cells i e visual illusions arereceptive fields of cells – i.e. visual illusions are an unavoidable consequence of the mechanics of our visual hardwareof our visual hardware

Recall last slide...Recall last slide...

Same local statisticsSame local statistics…

The fundamental unit of visual coding: Receptive Fields

SO WHAT?

B i b ildi bl k f i l ti• Basic building blocks of visual perception• Information about absolutes (both brightness and ‘colour’) lost –

contrast & context sensitivity only (c.f. illusions)y y ( )• Brain ‘looks for’ fundamental structures that are/have been

behaviourally relevant in ontogenetic and phylogenetic history: fast ‘h d ’ iti‘hardware’ recognition

Visual pathways: The split of information streams

From the eye to the brain

LGN: 6 layers:y

Magnocellular layers – primary input from peripheral retina –non spectrally opponent ganglions, large receptive fieldsy g g g

Parvocellular layers – primary input from the foveal region –spectrally opponent cells, small receptive fields

Area V1 (Visual Cortex): more complex cells (I.e. with more complex receptive fields)Deals with What? And Where? …Separately?

Visual pathways: The split of information streams

SO WHAT?

Eyes may be serving 2 relatively separate visual systems:

• Fast response, achromatic system, motion sensitive, low res. (Magnocellular layers)

Eyes may be serving 2 relatively separate visual systems:

(Magnocellular layers)• Slow response trichromatic system, motion insensitive, high res.

(Parvocellular layers)

How do photoreceptors change EM radiation into neural activity?

Rods & Cones:

Outersegment: billions of light sensitive pigment moleculesOutersegment: billions of light sensitive pigment moleculesMolecules embedded in disks, stacked like pancakesRods: Pigment is Rhodopsin

γ

How well can you really see? Resolution and acuity

Physical limits on resolving power

Three things determine resolution: 1. Optical filtering, 2. receptor sampling, (and 3. receptive field organisation)

1. Real optical system: aberrations, diffraction at entry apertureResolution limited to 30arcsec

2. Photoreceptors sample retinal image -> neural image representation.Spacing well matched to optics (Sampling theory)

Visual acuity is a function of contrast sensitivity ~30sec

Vernier (hyper)acuity: Ability to localise position of objects – not a function of contrast sensitivity

Can detect misalignments of ~5secUnknown exactly how it’s done

How well can you really see? Resolution and acuity

SO WHAT?

• Obviously sets hard limit on how much detail required of aObviously sets hard limit on how much detail required of a display system

• VR systems not close to this for real-time display• Vernier acuity plays an important role in the visibility of aliasing

artefacts in digital images – simple analysis of the visual system would predict that some artefacts should not be seensystem would predict that some artefacts should not be seen (below the limit of supposed visual acuity)

Seeing the sun and the stars: Brightness adaption

Brightness adaption

How many orders of magnitude difference between the dimmest and the

g p

How many orders of magnitude difference between the dimmest and the brightest things we can see?

Seeing the sun and the stars: Brightness adaption

Brightness adaption (2)

Three mechanisms

g p ( )

Mechanical --- Pupil dilationPhotochemical --- Bleaching & regenerationNeural --- Changes in processingNeural --- Changes in processing

Ch i C t t S iti it P tt A it C l P tiChanges in: Contrast Sensitivity: Pattern Acuity: Colour Perception

Seeing the sun and the stars: Brightness adaption

The time course of adaptionp

Purkeinje break

Seeing the sun and the stars: Brightness adaption

SO WHAT?

• Most of the information or 'power' (in Fourier domain) of anMost of the information or power (in Fourier domain) of an image is in brightness contrast

• Using 3 adaption mechanisms, able to see effectively over a range of ~10 log units

• At different luminances contrast sensitivity, acuity, colour perception changes markedlyperception changes markedly

• Obvious implications for the design of a VR system (resource allocation etc.)

Colour perception

Colour perception

Colour perception

What is colour perception? How do we (efficiently) recreate it?p p ( y)

Relative stimulation of each cone type in your retina (RGB) in the context of some visual fieldvisual field

Different spectral distributions of light should be able to stimulate the photoreceptors identically:photoreceptors identically:

Spectral distributions

Receptor sensitivity SD a

SD bSD b

Distinct distributions that are perceived identically w.r.t some visual system -METAMERS

Colour Perception

SO WHAT?

• Sensible choice of some colour primaries should allow you toSensible choice of some colour primaries should allow you to re-create any visible colour simply (without recreating the whole C(λ) distribution

• Not quite as simple as that…but right primaries will produce a colour gamut that covers most visible colours

SummarySummary…• Understanding of “hardware”: insight into kinds of

information that can be codedR ti fi ld B i b ildi bl k f i l ti• Receptive fields: Basic building blocks of visual perception

• Resolution of human visual system sets limit on how much detail requiredmuch detail required

• At different luminances contrast sensitivity, acuity, colour perception changes markedlyperception changes markedly

• World is not seen “as it is”: Context important: need to understand how scenes perceivedunderstand how scenes perceived

• Colour: Metamers, colour primaries & limitations of colour gamutsg

Refs picture creditsRefs, picture creditsRef 1: [Ferwerder] Ferwerda J A (2001) Elements of Early Vision for ComputerRef 1: [Ferwerder] Ferwerda, J. A. (2001) Elements of Early Vision for Computer

Graphics, IEEE Computer Graphics and Applications, 21(5), pp. 22-33.

Ref 2: [Atkinson] R.C. Atkinson, ed., Steven’s Handbook of Experimental Psychology, 2nd ed John Wiley & Sons New York 19882nd ed., John Wiley & Sons, New York, 1988.

Ref 3: [Purves & Lotto] www.lottolab.org, also D. Purves & R. Beau Lotto, Why we see what we do: An Empirical Theory of Vision, Sinauer Associates, 2003

Ref 7: [Sekuler & Blake] 7. R. Sekuler and R. Blake, Perception, McGraw-Hill, New York, 1994.

Ref 15: [Spillman & Werner] L. Spillman and J.S. Werner, eds., Visual Perception: The Neurophysiological Foundations, Academic Press, San Diego, 1990.

Ref 28: [Bollin & Mayer] M R Bolin and G M Meyer “A Frequency Based Ray Tracer ”Ref 28: [Bollin & Mayer] M.R. Bolin and G.M. Meyer, A Frequency Based Ray Tracer, Proc. Siggraph 95, ACM Press, New York, 1995, pp. 409-418.