Visual Perception and Cognition

UNC

Roadmap

• Visual Perception: how it works• Fun with Visual Perception: Party Tricks and

other Amusements• Impact on Visualization: What do we know

about visual perception that can inform our visualization practice?▫Effects due to visual mechanism that we can

use to our advantage▫General Guidelines to follow (Bertin, XXX, )▫Design Examples

UNC

Visual Perception

• What is visual perception?▫The process of knowing or being aware of

information through the eyes.▫The process of acquiring, interpreting,

selecting, and organizing sensory information.

UNC

Related Disciplines

• Psychophysics: Applying methods of physics to measuring human perceptual systems.▫ How fast must light flicker until we perceive

it as constant?▫What change in brightness can we perceive?

• Cognitive psychology: Understanding how people think. In this context, how it relates to perception.

UNCSlide adapted from Stone & ZellwegerVisual System

• Light path▫ Cornea, pupil, lens, retina▫ Optic nerve, brain

• Retinal cells▫ Rods and cones▫ Unevenly distributed

• Cones▫ Three “color receptors”▫ Concentrated in fovea

• Rods▫ Low-light receptor▫ Peripheral vision

From Gray’s Anatomy

UNC

Steven’s Power Laws

p < 1 : underestimatep > 1 : overestimate

Stevens, On the Theory of Scales of Measurement, Science 103:2684, 1946

UNC

Pictures vs. Eyes• Pictures

▫ Generally produced by purpose built cameras, or computer applications, or drawings.

▫ For instance a camera has good optics, focus, white balance exposure controls. It captures a large image at constant high quality resolution (spatial, luminance, hue).

• Eyes▫ Heuristics developed by evolution using inexpensive

biophysical hardware to keep operator alive.▫ Human visual system is different in that it has relatively poor

optics. It is constantly scanning, adjusting focus, white balance, exposure. It captures detail only in foveal area and very coarsely in the peripheral vision areas. We produce a 3D spatial reconstruction of our world based on our 2D snapshots.

UNCOne Simple Model of PerceptualProcessing

• Three stage process▫Parallel extraction of low-level properties of scene▫Pattern perception▫Sequential goal-directed processing

Stage 1

Stage 2

Stage 3 Early, parallel

detection of color, texture, shape, spatial attributes

Dividing visual field into regionsand simplepatterns

Holding objects in working memory bydemands of active attention

UNC

Stage 1 - Low-level, Parallel

• Neurons in eye & brain responsible for different kinds of information (orientation, color, texture, movement, etc.)

• Arrays of neurons work in parallel• Occurs “automatically”• Rapid• Information is transitory, briefly held in iconic

store• Bottom-up data-driven model of processing• Often called “pre-attentive” processing

UNC

Pattern Perception

• Slow serial processing• Involves working and long-term memory• A combination of bottom-up feature

processing and top-down attentional mechanisms

• Different visual systems for object recognition and visually guided motion

UNCStage 3 – Sequential Goal-Directed(Attentive)• Splits into subsystems for object recognition

and for interacting with environment• Increasing evidence supports independence

of systems for symbolic object manipulation and for locomotion & action

• First subsystem then interfaces to verbal linguistic portion of brain, second interfaces to motor systems that control muscle movements

UNC

Stage 3 – Attributes

• Top-down attention-driven model of processing

• Slow serial processing• Involves working and long term memory• A few objects are constructed from the

available patterns to provide answers to visual queries

UNC

Visual Working Memory

• different from verbal working memory• low capacity (3-5?)• locations egocentric• controlled by attention• time to change attention: 100 ms• time to get gist: 100 ms• not fed automatically to long term memory

UNC

Preattentive Processing• How does human visual system analyze images?

▫ Some things seem to be done preattentively, without the need for focused attention

▫Generally take less than 200-250 msecs (eye movements take 200 msecs)

▫ Seems to be done in parallel by low-level vision system▫An important contribution vision science makes to data

visualization is that a limited set of visual properties can be detected very rapidly and accurately by the low-level visual system

http://www.csc.ncsu.edu/faculty/healey/PP/index.html

UNC

How many 3’s

Watch this space and tell me how many 3’s you see…

(1 second)

UNC

How many 3’s1281768756138976546984506985604982826762980985845822450985645894509845098094358590910302099059595957725646750506789045678845789809821677654876364908560912949686

UNC

How many 3’sAnswer?

Ready to try again?

UNC

How many 3’s1281768756138976546984506985604982826762980985845822450985645894509845098094358590910302099059595957725646750506789045678845789809821677654876364908560912949686

UNC

How many 3’sWhat about now? Do you have the correct answer?

So, pop-outs (pre-attentive selection) helps us with this task.

What else could help us with this task?

Can you think of other types of tasks that pre-attentive recognition could help us with?

UNCFun with Visual Perception:Party Tricks and other Amusements• Our visual system because it is built from the

ground up, and has evolved over millions of years, contains many “artifacts”.

• These make for good party tricks• And they help inform us about ways in which

we can take advantage of the vision system’s mechanics to help with visual understanding (especially quick recognition).

UNC

UNC

UNC

Stroop Effect

UNC

Stroop Effect

• Read the words on the following slide as quickly as you can.

Ready?

UNC

OAKWATERFLOWERBOATHOUSETREESTONEPEANUTHORSERADIOFOLDERSTREETFLYQUAKESTORMGRASSZIP

UNC

Stroop Effect

• OK. Let’s do it again. Read the words on the next slide as quickly as you can.

Ready?

UNC

REDBLACKYELLOWGREENREDBLACKREDYELLOWGREENBLUEYELLOWBLACKBLUEREDGREENYELLOWBLACK

UNC

• Did you notice a difference?

• Interference from engaging higher level processing. Actually you can probably do again at similar speed if you focus just on written word (left brain) and no connotations (right brain).

• A Whole New Mind: Why Right-Brainers Will Rule the Future, Daniel Pink

UNC

Examples from Healey

UNC

Differing Amounts of “pop-out”

• Task: for the next two slides, you will see two squares. Your job is to see how quickly you can pick out the square containing the red circle.

UNC

UNC

UNC

• Ready for next one?

UNC

UNC

• Good Job .

• For the next slides your job is to pick out the two groupings. Each square contains two groups (which have common features). The two groups can be separated by a line splitting the square into two rectangles.

• Raise your hand when you believe you know the answer.

UNC

UNC

UNC

UNC

UNC

UNC

UNC

• So, you were likely best at this task when groups differed by both color and shape.

• Still not bad when differed by just one attribute (color or shape—which was easier because it a better pre-attentive cue?).

• Not so good when difference was mixture of color and shape (i.e. red squares and blue circles versus blue squares and red circles).

UNC

What you can’t see…

UNC

UNC

Best Resource Award!

• A great resource for looking at many of these visual effects AND learning about the science behind them is Christopher Healey’s (NCSU) webpage on Perception in Visualization.

• Let’s take a brief look. Be sure to try the java applets.

UNC

Other Good Resources

• Michael Bach website

UNC

Impact on Visualization

• We knows things from carefully controlled studies (not so many of these), and lots of practical empirical experience.

• First, an example of controlled study to evaluate which visual elements were better for gauging magnitude (how much of something)

UNCControlled Study: Position vs Area/Angle

Figure 3. Graphs from position-angle experiment.

UNC5 ways to compare 2 magnitudes

Estimate the relative magnitude of the pair of bars marked with •

W.S. Cleveland R. McGill, Graphical perception,JASA 39, pp.521-554, 1984

UNC

Results: Error in Judgements

UNC

How can we make use of this?

• Best way is position or adjacent bar chart• How can you do this if you want to compare

different values on same chart.

• ADD EXAMPLE of dynamic stacked bar charts

• Best practices with bar charts (we will cover later in charts module) http://www.infovis.net/printMag.php?lang=2&num=158

UNC

Potential Preattentive Features

lengthwidthsizecurvaturenumberterminatorsintersectionclosure

hueintensityflickerdirection of motionbinocular lustrestereoscopic depth3-D depth cueslighting direction

UNCExamples of pre-attentive tasks • Target detection

▫Is something there?• Boundary detection

− Can the elements be grouped?• Counting

− How many elements of a certain type are present?

UNC

Key Perceptual Properties

• Texture• 3D• Motion• Shape• Groupings

▫Spatial▫Multiresolution (zoom)

• Brightness/Luminance• Color

UNC

Texture

from: Jürgen Döllner

from: Colin Ware

UNCPre-Attentive Cues With Textures

• A visual texture represents that visual sensation that allows us to pre-attentively differentiate two adjacent, possibly structured parts in our visual field without eye movement▫visual textures include micro-structures, patterns, profiles,

etc.▫ to identify textures, an observations of about 160-200 ms is

sufficient (cognitively controlled processes require about 300-400 ms)

• Classification of textures is based on▫coarseness, contrast, directionality (orientation), scale, line-

likeness, regularity, roughness• Textures improve perception of position and orientation• Texture communicate information about the 3D structure

regardless of their coloringSlide adapted from Klaus Mueller/ Jürgen Döllner

UNCPre-Attentive Cues With Textures• Same surface with and without texture

• Textures that do not include information are to be avoided in visualization▫ recall Tufte’s aesthetic principle that irrelevant decoration (=

chart junk) should be avoided• Subtle textures for 3D visualizations, however, can be important

elements of visual designSlide adapted from Klaus Mueller/ Colin Ware

UNC

Texture Perception• Textons

▫ fundamental micro-structures in generic natural images ▫ -basic elements in pre-attentive visual perception

• Textons can be classified into three general categories:▫ 1. elongated blobs (line segments, rectangles, ellipses) with specific

properties such as hue, orientation, and width, at different level of scales▫ 2. terminators (end of line segments)▫ 3. crossings of line segments

• Julesz believes that only a difference in textons or in their density can be detected pre-attentively▫ - no positional information about neighboring textons is available without

focused attention▫ - pre-attentive processing occurs in parallel (fast!)▫ - focused attention occurs in serial (slower!)

• Example: Look at two objects on right▫ although the two objects look very different in isolation▫ they are actually the same texton

Objects

Their textons

GaborPrimitves

Slide adapted from Klaus Mueller/ Jürgen Döllner

UNCRelation to Symbol and Texture Design• When designing textures to indicate different regions of a visualization, make sure

that the textons are as different as possible• The same rules apply when designing symbol sets• Example: A tactical map may require the following symbols:

▫ aircraft targets▫ tank targets▫ building targets▫ infantry position targets

• Each of these target types can be classified as friendly or hostile• Targets exist whose presence is suspected but not confirmed (this uncertainty must

be encoded)• Set of symbols designed to represent different classes of objects

▫ - symbols should be as distinct as possible with respect to their pre-attentive processing

▫ - recall: military reconnaissance must occur FAST!Slide adapted from Klaus Mueller/ Jürgen Döllner

UNC

Textron use

• Textron on military type map• Textron for fluid/air flows

UNC

Information Display in 3D: Depth Cues

• 3D display should provide depth cues• Linear perspective:

▫more distant objects become smaller in the image can indicate focus, importance, or ordering

▫- elements of a uniform texture become smaller with distance can give shape cues

• Shadows:▫show the relative height of objects above a surface▫provide strong depth cues for objects in motion▫can be semi-realistic and still work as a depth cue

• Occlusion:▫- very powerful depth cue

Slide adapted from Klaus Mueller/ Colin Ware

UNC

Information Display in 3D: Depth Cues

• Shading:▫ - shape cues from shading (shape-from-shading)

diffuse + specular + shadows

shape from shading(hole vs. hill)

specular can reveal fine detail

assume single light source

having more than one lightsource can lead to confusionSlide adapted from Klaus Mueller/ Colin Ware

UNC

Information Display in 3D: Depth Cues

• Other depth cues:▫ depth of focus▫ motion parallax (structure from motion) --> how objects relate

under motion (see next slide for examples)▫ steroscopic depth (binocular displays)

• For fine-scale judgement, for example, threading a needle:▫ stereo is important, and shadows and occlusion

• For large-scale judgement▫ linear perspective, motion parallax, and perspective are important▫ stereo is not so important

• However, for information visualizationdisplays, one may exploit focus to emphasize importance,despite depth relationshipsSlide adapted from Klaus Mueller/ Jürgen

UNC

Motion• 3D Structure from Motion

▫ Simple example of 3D structure from motion animation▫ Second example with user control

• There is lots more to motion, but we won’t cover.

UNC

Shape, Symbol• Symbols should be rapidly perceived and

differentiated• Application for maps, military, etc.

UNC

Uses of Symbols

• See Bertin’s work.

(Generally we won’t cover this as separate section due to time constraints)

UNC

Multiscale Resolution

• Images at coarse scale can still be recognized. See Dali’s painting (different at different viewing distances).

• Shark Picture. (look closely). From artist Chris Jordan (select running the numbers II, scroll down to hammerhead shark).

UNC

Luminance/Brightness

• Luminance▫Measured amount of light coming from

some place• Brightness

▫Perceived amount of light coming from source

UNC

Human Perception of Color

UNCPerceptual Color Models

What is the gamut the human eye can see?

CIE standard based on experimental measures

UNCCIE updated:

CIELAB (print)CIELUV (display)

UNCColor Models

• RGB (based on technology model)• HSB (HVS) model (based on

perceptual color model)▫ Hue - what people think of color▫ Saturation - intensity, whiteness▫ Brightness (Value) - light/dark

UNC

Contrast

• Important for fg-bg colors to differ in brightness

Hello, here is some text. Can you read what it says?

Hello, here is some text. Can you read what it says?

Hello, here is some text. Can you read what it says?

Hello, here is some text. Can you read what it says?

Hello, here is some text. Can you read what it says?

Hello, here is some text. Can you read what it says?

UNC

Color Background Contrast

• Focus on the black circle for a few seconds, then switch to one of the white fields. For a moment, you will see a white circle (the complementary color to the black circle). This is due to the local change of the adaption level.

From: Colin Ware

• Another example (used in Florida Recount joke).

UNC

Successive Contrast• Surround matters, especially for

colors• Afterburn effect:

▫focus on colored panels▫then switch to white panel▫do this for saturated and

non-saturated backgrounds• Color contrast effects

▫use saturated colors sparingly, they may cause undesired effects

▫neutral borders can helpFrom Klaus Mueller

UNC

Chromatic Color is Irrelevant…

• To perceiving object shapes• To perceiving layout of objects in space• To perceiving how objects are moving• Therefore, to much of modern life

Many people go much of their life without realizing they are color blind.

UNC

Color is Critical…

• To help us break camouflage• To judge the condition of objects (food)• To determine material types• Extremely useful for coding information

UNC

Implications• Color perception is relative• We are sensitive to small differences- hence need sixteen

million colors when driving computer displays• Not sensitive to absolute values- hence we can only use

less than 10 or 12 colors for coding• Brad’s view

▫Can use successfully for labeling small number of categorical data (12 or less).

▫Can in certain constrained cases be utilized as continuous scale (isoluminance example). But must be careful here; often not used properly in this case.

UNC

Spatial Map labeling example

UNC

Nautical Maps

What are goals?

Contrast these two:What visualization choices were made?

UNCElectronic Chart Example

Color in two ways:1) Semantic

Labels: Blue vs Brown is used to indicate region type (water, land)

2) Within these two colors (interval) shades of blue to indicate depth of water and shades on brown to indicate structures on land.

UNC

Applications

• Color interfaces• Color coding/labeling (for nominal data)• Color sequences for ordinal data• Continuous Color for continuous data

• Call attention to specific data• Increase appeal, memorability• Increase number of dimensions for encoding

data

UNC

Color Categories

• Are there certain canonical colors?▫Post & Greene ‘86

had people namedifferent colors on amonitor

▫Pictured are oneswith > 75%commonality

Black version: Colin Ware, Color Version: Post & Greene ‘86

UNC

Gender Differences for Color?

• Number of colors men perceive vs women?▫No difference (millions)

• Number of colors used in describing objects? How fast one call recall name (label) of color?▫Women are superior (more, faster) than

men (Arthur, H., Johnson, G., & Young, A. (2007). Gender differences and color: Content and emotion of written descriptions. Social Behavior and Personality, 35(6), 827-834. doi:10.2224/sbp.2007.35.6.827).

UNC

Color for Categories

• Can different colors be used for categories of nominal variables?▫Yes▫Ware’s suggestion: 12 colors

red, green, yellow, blue, black, white, pink, cyan, gray, orange, brown, purple

UNCOrdinal data Principles

▫Order: ordered values should be represented by perceptually-ordered colors

▫Separation: significantly different levels should be represented by distinguishable colors

▫Many good choices can be derived from “strips” through perceptually uniform color spaces (grey scale, heated object scale, etc).

▫Luminance: good for showing form▫Many hues: useful for showing readable values

UNCInterval Sequences: Contour Lines and Color• Both can indicate regions

▫Contour by showing the boundaries▫Color to distinguish different objects/groups.

• Choice of spacing▫Regular intervals to enable interval comparison▫Specific values to highlight regions (sea level)▫Choice of spacing

• Can attempt to use Uniform Color Space

UNC

Color for Sequences (continuous)

Can you order these (low->hi)

UNC

Possible Color Sequences

Gray scale Single sequencepart spectral scale

Full spectral scale Single sequencesingle hue scale

Double-endedmultiple hue scale

UNC

Continuous Uses: Color Scales

• Examples of two window and levels of same data (lung, soft tissue). Missing are psuedo, hue Color scales. Applied to medical images (from Hemminger Radiology work).

UNC

Color for Continuous Variables▫Is done, but beware this is tricky, must be

careful▫For example see

Ware, C., & Beatty, J. (1988). Using color dimensions to display data dimensions. Human Factors, 30(2), 127-142.

UNCColor as Scale + Luminance• Isoluminance Color

Display (Hemminger). • Studies use of color vs

luminance for showing two dimensions of finding (anatomical structure versus cancer signal) on map.

• What’s obvious shortcoming with this visualization? (story)

UNC

• Here’s VisCheck on my Isolum image

UNC

Colorblind and Color Scales

• Vischeck by Bob Dougherty and Alex Wade. Allows you to simulate what a colorblind person would see when viewing your image or website.

• Daltonize (part of vischeck) will correct (suggest) alternative color scheme so colorblind person can appreciate detail in image (switch Red/Green use to other hues).

UNC

Choosing Color Scales for Interfaces

• Types of color scales▫Continuous/Interval Data use continuous

color scale (mono/divergent)▫Categorical Data use labeling/qualitative

color scale

UNC

Color Scales

• Color Brewer Tool developed by Cynthia Brewer for GIS maps, is very helpful for picking safe continuous/discrete scales, and the number of colors in scales. She avoids uses saturated colors, and developed diverging color scales. Safe means OK for▫Colorblind folks ▫print on different devices▫photocopying

UNC

Color Schemes

• Color Scheme Designer by Petr Stanicek. Allows you to easily visualize and modify/compare color scheme choices for several common (designer recommended) color palettes for interfaces▫Mono▫Complement▫Triad▫Analogic▫Accented analogic

UNC

• Generally don’t use complementary colors (disturbing to human visual system, hard to look at for a long time). Example: RG protein (microarray type colormaps, use high contrast (bad) and direct complementary colors (red/green) bad. Also use Blue/Yellow, but with similar problem. Theresa Maria Rhyne designed an alternative one. Solution was green/purple, which results in reasonable color blindness results for all three color blindness conditions (different but OK).

• If you want contrast, use nearly complementary, or accented analogic.

UNC

• Tough to get 10 different distinguishable color labeling palette, that are equally easily distinguished. This is why Brewer tool and others don’t go above this for categorical data. Some folks try to extend by varying something else (like value/luminance) but this complicates by suggesting categories which are not present, so probably not appropriate to the task.

• A field guide to digital color by Maureen Stone. Great coverage, especially lots of information on color calibration for digital displays.

• The interaction of Color: Josef Albers

UNC

Color Themes

• Adobe’s Kuler tool. Social media tool allowing you to save and share your color theme schemes, and to search and see others as well. ▫Saves to your personal library (mykuler)▫Explores/searches yours/worlds themes▫Create theme from colormap or from your

image

UNC

Color Scheme Takeaways

• Consider CUT-DDV framework when choosing color scheme. Make conscious choice of type, number of levels, color, transparency, etc.

• Use freely available tools to help design and test colors for your visualization; try several alternatives schemes

• Check it for safe use

UNC

Bivariate Maps• Rows and columns: to preserve

univariate information, display parameters should should be perceptually orthogonal (ideally)

• Display map data with more than one variable per map location is challenging.

• First do we have to use color, or contours to indicate regions (states, counties)?

• What visual types do we have left to show the variables present at each map location?

UNCHue Saturation Bivariate Map

Tufte ‘83, pg. 153.

UNC

• Do we think this works? • How easy to perceive separate channels

of information?▫Population density▫Change in population

UNC

UNC

• Absolute number of disability, does this inform? If number is proportional to population is it interesting?

• How well does the separate symbol (wheelchair) which varies in size to show quantity, work?

• How well does the combination of this symbol (wheelchair) work in conjunction with quantitative variable for number of people with disability?

UNC

UNC

• How effective is the color for showing distance to the Mississippi river?

• Could we leave it out? Show river with more emphasis and depend on map information to visually show correlation to location of river?

UNC

Consider Data

• Nominal, ordinal, interval or ratio• Map or entity based coding• Amount of detail• Rate of change, i.e. is high spatial frequency

present in a variable? If so may require different choice (think West Nile Virus example with widely varying rates all across the US).

UNCConsider Audience (User)

• Color deficient viewers?▫Don’t depend on red-green differentiation▫Use redundant scales

• Application area conventions?▫Use familiar scales (or at least know when you’re not)

• Color associations with variables?▫Use associated color

• Color associations with data ranges?▫Use red for bad range (in U.S.)▫Use red for hot

UNC

Take home messages

• Use luminance for detail, shape and form• Use color for coding - few colors• Minimize contrast effects• Strong colors for small areas - contrast in

luminance with background• Subtle colors can be used to segment large

areas

UNC

More messages

• Color excellent for multi-dimensional data• Use additional tools to get quantities• Beware of artifacts due to color re-sampling• We need precise color for specification and

standardization

UNC

UNC

• What’s the best way to visualize this information? • Would map-based be better?• Can I easily pick out individual curves by color?

(more than 10-12) • How about height/chart figure for values?• What about a data table?• What are the questions we’re trying to answer?

UNC

Influence on Visualization

• Why we care▫Exploit strengths, avoid weaknesses▫Optimize, not interfere

• Design criteria▫Effectiveness▫Expressiveness▫No false messages

UNC

Design criteria: Effectiveness

• Faster to interpret• More distinctions• Fewer errors

0 1 2 3 4 5 6 7

This?

Or this?

UNC

Sensory vs. Arbitrary Symbols

• Sensory:▫Understanding without training▫Resistance to instructional bias▫Sensory immediacy

Hard-wired and fast▫Cross-cultural Validity

• Arbitrary▫Hard to learn▫Easy to forget▫Embedded in culture and applications

UNC

Which Properties are Appropriate for Which Information Types?

Final Take Aways

UNC

Channel Properties

Luminance Channel Chromatic Channels• Detail• Form• Shading• Motion • Stereo

• Surfaces of things• Labels• Categories,Berlin and

Kay (about 6-10)• Red, green, yellow and

blue are special (unique hues)

UNCRankings: Encoding quantitative data

Cleveland & McGill 1984, adapted from Spence 2006

UNC

Visual Property vs Data Type

Stephen Few’s Table:

Attribute Quantitative Qualitative

Line length

2-D position

Orientation

Line width

Size

Shape

Curvature

Added marks

Enclosure

Hue

Intensity

UNCInterpretations of Visual Properties• Some properties can be discriminated more accurately but

don’t have intrinsic meaning. (Senay & Ingatious 97, Kosslyn, others)

• If there are intrinsic meanings use them (positive advantage and avoid confusion). And be aware when you use them that you are not sending conflicting messages.

▫ Density (Greyscale)Darker -> More

▫ Size / Length / AreaLarger -> More

▫ PositionLeftmost -> first, Topmost -> first (culture/language specific)

▫ HueCulture specific meanings

▫ Slopeno intrinsic meaning ???