IAT 355 Visual Analytics

Cognition and Tasks Lyn Bartram

Outline

•  Overview •  1. Role – How visualizations aid cognition? •  2. Tasks – What does the visualization assist?

Cognition | IAT 355

Basic Premise

•  Understanding (the cognitive aspects) is the crucial part of InfoVis

•  Visualization is simply a tool useful for aiding comprehension and understanding

•  Provides external cognition aids

Cognition | IAT 355

How Are Graphics Used?

•  What does a visualization or graphic image provide for us?

•  Exercise: tell me how to drive from here to the border assuming I don’t know place names or distances

•  How are you describing it?

Cognition | IAT 355

Memory and Cognition Executive Control

System

Articulatory Loop

Recycling items by articulatory processes Phonological Memory

Control and Decisions Reasoning

Language Comprehension Directing Attention

Transferring Information

Visual Spatial Sketchpad

Visual Memory Visual Spatial Memory

Working Memory Model (cf. Baddeley & Hitch) Cognition | IAT 355

How Are Graphics Used [Larke & Simon 87]?

•  Graphical visualization supports more efficient task performance by: •  Allowing substitution of rapid perceptual

influences for difficult logical inferences •  Reducing search for information required for

task completion

•  Sometimes text is better

Cognition | IAT 355

Visual Spatial Sketchpad

Visual Memory Visual Spatial Memory

Articulatory Loop

Recycling items by articulatory processes Phonological Memory

Understanding

•  People utilize an internal model that is generated based on what is observed •  Spatial mental models

•  Tversky calls the internal model a cognitive map •  Think about that term

•  What does “map” entail?

Cognition | IAT 355

Cognitive Map

•  Just don’t have one big one •  Have large number of these for all different kinds of

things •  For simple or easily learned layouts

•  Spatial models and spatial memory

•  When things are more complex •  Cognitive collage [Tversky 93]

•  thematic overlays of multimedia from different points of view •  Not as coherent as a single map

Cognition | IAT 355

Example

•  You’re taking the Sky Train train to get to SFU Surrey •  You have some existing internal model of the system, stops, how to

get there •  On train, you glance at map for help •  Refines your internal model, clarifying items and extending it •  Note that it’s still not perfect, no internal model ever is

•  You’re telling a friend how to get to SFU Surrey from Burnaby •  What instructions do you give?

Cognition | IAT 355

Example

•  You’re taking the Sky Train train to get to SFU Surrey •  You have some existing internal model of the system, stops, how to

get there •  On train, you glance at map for help •  Refines your internal model, clarifying items and extending it •  Note that it’s still not perfect, no internal model ever is

•  You’re telling a friend how to get to SFU Surrey from Burnaby •  Change at Columbia •  Direction and number of stops

Cognition | IAT 355

Question

•  Which direction do you drive to get from Windsor, Ontario to Detroit, Michigan?

Cognition | IAT 355

Example

•  Answer: NORTH

Cognition | IAT 355

Windsor/Detroit

•  If you answered West, you likely used this mental map: •  “Michigan is West of Ontario, thus Detroit is west of Windsor”

•  If you answered South, likely you reasoned •  “Ontario/Canada is North of Michigan/USA”

•  We reason relatively about location and distance •  Use cognitive reference points

Cognition | IAT 355

Spatial memory and reasoning

•  We use spatial reasoning to encode locations of objects in the world

•  Information visualization representations use •  Hierarchy •  Layout •  Relative relationships and connectivity

•  We build cognitive maps and collages of these constructs

Cognition | IAT 355

1. Process Models

•  (Recall the user and cognitive models from HCI?)

•  Process by which a person looks at a graphic and makes some use of it •  A number of substeps probably exist

•  Can you describe process?

Cognition | IAT 355

Don Norman’s Action Cycle

•  Two “Gulfs” to be bridged by cognitive activity

•  Gulf of Execution •  What do I do to change the

display? •  Gulf of Evaluation

•  How do I interpret the display?

Form Intention

Form Action plan

ExecuteAction

Evaluation

Interpretation

Perception

Change in World

Gulfofexecution

Gulfofevaluation

Goal

Cognition | IAT 355

Process Model 1

•  Spence •  Navigation - Creation and interpretation of an internal

mental model

Cognition | IAT 355

Navigation

Content

Browsing strategy

Internal model

Interpretation

Browse Model

Interpret

Formulate a browsing strategy

New view

Cognitive map

Cognition | IAT 355

Example

Figure 5.15 As the range of S4 is moved to higher values, the corresponding values of S3 move to lower values, indicating a trade-off

(a) (b) (c)

Cognition | IAT 355

DATA

PERCEPTION

INTERPRETATION

REPRESENTATION of data

PRESENTATION of the represented data

INTERACTION to select the required view of data

The scope of this book

HIGHER-ORDER COGNITIVE PROCESSES

Internal modellingStrategy formulation

Problem (re)formulationEvaluation of options

Decision making

etc.

Cognition | IAT 355

Process Model 2

•  Card, Mackinlay, Shneiderman •  Information Visualization: Using Vision to Think

•  Knowledge crystallization task •  Gather info for some purpose •  make sense of it by constructing a representational framework •  package it into a form for communication or action

Cognition | IAT 355

Knowledge Crystallization

•  Information foraging •  Search for schema (representation) •  Instantiate schema •  Problem solve to trade off features •  Search for a new schema that reduces problem to a simple trade-

off •  Package the patterns found in some output product

Cognition | IAT 355

Reorder Cluster Class Average Promote Detect pattern Abstract

Knowledge Crystallization

Instantiate schema

Search for schema

Forage for data

Problem-solve

Author, decide or act

Task

Extract Compose

Read fact Read comparison Read pattern Manipulate Create Delete

Instantiate

Overview Zoom Filter Details-on-demand Browse Search query

Cognition | IAT 355

How Vis Amplifies Cognition

•  Increasing memory and processing resources available •  Reducing search for information •  Enhancing the recognition of patterns •  Enabling perceptual inference operations •  Using perceptual attention mechanisms for monitoring •  Encoding info in a manipulable medium

Cognition | IAT 355

Process

task

Raw data

Data tables

Visual Structures Views

Data transformations

Visual mappings

View transformations

Cognition | IAT 355

User Tasks

•  What things will people want to accomplish using information visualizations? •  search vs. browsing

Cognition | IAT 355

Browsing vs. Search

•  Important difference in activities •  Appears that information visualization may have more

to offer to browsing

•  But…browsing is a softer, fuzzier activity •  So, how do we articulate utility?

•  Maybe describe when it’s useful •  When is browsing useful?

Cognition | IAT 355

Browsing

Useful when •  Good underlying structure so that items close to one another

can be inferred to be similar •  Users are unfamiliar with collection contents •  Users have limited understanding of how system is organized

and prefer less cognitively loaded method of exploration •  Users have difficulty verbalizing underlying information need •  Information is easier to recognize than describe

Lin ‘97

Cognition | IAT 355

Thought: what does this say about Vis

•  Maybe infovis isn’t about answering questions or solving problems

•  Maybe it’s about asking better questions

•  Hypothesis formulation •  Goal-directed browsing •  Fuzzy search

Cognition | IAT 355

Example from Last Time

Example courtesy of Chris North

Which state has the highest income? Is there a relationship between income and education? Are there any outliers? Questions:

Cognition | IAT 355

Exercise

•  What are the (types of) tasks being done here? •  Can you think of others?

•  Let’s develop a list

Cognition | IAT 355

Task Taxonomies

•  Number of different ones exist, important to understand what process they focus on •  Creating an artifact •  Human tasks •  Tasks using visualization system •  ...

Cognition | IAT 355

User Tasks

•  Amar & Stasko created a taxonomy of user tasks in visualization environments

•  10 basic actions •  Retrieve Value, Filter, Compute Derived Value, Find

Extremum, Sort, Determine Range, Characterize Distribution, Find Anomalies, Cluster, Correlate

Cognition | IAT 355

1. Retrieve Value

•  General Description: •  Given a set of specific cases, find attributes of those cases.

•  Examples: •  What is the mileage per gallon of the Audi TT? •  How long is the movie Gone with the Wind?

Cognition | IAT 355

2. Filter

•  General Description: •  Given some concrete conditions on attribute values, find data

cases satisfying those conditions.

•  Examples: •  What Kellogg's cereals have high fiber? •  What comedies have won awards? •  Which funds underperformed the S&P-500?

Cognition | IAT 355

3. Compute Derived Value

•  General Description: •  Given a set of data cases, compute an aggregate numeric

representation of those data cases.

•  Examples: •  What is the gross income of all stores combined? •  How many manufacturers of cars are there? •  What is the average calorie content of Post cereals?

Cognition | IAT 355

4. Find Extreme

•  General Description: •  Find data cases possessing an extreme value of an attribute

over its range within the data set. •  Examples:

•  What is the car with the highest MPG? •  What director/film has won the most awards? •  Which national hockey team has won the most Olympic Gold

medals since 1912?

Cognition | IAT 355

5. Sort

•  General Description: •  Given a set of data cases, rank them according to some ordinal

metric.

•  Examples: •  Order the cars by weight. •  Rank the cereals by calories.

Cognition | IAT 355

6. Determine Range

•  General Description: •  Given a set of data cases and an attribute of interest, find the

span of values within the set.

•  Examples: •  What is the range of film lengths? •  What is the range of car horsepowers? •  What actresses are in the data set?

Cognition | IAT 355

7. Characterize Distribution

•  General Description: •  Given a set of data cases and a quantitative attribute of

interest, characterize the distribution of that attribute values over the set.

•  Examples: •  What is the distribution of carbohydrates in cereals? •  What is the age distribution of shoppers?

Cognition | IAT 355

8. Find Anomalies

•  General Description: •  Identify any anomalies within a given set of data cases with

respect to a given relationship or expectation, e.g. statistical outliers.

•  Examples: •  Are there any cereals that have high calories but low sugar? •  Are there exceptions to the relationship between horsepower

and acceleration?

Cognition | IAT 355

9. Cluster

•  General Description: •  Given a set of data cases, find clusters of similar attribute

values. •  Note: you need to determine what “similar” is

•  Examples: •  Are there groups of cereals w/ similar fat/calories/sugar? •  Are all comedies the same length?

Cognition | IAT 355

10. Correlate

•  General Description: •  Given a set of data cases and two attributes, determine useful

relationships between the values of those attributes.

•  Examples: •  Is there a correlation between carbohydrates and fat? •  Is there a correlation between country of origin and MPG? •  Do different genders have a preferred payment method? •  Is there a trend of increasing film length over the years?

Cognition | IAT 355

Discussion: Compound Tasks

•  “Sort the cereal manufacturers by average fat content” •  Compute derived value; Sort

•  “Which actors have co-starred with Julia Roberts?” •  Filter; Retrieve value

•  “What’s the variation in cereal sugar content”? •  Compute derived value; determine range; characterise

distribution

Cognition | IAT 355

What was left out?

•  Basic math •  “Which cereal has more sugar, Cheerios or Special K?” •  “Compare the average MPG of American and Japanese cars.”

•  Uncertain criteria •  “Does cereal (X, Y, Z…) sound tasty?” •  “What are the characteristics of the most valued customers?”

•  Higher-level tasks •  “How do mutual funds get rated?” •  “Are there car aspects that Toyota has concentrated on?”

•  More qualitative comparison •  “How does the Toyota RAV4 compare to the Honda CRV?” •  “What other cereals are most similar to Trix?”

Cognition | IAT 355

Cognition Summary

•  Visualization Helps Cognition •  Aids the user by:

•  Helping Knowledge creation process •  Helping with knowledge seeking tasks

•  Models: •  Process models •  Task taxonomies

Cognition | IAT 355

User Tasks

•  Wehrend & Lewis (Vis ‘90) created a low-level, domain independent taxonomy of user tasks in visualization environments

•  Eleven basic actions •  identify, locate, distinguish, categorize, cluster, distribution,

rank, compare within relations, compare between relations, associate, correlate

Cognition | IAT 355

W&L 1

•  Locate •  Finding something that one knows about already

Cognition | IAT 355

W&L 2

•  Identify •  Describe an object not necessarily known previously

Cognition | IAT 355

W&L 3

•  Distinguish •  Detecting different values of same variable

Cognition | IAT 355

W&L 4

•  Categorize •  Define divisions that visual objects can be sorted by

Cognition | IAT 355

W&L 5

•  Cluster •  Determining whether data items are clustered or not

Cognition | IAT 355

W&L 6

•  Distribution •  Describe overall pattern of data

Cognition | IAT 355

W&L 7

•  Rank •  Finding best and worst, for example

Cognition | IAT 355

W&L 8

•  Compare within entities •  Decide something based on attributes of similar objects

Cognition | IAT 355

W&L 9

•  Compare between relations •  Different or sets of entities used as basis of comparison

Cognition | IAT 355

W&L 10

•  Associate •  Form relationship between objects on display

Cognition | IAT 355

W&L 11

•  Correlate •  Determine which objects share similar attributes

Cognition | IAT 355

Visual Task Taxonomy (Zhou & Feiner)

# Relational tasks Associate -- Collocate -- Connect -- Unite -- Attach Background Categorize -- MarkDistribute Cluster -- Outline -- Individualize Compare -- Differentiate -- Intersect

Correlate -- Plot -- MarkCompose Distinguish -- MarkDistribue -- Isolate Emphasize -- Focus -- Isolate -- Reinforce Generalize -- Merge Identify -- Name -- Portray -- Individualize -- Profile

Locate -- Position -- Situate -- Pinpoint -- Outline Rank -- Time Reveal -- Expose -- Itemize -- Specify -- Separate Switch

# Direct visual # organizing and # encoding tasks Encode -- Label -- Symbolize -- -- Quantify -- -- Iconify -- Portray -- Tabulate -- Plot -- Structure -- Trace -- Map

Cognition | IAT 355

Interpretation

•  The nested items are refinements of particular ways of achieving task

•  E.g., To locate an item, we might use the more specific visual task pinpoint

Locate -- Position -- Situate -- Pinpoint -- Outline

Cognition | IAT 355

Dimensions

•  Visual tasks have two main dimensions 1.  Visual accomplishments - describe presentation intents that

task might help to achieve 2.  Visual implications - particular type of visual action that visual

task may carry out

Cognition | IAT 355

1. Visual Accomplishments

•  All about presentation intent •  Classified into two categories:

•  Tasks that inform the user (e.g., make a presentation with ppt) •  Tasks that enable user to explore or compute (e.g., decide

which stock to buy)

•  Each of these can be broken down further

Cognition | IAT 355

Visual Accomplishments Inform Enable

Elaborate Summarize Explore Compute

Search Verify Sum Differentiate Emphasize Reveal

Associate Background Categorize Cluster Compare Correlate Distinguish Generalize Identify Locate Rank

Correlate Locate Rank

Correlate Locate Rank

Categorize Cluster Compare Correlate Distinguish Emphasize Identify Locate Rank Reveal

Categorize Compare Correlate Distinguish Identify Locate Rank Reveal

Cognition | IAT 355

2. Visual Implications

•  Categorize various visual tasks by whether they imply •  Certain types of visual organization •  Certain ways of visual signaling •  Certain paths of visual transformation

Cognition | IAT 355

Making InfoVis More Task-Focused

•  InfoVis •  Representational Primacy

•  Show the data truthfully in meaningful ways

•  Analytic Primacy •  Support user analysis and tasks

Amar & Stasko InfoVis ‘04

Cognition | IAT 355

User Tasks

•  Examples of “higher-level” tasks •  Complex decision making, especially under uncertainty •  Learning a domain •  Identifying the nature of trends •  Predicting the future •  …

Cognition | IAT 355

Closing the Gaps

Narrowing the gaps between representation and analysis

Cognition | IAT 355

Narrowing Methods

•  Knowledge Precepts (for design and evaluation) •  Worldview Gap

1.  Determine domain parameters 2.  Expose multivariate explanation 3.  Facilitate hypothesis testing

•  Rationale Gap 1.  Expose uncertainty 2.  Concretize relationships 3.  Expose cause and effect

Cognition | IAT 355

Visual Analytics

•  Formation of abstract visual metaphors in combination with human information discourse (interaction) that enables detection of the expected and discovery of the unexpected within massive, dynamically changing information spaces •  Intelligence analysis •  Bioinformatics •  Financial analysis

Wong & Thomas IEEE CG&A, ‘04

Cognition | IAT 355