Upload
votruc
View
217
Download
0
Embed Size (px)
Citation preview
1
THE UNIVERSITYTHE UNIVERSITYOF BRITISH COLUMBIAOF BRITISH COLUMBIA
Intersensory Interactions and HumanComputer Interaction (HCI)
Brian Fisher
Fels: Design of Interactive Multimodal Graphics
Intersensory Interactions
• Intro and metacognitive gap
• Integrating cogsci theory with design
• Cognitive Architecture
– Modularity and multimodal interaction
• Information hiding-- conflict resolution
• Cognitive impenetrability
• Performance differences between modules
• Recalibration
– Spatial indexes in complex environments• Multimodal cue matching within modules
Fels: Design of Interactive Multimodal Graphics
Key Points for HCI Practice
• The Metacognitive Gap: The need for agrounded Cognitive Science approach
• Reflective design practice methods– Integrating CogSci with interaction design– Iterative design cycle
• Examples of extended HCI– Cognitive architecture: Multimodal displays and
how they are understood– Situated cognition: embodied interaction with
complex displays
Fels: Design of Interactive Multimodal Graphics
Introduction
History -- why we need to adapt HCImethods for multimodal interaction
2
Fels: Design of Interactive Multimodal Graphics
HCI History
• “Classic” HCI: User/Task/Tool Model– Task, Protocol & GOMS/keystroke analysis
• Command-line, menus, workplace systems
• Theoretical underpinnings– Cogsci of conscious thought
– Learning, Memory, Reasoning
– Sequences of operations
Fels: Design of Interactive Multimodal Graphics
Ergonomic HCI domain limits
TaskTool
User
What if I am doing this for fun?What if I want new insights?What if I want to communicate with someone?What if I am exploring a complex environment?
Fels: Design of Interactive Multimodal Graphics
Evolution towards multimodal displays
• WIMP interface: visual semantics– Metaphorical tool icons on desktop
– Direct manipulation
• Information visualization: Informationprocessing in the visual system– Visual analogs of information
– Spatial Instruments
Fels: Design of Interactive Multimodal Graphics
Iterative development cycle
Walkthrough or experiment
Implement prototype
DesignDesign TestTest
Spiral with increasing detail and test specificity
How? What?
3
Fels: Design of Interactive Multimodal Graphics
Future challenges
• Perception, cognition, and action in an immersive
multimodal environment populated with objects
events and actors
• Applications in entertainment, cognition,
communication
• Blend of virtual and real spaces… with seams
– Are the rules consistent?
– Can users shift between them?
– Can frames support rule shifts?Fels: Design of Interactive Multimodal Graphics
Opportunities for creative design
• Environments: Affordances for exploration– Spatial cognition, human space constancy theory
• Support for creative & logical thinking– Problem solving, embodied cognition models
• Media-based communication & collaboration– Metacognition, distributed cognition
• Experience (Kansei) engineering: Movingbeyond usability
Fels: Design of Interactive Multimodal Graphics
Effective Interface Design for RichSensory Environments
The interaction between display characteristics andthe information processing characteristics of the
user’s perceptual, motor, and cognitive processeswill largely determine interface performance
Fels: Design of Interactive Multimodal Graphics
Metacognitive Gap
• Intuitions about thoughts,goals and plans (“folkPsychology”) are reasonably accurate
• Intuitions about how people see, hear, andremember are very inaccurate
• Lack of awareness of the limits of intuition isthe “Metacognitive gap”
• Design-by-intuition leads to bad userexperience
4
Fels: Design of Interactive Multimodal Graphics
Evolving interaction design models
• Guidelines are (still) inadequate• User-centred design inadequate for
rich sensory environments• Need for theory-rich, evidence-based
approach: design for lower-levelprocesses
• Must integrate with higher-ordercognitive task analysis and user-centred design practices
Fels: Design of Interactive Multimodal Graphics
Bridging the theory/practice gap
• Convincing designers that there is something to
understand—the “metacognitive gap” of folk
Psychology
• Convincing Cognitive Scientists to answer relevant
questions—Complex data displays and multiple
tasks and the Psychophysics reductionist approach
• Integrating research and design—Finding a
common language
Fels: Design of Interactive Multimodal Graphics
Reflective HCI Practice (Schön)
Test: walkthrough, field study or experiment
Exp:“Lab sense”,FS: Observation
Ideas and hypotheses
Evaluation, Mapping
Implement
Interaction Design
Information foraging
(Online) Literature:PsychologyKinesiologySociologyAnthropologyArchitecture…
Fels: Design of Interactive Multimodal Graphics
Fields of interest: ExperimentalPsychology:
– Founded~ 100 years ago
– Areas of study• Psychophysics—Vision, hearing, tactile
senses
• Attention—Endogenous, exogenous,sustained
• Learning and memory
• Goal is often information processingalgorithms
• Discussed in Module 2
5
Fels: Design of Interactive Multimodal Graphics
Useful things from Psychology
• Perception and attention– Visual & auditory acuity & discrimination– Colour perception– Salient display changes, change blindness
• Learning and memory– Primacy, recency– Skill acquisition– STM limits– State-dependent learning
• Decision making– Base-rate neglect– End effects, biases
Fels: Design of Interactive Multimodal Graphics
Fields of interest: Kinesiology &related disciplines
• Science of human movement– Neuroscience
– Mechanics
– Anthropometry
• Examples: Fitts’ law, GOMS/Keystroke
• Goal is often perceptually guided behaviour
• Discussed in Module 3
Fels: Design of Interactive Multimodal Graphics
Fields of interest: Cognitive Science
• Cogsci Society founded 22 years ago• Combines Experimental Psychology,
AI, Philosophy and Neurophysiology• 3 levels of analysis
– Semantics: Intentions, Goals, andMeanings
– Syntax: Information processing– Implementation: Neural processing
• Goal is often Cognitive Architecture
Fels: Design of Interactive Multimodal Graphics
Other fields of interest
• Psycholinguistics
• Social cognition
• Cultural Psychology
• Communication theory
• Embodied communication,paralinguistics
• Anthropology
6
Fels: Design of Interactive Multimodal Graphics
Example of metacognitive gap
• The “computer metaphor” of mind
• Intuition: Single mental processor reads all
senses and performs a variety of tasks.
• Cognitive Science: Processing modules
operate in parallel.
– Restricted flow of information and control.
– Processing characteristics are counterintuitive
Fels: Design of Interactive Multimodal Graphics
“Horizontal” Modularity
• Model Human Processor (MHP)
• Serial stages of processing
• Information flow is Bottom-up
– Cognitive impenetrability
– “Seeing is believing”
Action
Cognition
Perception
Fels: Design of Interactive Multimodal Graphics
Horizontal modularity restrictions
Cognitive impenetrability (Pylyshyn,1984) refers to the inability of observersto use semantic information (such aswhat the person believes or intends to do)to influence the operation of the inputstage.
Fels: Design of Interactive Multimodal Graphics
Vertical modularity (Fodor)
Phoneme perception
Voice recognition
Auditory localization
Cognitive processing
7
Fels: Design of Interactive Multimodal Graphics
Vertical modularity restrictions
Information encapsulation (Fodor, 1983)refers to structural barriers within the
cognitive architecture that prevents internaldata stores from being shared between
modules in the same stage of processing.
Fels: Design of Interactive Multimodal Graphics
Events are processed by many channels
• Intuitions about thinking
– Fails at low levels
• Cognitive Architecture
– Multiple brain areas
– Interconnected
– Informationally encapsulated
– Multimodal inputs parsed from scene
Cognition
DorsalSystem
Bimodalspeech
Ventralsystem
…
Sensory world
Fels: Design of Interactive Multimodal Graphics
Modularity of reading
• Name the colour of the text
• Respond as quickly as possible
• Measure response time
• 2 trials
Fels: Design of Interactive Multimodal Graphics
DogDog
CatCat
FishFish
BirdBird
CowCow
HorseHorse
PigPig
CowCow
FishFish
8
Fels: Design of Interactive Multimodal Graphics
GreenGreenRedRed
OrangeOrangeRedRedBlueBlueBlueBlue
OrangeOrangeGreenGreenRedRed
Fels: Design of Interactive Multimodal Graphics
Cognitive Impenetrability & Modularity
• Stroop effect– Reading is data-driven module– Competition for response
• Other Modularity phenomena– Modularity of perception for action– Modularity of visual/auditory integration– Modularity of eye movement control– Modularity for Models of Minds
Fels: Design of Interactive Multimodal Graphics
Advantages of modular processing
• Download task to module, reducecognitive bottlenecks
• Fast, effortless information processing
• Near-optimal information integrationbetween cues and sensory modalities– Fuzzy logic cue integration
– Bayesian categorization
Fels: Design of Interactive Multimodal Graphics
"Civilization advances by extending thenumber of important operations whichwe can perform without thinkingabout them."
Alfred North Whitehead
What are the disadvantages?
9
Fels: Design of Interactive Multimodal Graphics
Disadvantage: Processing rigidity
• Stress within a module is not accessible• Complex stimuli may be processed
differently in different modules• Tasks may access different modules with
different performance characteristics• Virtual environments may introduce
discrepancies that impact different modules(and tasks) differently
Fels: Design of Interactive Multimodal Graphics
Stress within a module may beundetected
Poor cognitive access tolow-level processes
Action
Cognition
Perception
Fels: Design of Interactive Multimodal Graphics
Example: VDT stress syndrome
• Users complain of headache, visionproblems etc.
• Reports anecdotal, but readingimpairment is observed
• Also pupillary tremor and regressivesaccades
Fels: Design of Interactive Multimodal Graphics
Reflective HCI Practice (after Schön)
Test: walkthrough, field study or experiment
Exp:“Lab sense”,FS: Observation
Ideas and hypotheses
Evaluation, Mapping
Implement
Interaction Design
Information foraging
(Online) Literature:PsychologyKinesiologySociologyAnthropologyArchitecture…
10
Fels: Design of Interactive Multimodal Graphics
Reflective HCI Practice: Foraging
Test: walkthrough, field study or experiment
Exp:“Lab sense”,FS: Observation
Information foraging
(Online) Literature:PsychologyKinesiologySociologyAnthropologyArchitecture…
Fels: Design of Interactive Multimodal Graphics
Theory of space constancy in activevision
• Cognition needs the illusion of a stable world
• Eye movements should create confusingimage shifts
• Maintaining space constancy requires– Efference copy
– Passive blur
– Lateral masking
– Saccadic suppression
Fels: Design of Interactive Multimodal Graphics
Reflective HCI Practice: Hypothesizing
Test: walkthrough, field study or experiment
Exp:“Lab sense”,FS: Observation
Ideas and hypotheses
Evaluation, Mapping
Information foraging
(Online) Literature:PsychologyKinesiologySociologyAnthropologyArchitecture…
Fels: Design of Interactive Multimodal Graphics
VDT fatigue study
• Hypothesis “Sampling” raster duringsaccades reduces intra-saccadic blur, andmay overload saccadic suppression
• Space constancy perspective allowed us to:– Isolate the important factors in a complex
situation
– Find a more sensitive task and measure
• Study examines detection of movementduring saccade (w Bridgeman & Macnik)
11
Fels: Design of Interactive Multimodal Graphics
VDT fatigue study recommendations
• Suppression thresholds elevated forflickering stimuli
• Effect is reduced >250 Hz
• Problems will be greatest– large saccades
– high-contrast display
• Work arounds include blanking displayduring saccades
Fels: Design of Interactive Multimodal Graphics
Reflective HCI Practice: Testing
Test: walkthrough, field study or experiment
Exp:“Lab sense”,FS: Observation
Ideas and hypotheses
Evaluation, Mapping
Implement
Interaction Design
Information foraging
(Online) Literature:PsychologyKinesiologySociologyAnthropologyArchitecture…
Fels: Design of Interactive Multimodal Graphics
Disadvantage: Each module mustsolve feature assignment problem.
• Modules can’t accept information from othermodules: Information encapsulation
• Different modules should have access to adifferent set of matching cues.
• Illusory conjunctions can occur inmultimodal environments:– Phoneme perception: The McGurk effect
– Auditory localization: The ventriloquist effect
Fels: Design of Interactive Multimodal Graphics
Study: Impact of display errors onmultimodal perception
• Immersive environments typicallyhave display errors– Location of events is not precise– Timing is not precise– Graphics can be low-fidelity
• As immersive environments addsound and touch, what will be theimpact of these errors?
12
Fels: Design of Interactive Multimodal Graphics
Reflective HCI Practice:Foraging
Test: walkthrough, field study or experiment
Exp:“Lab sense”,FS: Observation
Information foraging
(Online) Literature:PsychologyKinesiologySociologyAnthropologyArchitecture…
Fels: Design of Interactive Multimodal Graphics
Recalibration by pairing (Epstein, 1975)
• Individual senses adapt to display• Sensory modalities calibrate each other:
haptics, vision, sound– Observed actions calibrate visual space (space
constancy)– Vision calibrates hearing for the location of a
multimodal event– Sound calibrates vision for the time of a
multimodal event
• Result is an after-effect: remapping ofauditory (visual, haptic) space
Fels: Design of Interactive Multimodal Graphics
Reflective HCI Practice: Hypothesizing
Test: walkthrough, field study or experiment
Exp:“Lab sense”,FS: Observation
Ideas and hypotheses
Evaluation, Mapping
Information foraging
(Online) Literature:PsychologyKinesiologySociologyAnthropologyArchitecture…
Fels: Design of Interactive Multimodal Graphics
Impact of information encapsulation
• Multimodal environment with errors intiming and location
• The same event might give rise to a singlemultimodal construct in one task, and twounimodal events for another.– Vary location of visual and auditory phonemes
in a simple teleconferencing-style video display
– Vary information carried by using syntheticspeech stimuli (5 levels).
13
Fels: Design of Interactive Multimodal Graphics
Ventriloquism meets the McGurkeffect.
• Vary location of visual and auditory phonemesin a simple teleconferencing-style video display
• Vary information carried by using syntheticspeech stimuli (5 levels).
• Subjects report sound location and syllableheard,
• Analyses included testing a variety ofmathematical models of information integrationby fitting free parameters with STEPIT.
Fels: Design of Interactive Multimodal Graphics
Use of mathematical modeling toolsallow us to address
• Sensory input from a number of channelssimultaneously
• How stimuli from multiple channels arematched and partitioned into mentalrepresentations
• How information from multiple senses isintegrated to give rise to trans-modalmental events
Fels: Design of Interactive Multimodal Graphics
Reflective HCI Practice: Testing
Test: walkthrough, field study or experiment
Exp:“Lab sense”,FS: Observation
Ideas and hypotheses
Evaluation, Mapping
Implement
Interaction Design
Information foraging
(Online) Literature:PsychologyKinesiologySociologyAnthropologyArchitecture…
Fels: Design of Interactive Multimodal Graphics
Results:
• Visual capture of auditory source location, resultingin a shifting of unimodal auditory locationestimation (ventriloquism after-effect).
• No effect of location difference on phonemeperception as measured by statistical or modelingtests.
• No correlation between errors in the two tasks (i.e.subjects could not selectively attend to the auditoryphoneme on trials when visual capture failed).
• Overall, modularity of phoneme perception issupported.
14
Fels: Design of Interactive Multimodal Graphics
Disadvantage: Changes in task interactwith modules to change performance
• 2 visual systems—“ventral stream” forrecognition and “dorsal stream” for action.
• Where vs how
• Different impact of illusions• Lesion data
Fels: Design of Interactive Multimodal Graphics
Functional Neuroanatomy ofperception for action.
2 visual systems—“ventral stream” for cognition and“dorsal stream” for motor performance.
Fels: Design of Interactive Multimodal Graphics
2 visual systems lesion evidence
identification, shaperecognition, objectorientation
object manipulation(orientation matching,grip scaling)
Posteriorparietal (RV)
object manipulation(orientation matching,grip scaling)
identification, shaperecognition, objectorientation
Ventrolateraloccipital (DF)
pointingdetection andidentification
V1 (blindsight)
spared abilitiesperformancedeficits
lesion
Fels: Design of Interactive Multimodal Graphics
2 visual system illusions
pointinginduced motion,location report
Moving or off-centre frame
pointingdetection ofdisplacement,location report
displacementduringsaccade
grip scalingsize reportTichner circles
spared abilitiesdeficitsstimuli
15
Fels: Design of Interactive Multimodal Graphics
Research with videoconferencingand abstract displays
• Targeting sound: cognitive better thanmotor– Subs aware of visual and auditory
locations, but point to visual
• Targeting vision with context: Lessfeedback is better– Pointing with no visual feedback better
– Lagged cursor better than unlagged
Fels: Design of Interactive Multimodal Graphics
Interpreting pointing studies
• Pointing studies counterintuitive, but
predicted by response characteristics
of neurons in dorsal/ventral to visual
and auditory stimuli
• More in “Graphics in the Large” panel,
Thursday 3:30
Fels: Design of Interactive Multimodal Graphics
2 visual system illusions
apparent location ofsound
pointingSound withdisplacedvisualdistractor
pointinginduced motion,location report
Moving or off-centre frame
pointingdetection ofdisplacement,location report
displacementduringsaccade
grip scalingsize reportTichner circles
spared abilitiesdeficitsstimuli
Fels: Design of Interactive Multimodal Graphics
Modularity Recap
• Displays can interact with active perceptionto cause hidden stress: VDT study
• Displays can impact matching cues formultimodal cue integration– Support or frustrate recalibration by pairing– Lead to discrepancies in number and
composition if stimuli in different modules• Displays can cause disagreement between
motor performance and cognitive measures• Sometimes removing information helps
16
Fels: Design of Interactive Multimodal Graphics
Extending to complex worlds
• Previous studies in simple worlds, with afew visual and auditory events
• Multimodal environments are complex– Virtual worlds
– Augmented reality
– Ubiquitous computing
• How are multiple multimodal events dealtwith in the cognitive architecture?
Fels: Design of Interactive Multimodal Graphics
Indexical cognition (Pylyshyn)
Fels: Design of Interactive Multimodal Graphics
“FINSTs... make thoughts true”
• Perception– “Hotlink” tokens– Drawn to salient events– Object-centred, “sticky”– Visual routines– Finite number ~ 4
• Cognition– Maintain object history– Implicit memory of object associations– Sparse cognitive representation– Just-in-time delivery of information– Atom of intentionality
Fels: Design of Interactive Multimodal Graphics
Mental representations of complex worlds
• Cognitive architecture perspective requiresthat links be established between lowerlevel perceptual qualities and cognitivesymbols—i.e. a pointer, called a FINST.
• FINSTing allows us to interact withperceptual objects and events without theneed for mental images per se.
• Symbolic representation + pointers makesdifferent predictions than intuitive picture-in-the-head
17
Fels: Design of Interactive Multimodal Graphics
Indexical cognition (Pylyshyn)
Fels: Design of Interactive Multimodal Graphics
Naïve view of FINSTs in Cognitive Arch
Phoneme perception
Voice recognition
Auditory localization
Cognitive processing
Action (motor space)
FINSTs
Fels: Design of Interactive Multimodal Graphics
Another view of FINSTs
Phoneme perception
Voice recognition
Auditory localization
Cognitive processing
Action (motor space)
FINSTs
Fels: Design of Interactive Multimodal Graphics
More about FINSTs
• FINSTs Link mind & perceptual world– Visual routines: (collinear, inside,
subitizing)– History of an object– Object-centred, “sticky”– Drawn to salient changes-- onsets,
luminance increments, oddballs– Finite number ~ 4-7– FINSTs + ANCHORs for motor behaviour
18
Fels: Design of Interactive Multimodal Graphics
More about ANCHORs
• ANCHORs link mind & action– Remembered locations for eye
movements
– Direct interaction with items off the retina
– Fast, robust motor performance by actionroutines
– Affordances for action
Fels: Design of Interactive Multimodal Graphics
Multiple object tracking demo
Fels: Design of Interactive Multimodal Graphics
Another trial (Scholl)
Fels: Design of Interactive Multimodal Graphics
Another trial (Scholl)
19
Fels: Design of Interactive Multimodal Graphics
Application
Fels: Design of Interactive Multimodal Graphics
Mental representations of complexenvironments
• Cognitive architecture perspective requires thatlinks be established between lower level perceptualqualities and cognitive symbols—i.e. a pointer,called a FINST.
• FINSTing allows us to interact with perceptualobjects and events without the need for mentalimages per se.
• Symbolic representation + pointers makes differentpredictions than intuitive picture-in-the-head
• Coping with spatial transformations in complexdata spaces
Fels: Design of Interactive Multimodal Graphics
Module advantages
• High-realism interface designs improve
performance by “downloading” information
processing operations to input modules.
• Interaction of display characteristics with
capabilities and characteristics of the
functional architecture will determine
performance.
Fels: Design of Interactive Multimodal Graphics
Module disadvantages
• Coordination– Distortions in location, timing, and category-relevant
information may lead to the formation of conflictingrepresentations in different modules.
• Processing inflexibility– Errors and conflicts within a module can create errors and
increase cognitive load. (CRT flicker example)
• Information hidingCognitive impenetrability of modules makes it difficult for
operators to determine the reasons for their poorperformance.
20
Fels: Design of Interactive Multimodal Graphics
Inseparability of Mind & World
• Embodied cognition-- mind/body
• Situated cognition-- mind/world
• Distributed cognition-- mind/mind
• Ecological theories (Vygotski, Luria,Bateson, Gibson) can be linked tosensory phenomena and informinteraction design
Fels: Design of Interactive Multimodal Graphics
Key Points for HCI Practice
• The Metacognitive Gap: The need for agrounded Cognitive Science approach
• Reflective design practice methods– Integrating CogSci with interaction design– Iterative design cycle
• Examples of extended HCI– Cognitive architecture: Multimodal displays and
how they are understood– Situated cognition: embodied interaction with
complex displays
Fels: Design of Interactive Multimodal Graphics
What to expect in the next talk
• More about perceptual and attentivebottlenecks
• Change blindness
• Three flavours of (in)attention
• Implications for visual displays
• Coercive graphics
Fels: Design of Interactive Multimodal Graphics
Environment/Inhabitant/Representation
• Information visualization
• Perceptual/deictic/situated Cognition
• Cognitive Architecture, perception, attention
• Display as extension of mental model
21
Fels: Design of Interactive Multimodal Graphics
Thirteen Ways of Looking at a BlackbirdWallace Stevens
I
Among twenty snowy mountains, The only moving thing Was the eye of a blackbird.
II I was of three minds, Like a tree In which there are three blackbirds.
III The blackbird whirled in the autumn
winds. It was a small part of the pantomime.
IV A man and a woman Are one. A man and a woman and a blackbird Are one.
V I do not know which to prefer, The beauty of inflections Or the beauty of innuendoes, The blackbird whistling Or just after.
VI Icicles filled the long window With barbaric glass. The shadow of the blackbird Crossed it, to and for. The mood Traced in the shadow An indecipherable cause.
VII O thin men of Haddam, Why do you imagine golden birds? Do you not see how the blackbird Walks around the feet Of the women about you?
VIII I know noble accents And lucid, inescapable rhythms; But I know, too, That the blackbird is involved In what I know.
IX When the blackbird flew out of sight, It marked the edge Of one of many circles.
X At the sight of blackbirds Flying in a green light, Even the bawds of euphony Would cry out sharply.
XI
He rode over Connecticut
In a glass coach.
Once, a fear pierced him,
In that he mistook
The shadow of his equipage
For blackbirds.
XII
The river is moving.
The blackbird must be flying.
XIII
It was evening all afternoon.
It was snowing
And it was going to snow.
The blackbird sat
In the cedar-limbs.