Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
09/13/2010 1
CPEN541 Human Interface Technologies
Instructor: Sidney Fels Term: W2020/2021
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Lots of new interface technologies • Survey of some of the dominant H.I.T. • Look at H.I.T. in context of driving trends • Spark creativity for new H.I.T. and applications • Integrate skills to develop new technologies
Why This Course?
2
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• User-centred design issues – introduction to HCI covers this (see CPEN441 for
example) • In depth analysis techniques for H.I.T. • Underlying physics of some H.I.T.
What I don’t cover
3
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• The communication of human experience is central to the future of computing
• Techniques needed for: – sensing, encoding, transmitting, storing, indexing,
retrieving, compressing, recognizing and synthesizing • Human body has many I/O channels • Integrate Cognitive, Physical and Emotional
aspects of interaction • Interface should disappear.
Course Basis
4
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
Course Outline
5
http://courses.ece.ubc.ca/518/
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
1 Paper review – submit on Canvas website
2 Lecture - group prepared – presentations done towards last half of course as video
lectures plus class discussion – groups formed on Canvas
Assignments
6
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Three phase project to design, prototype and evaluate a new HIT idea – Phase 1: specification
A. 1 page abstract B. deliver specification document
– Phase 2: design A. design outline B. deliver design document (Oct 22)
– Phase 3: implementation and initial user testing A. Experiment Demonstration B. Video C. technical report D. conference paper E. conference style oral presentation
• New interface technology – new use of sensing technology, display tech., new metaphor
• Project done in groups up to 3 people
Project
7
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Use human I/O capabilities as outline • Use examples from literature to illustrate H.I.T.
for given human I/O • Most H.I.T. reports are system oriented • A lot of reading...
Course Structure
8
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
1. Exposure to tip of H.I.T. iceberg 2. Prototype and Evaluate a novel Human-Machine
interface 3. Develop research oriented thinking about HIT 4. Paper review 5. Write conference/journal style paper 6. Present results in conference style
Course Goals
9
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Introduction to Human Information Processing – Input
• Visual channel • Auditory channel • Position and Motion Sensing Channel • Somatic Channel • Taste and Smell Channels
– Output • Intentional
– neuromuscular, movable, verbal • Non-intentional
– GSR, Heart Rate, Brain, Muscle, other
Introduction
10
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Decisions – Tracking – Memory – Learning – Indviduals vs. Groups
Introduction
11
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Driving Trends for Human Interface Tech. – Virtual Reality, Immersive Environments, AR – Ubiquitous computing/Intelligent Environments – Wearable Computing, Tangible Bits, – Games, Arts, Interactive Theatre, Interactive Art – WWW, Agents, Collaborative work
Introduction
12
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Senses electromagnetic radiation (wavelength = 0.3-0.7 microns)
• 2 eyes for binocular vision • 100,000 fixation points (100deg circular) • Types of eye movement (six muscles):
– compensatory (must have target) – pursuit (must have target) – Tremor, flick and drift – saccadic (jump from one fixation to another)
Visual Channel
13
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
An Eyeball
14
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
Field of View
15
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Rods and Cones – Fovea is all cones (6 million) – Periphery is mostly rods (125 million) – interleaved
• Rods activate neurons in groups – higher sensitivity less resolution
• Cones are more one-to-one – lower sensitivity more resolution
Visual Channel
16
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
Fovea and Periphery
17
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• 8Hz gives sensation of motion. • Familiarity helps interpret movement • Movement implies life • Movement links images (strongly)
Perceiving Motion
18
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Seeing depth: – Binocular vision:
• Disparity can be used to determine distance • Frontal plane horoptors (Helmholtz) • non-euclidean space
– Other cues: • overlap, relative size, relative height, atmospheric perspective,
texture gradients, parallel line convergence, motion parallax, accommodation and convergence
• Seeing size: – size constancy
Visual Channel
19
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Colour perception is very complicated - refer to readings
• Adapts to light conditions • lots of illusions to play with size and distance • other interesting things:
– retina is reflective – eye blink does not affect perception – pupil is normally black and circular – attention and gaze direction are correlated – people wear glasses and/or contacts
Visual Channel
20
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Senses mechanical vibration of air molecules – sound travels about 350m/s (1260km/hr)
• Human range is 16Hz to 20,000Hz • Pressure waveform causes hair cells to move
(23,500 cells) • Perceived loudness is approximately logarithmic • Perceived sound is highly dependent upon
environment – without reverberation unusual effects are noticed
Auditory Channel
21
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
Perceived Loudness
22
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
The Ear
23
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
The Ear (ctd.)
24
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Ears can localize sound • Least sensitive on median plane (due to
symmetry) • Perception of multiple pure tones complicated • Ears are well adapted for speech
Auditory Channel
25
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Inner ear has mechanisms for attitude • Body has proprioceptors
Position and Motion Sensing
26
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
Vestibular Mechanisms
27
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Sense of touch • sensations of:
– heat (temp), cold (temp), touch (pressure), pain (various)
• 7 distinctive receptors • one cold and one warm receptors
– more cold than warm – over 45 deg can activate some cold sensors – sensitive to changes in temperature
Somatic
28
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
When does Hot feel Cold?
29
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Tactual Sensing – rate is very important
• light touch quickly applied produces sensation – Hair acts as lever – same as proprioceptors – negative adaptation occurs
• high pass filter effect – 20 Hz is maximum for separability
• above 20Hz it is like audio signal
• Pain sensing – mechanical, chemical, thermal or electrical sensitive
Somatic
30
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• “Chemical” senses • taste buds for
– sensations of sour, salty, bitter and sweet – receptor issues unresolved – extremely complex and poorly understood
• Olfactory cells for: – different theories: chemical, infrared absorption, … – different perceptual mappings:
• small prism • four odours: fragrant, acrid, burnt and caprylic
– Acuity is great - 10,000 times more sensitive than taste
Taste and Smell
31
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
Taste Buds
32
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
How to classify smell?
33
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Usually combination of senses active • We also can sense:
– time (protensity) – probability – intensity
• Break-off phenomenon
Summary of Input Channels
34
01/01/2021 CPEN541 Copyright 2021, Sidney Fels 35
01/01/2021 CPEN541 Copyright 2021, Sidney Fels 36
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Motor control associated with cerebral cortex • volitional and non-volitional
– can see in facial expression • muscles contract when stimulated by nerves
Output: Neuromuscular
37
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
A Mapping of Brain Control
38
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Affordances – keyboards, touch pads, phone dials, etc.
• verbal control/non-verbal control • tongue movement • breath control • facial control • gait
Intended Output: Movable Controls
39
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
Speech Requires Control
40
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• GSR • EEG • EKG • EMG
Extracted Output:
41
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Significant constraint for learning and working effectively
• Magic number seven, Plus or Minus Two (Miller, 1956) – Chunking is key
• Capacity: 43 billion bits to 1.5 million bits? • Short term memory and long term memory
– different models for how memory is structured • Brain: 1012 neurons and 1015 connections
– connections change strength as we learn
Human Memory and Learning
42
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Human Cognitive Architecture (Modularity of Mind)
• Basic sensory, motor and cognitive performance characteristics
• Human sensori-motor abilities : good news & bad news
• Attention and Memory
• Models of Human Performance
Human Cognitive Performance
43
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Name the color of the text
• Respond as quickly as possible
• Measure response time
• 2 trials
Proof of automatic processing in brain
44
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Dog • Cat • Fish • Bird • Cow • Horse • Pig
Name the color of the text:
45
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Green
• Red
• Orange
• Red
• Blue
• Blue
• Orange
Name the color of the text:
46
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Subsystems may operate in parallel (theory):
• Input (perception): – Visual subsystem for what we see (most studied) – Acoustic subsystem for what we hear – Haptic subsystem for what we feel – Some subsystems are multimodal (e.g. speech understanding)
• Output (action): – Vocal (articulatory) subsystem for what we speak – Motor subsystem for how we move
Perception & action subsystems
47
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• In upcoming images, – Image will blink or flicker – Image changes with each blink – Raise your hand as soon as you identify change
• Images from O’Regan, Rensink & Clark 1999 (ron rensink of this dept)
Visual bottleneck demo
48
01/01/2021 CPEN541 Copyright 2021, Sidney Fels 49
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
Look for change between blinks
50
01/01/2021 CPEN541 Copyright 2021, Sidney Fels 51
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• The thing that changed wasn’t what we were attending to
• Change is very salient (the no-blink cases) but masked by another change at the same time (blink or flicker)
• Implications: – Mental image is an illusion – Mud-splash on car windshield – Looking away from display, refresh, “busy” interfaces – Same for other senses? Ongoing research…
How does this happen
52
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• This phenomenon (also known as “change blindness”) must be taken into account when designing an interface
• Change blindness often dealt with as a constraint – appearance and behaviour of elements of an interface
• Example: error notification when filling in an online form – Instead of suddenly appearing next to the wrong field, leaving the
rest of the page unchanged...
– Use a progress bar to show that the info is being validated and place the notification in a separate pop up window
Why does it matter in HCI?
53
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• This phenomenon (also known as “change blindness”) must be taken into account when designing an interface
• Change blindness is often dealt with as a constraint, influencing appearance and behavior of elements of an interface
• Example: error notification when filling in an online form
– Instead of suddenly appearing next to the wrong field, leaving the rest of the page unchanged...
– Use a progress bar to show that the info is being validated and place the notification in a separate pop up window
54
Close this window
x
Please enter a valid zip code for your region
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Limitation of our perception of the world sometimes can be used to overcome technological limitation
• In this case change Blindness allows walking in a Virtual Environment considerably larger than the surrounding physical space!
Change Blindness as an opportunity
55
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Seems like it: – Camera: keep steady, adjust focal lens length – Eye: focal point always moving, yet we perceive the world as
being sharp and in focus.
• But how does it really work? – Camera: film is exposed all at once by light from scene – Eye: Moves to explore a scene as information is needed
Vision system: like a camera?
56
Camera is good metaphor for optics of eye, but a poor metaphor for vision!
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Imagine creating a mental model of a room’s layout & furnishings by touching it when blindfolded or in the dark
• Model is built up serially (over time); process speeded if we start with a memory of what was in the room last time we were there,
• But if the memory is inaccurate or does not reflect current state, may take us longer to find the changes - because we believe in an incorrect model.
Vision is really more like touch:
57
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Vision considered dominant in UI design: this is changing rapidly.
• Audition & touch are critical in our non-HCI information-gathering & interaction with the real world
• Seen less in synthetic interactions because technology hasn’t caught up with our bodies.
• → Hot research area!
Other senses
58
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Multitude of input/output channels – all active at once
• I/O mechanisms usually depend upon – cognitive context – emotional contexts
• All these channels available to assist humans • H.I.T. is about finding ways to manipulate and/or
measure these channels for: – improved performance (cognitive, physical or
emotional) – entertainment and expression
Summary
59
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Tangible Bits • Wearable Computing • Ubiquitous Computing, Pervasive Computing,
Intelligent Environments, IoT • Art, Music and Entertainment • World Wide Web • VR/AR • Information Appliances (see Invisible Computer)
– Don Norman’s design of everyday things • Personal assistants/Personal Robots/self-driving cars
Driving Trends for H.I.T.
60
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
Is there a future for the Personal Home Computer?
61
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Imagine when computing is as cheap as paper • Computers will be everywhere, for every need
– transparent computing – transparent communication
• Mark Weiser (Xerox PARC) • PARC Tab, Pad, Boards + infrastructure • GUI based direction still
Ubiquitous Computing/Pervasive Computing
62
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Computing should be in background – end of personal computer – not just portable!
• Key concepts: – location
• context awareness – scale
• applications: – doors, preference forwarding, call forwarding, diaries,
daily informational assistance, prosthetics
UbiComp/Pervasive Comp
63
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
UbiComp/Pervasive Comp
64
R.V.A.R.
V.R.
F.G. attention
B.G. attention
Computer
Computer Physical
Physical
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Dependencies: – H/W
• displays • computers • network infrastructure
– S/W • OS to handle real-time + constantly changing resources • Network protocols
UbiComp/Pervasive Comp
65
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Advantages – computing where and when you need it – contextually aware devices – lower cognitive load
• Disadvantages – privacy – dependency – interaction limited to physical world metaphors
UbiComp/Pervasive Comp
66
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Hiroshi Ishii’s group at MIT and others • Idea: couple virtual world to real, physical objects
– Interactive Surfaces – Couple bits and atoms – ambient media
• Main Goals – grasp & manipulate foreground with physical objects – awareness of background using ambient media
• Dependent upon good metaphor – need to really do user-centred design
Tangible Bits
67
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Leverage affordances from real world – Bricks (Fitzmaurice, Ishii, Buxton) – Clearboard (Ishii & Kobayashi) – metaDESK, ambientRoom, and more – Marble Answering Machine (Bishop), Props (Hinkley),
Live Wire (Jeremijenko) • Can you think of richly afforded physical devices?
– Doors, windows, cars, toys, dishes...
Tangible Computing
68
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• People already know what manipulations make sense – as long as metaphor is maintained, life is good
• persistence of data • make abstract concrete • composition is natural • nice match of function, form and augmentation
Tangible Bits: Advantages
69
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Mismatched metaphor – makes task harder
• Limited to real world interactions • Complex interactions may be difficult to express
– looping contructs – boolean operations
• Mechanical failure of physical devices
Tangible Bits: Disadvantages
70
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Smaller and cheaper computers can be embedded in clothing and everyday devices – available all the time – can have first person perspective – augment person’s/devices ability
• MIT/U. of Toronto group including Steve Mann, Thad Starner and others on Wearables
Wearable Computing/IoT
71
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Wearable examples – video camera (glasses) – head’s up display (glasses) – compute device (shoes) – body monitoring devices – communication devices – tracking devices – audio devices – etc.
Wearable Computing
72
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Applications: – altered realities
• freeze frame, colour – augmented realities
• extra information such as people id tags • prosthetics: visual, audio, memory
– devices that know us and respond to us
• Social implications? – New protocols possibly needed – security
Wearable Computing/IoT
73
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Real-time, interactive graphics with 3D models + display technology that gives user immersion in the model world with direct manipulation
• Popular in late 80’s and early 90’s – is changing to interactive information visualization
• drove a lot of HIT – 3D graphics, trackers, gloves, HMDs and more
• Ivan Sutherland (1965), Jaron Lanier, Myron Krueger and lots more...
Virtual Reality/Environments
74
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Applications: – entertainment – vehicle simulation
• airplanes, cars, expensive machinery – physical data visualization
• planet surfaces • NMR data
– information visualization • chemical models • mathematical relationships
VR/VE
75
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Research problems – Visual displays
• field of view, resolution – Audition (speech and non-speech, input and output) – Haptics (forcefeed back and tactile feedback) – Tracking (still) – Emotion – Motion sickness – Software tools and models – Evaluation
VR/VE
76
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Depends on: – high speed computing – high speed rendering – low latency – good engineering design
VR/VE
77
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Cyberspace, Information Space – its own reality – mediates human-human interaction
• Must use H.I.T. to access this space – intelligent agents
• mobile, goal oriented, user context awareness – Computer application interfaces
• Enabling Technologies – browsers, GUIs, direct manipulation devices, – email agents, meeting scheduling agents – face recog. & synthesis, speech synth & recog
World Wide Web/Info. Spaces
78
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Music lead the push for many alt. Controllers – keyboards, wah-wah pedals, pitch benders – Therimin, Sackbut
• Artists often push boundaries of tech. to: – explore human emotion – concepts and philosophy – expression
• Video games drive H.I.T. • Education - web • Medicine - VR, tracking
Entertainment, Art, Music
79
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Technologies – video processing and integration – gesture sensing and recognition
• air guitar – wireless applications – robotics – image processing – high speed graphics – alternate controllers of all shapes and sizes
Entertainment, Art, Music
80
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Main driving trends – UbiComp/IoT
• Tangible interfaces – Information Spaces/WWW
• agents • better GUIs
– VR/AR • immersive experience
– Entertainment/Art/Music/Medicine • explore boundaries of expression
Summary
81
01/01/2021 CPEN541 Copyright 2021, Sidney Fels
• Look at examples of many technologies in context of use – H.I.T. is generally application oriented initially
• become generally accepted and new appl’s found
Summary
82