HUMAN COMPUTER INTERACTION CIS 6930/4930 INTERACTION
DEVICES
Slide 2
INTERACTION PERFORMANCE 60s vs. Today Performance Hz -> GHz
Memory k -> GB Storage k -> TB
Slide 3
INTERACTION PERFORMANCE 60s vs. Today Input punch cards ->
Keyboards, Pens, tablets, mobile phones, mice, cameras, web cams
Output 10 character/sec -> Megapixel displays, HD capture and
display, color laser, surround sound, force feedback, VR
Substantial bandwidth increase!
Slide 4
INTERACTION FUTURE Gestural input Two-handed input 3D/6D I/O
Glass (1:52) Glass Others: voice, wearable, whole body, eye
trackers, data gloves, haptics, force feedback Engineering
research! Entire companies created around one single technology
Magic Leap
http://virginradiodubai.blob.core.windows.net/images/2014/05/googleglass_660.jpg
Slide 5
INTERACTION CURRENT TRENDS Multimodal (using car navigation via
buttons or voice) Helps disabled (especially those with different
levels of disability)
Slide 6
KEYBOARD AND KEYPADS QWERTY keyboards been around for a long
time (1870s Christopher Sholes) Cons: Not easy to learn Pros:
Familiarity Stats: Beginners: 1 keystroke per sec Average office
worker: 5 keystrokes (50 wpm) Experts: 15 keystrokes per sec (150
wpm) Is it possible to do better?
Slide 7
KEYBOARD AND KEYPADS How important is: Accuracy Training
Keyboard properties that matter Size Adjustability Reduces RSI,
better performance and comfort Mobile phone keyboards, blackberry
devices, etc.
Slide 8
KEYBOARD AND KEYPADS How important is: Accuracy Training
Keyboard properties that matter Size Adjustability Reduces RSI,
better performance and comfort Mobile phone keyboards, blackberry
devices, etc.
Slide 9
KEYBOARD LAYOUTS QWERTY Frequently used pairs far apart Fewer
typewriter jams Electronic approaches dont jam.. why use it? DVOARK
(1920s) 150 wpm->200 wpm Reducing errors Takes about one week to
switch Stops most from trying
Slide 10
KEYBOARD LAYOUTS ABCDE style Easier for non-typists Studies
show no improvement vs. QWERTY Number pads Whats in the top row?
Look at phones (slight faster), then look at calculators, keypads
Those for disabled Split keyboards KeyBowls orbiTouch Eyetrackers,
mice Dasher - 2d motion with word prediction Dasher
Slide 11
KEYS Current keyboards have been extensively tested Size Shape
Required force Spacing Speed vs. error rates for majority of users
Distinctive click gives audio feedback Why membrane keyboards are
slow (Atari 400?) Environment hazards might necessitate Usually
speed is not a factor
Slide 12
KEYS GUIDELINES Special keys should be denoted State keys (such
as caps, etc.) should have easily noted states Special curves or
dots for home keys for touch typists Inverted T Cursor movement
keys are important (though cross is easier for novices)
Slide 13
KEYS GUIDELINES Auto-repeat feature Improves performance But
only if repeat is customizable (motor impaired, young, old) Two
thinking points: Why are home keys fastest to type? Why are certain
keys larger? (Enter, Shift, Space bar) Another example of Fitts
Law
Slide 14
KEYPADS FOR SMALL DEVICES PDAs, Cellphones, Game consoles Fold
out keyboards Virtual keyboard Cloth keyboards (ElekSen) Most lack
haptic feedback?
Slide 15
KEYPADS FOR SMALL DEVICES Mobile phones Combine static keys
with dynamic soft keys Multi-tap a key to get to a character Study:
Predictive techniques greatly improve performance Ex. LetterWise =
20 wpm vs 15 wpm multitap Draw keyboard on screen and tap w/ pen
Speed: 20 to 30 wpm (Sears 93) Swipe Handwriting recognition (still
hard) Subset: Graffiti2 (uses unistrokes)
Slide 16
POINTING DEVICES Direct manipulation needs some pointing device
Factors: Size of device Accuracy Dimensionality Interaction Tasks:
Select menu selection, from a list Position 1D, 2D, 3D (ex. paint)
Orientation Control orientation or provide direct 3D orientation
input Path Multiple poses are recorded ex. to draw a line Quantify
control widgets that affect variables Text move text
Slide 17
POINTING DEVICES Faster w/ less error than keyboard Two types
(Box 9.1) Direct control device is on the screen surface
(touchscreen, stylus) Indirect control mouse, trackball, joystick,
touchpad
Slide 18
DIRECT-CONTROL POINTING First device lightpenlightpen Point to
a place on screen and press a button Pros: Easy to understand and
use Very fast for some operations (e.g. drawing) Cons: Hand gets
tired fast! Hand and pen blocks view of screen Fragile
Slide 19
DIRECT-CONTROL POINTING Evolved into the touchscreen Pros: Very
robust, no moving parts Cons: Depending on app, accuracy could be
an issue 1600x1600 res with acoustic wave Must be careful about
software design for selection (land-on strategy). If you dont show
a cursor of where you are selecting, users get confused User
confidence is improved with a good lift-off strategy Now
combination Nintendo DS Samsung Note
Slide 20
DIRECT-CONTROL POINTING Primarily for novice users or large
user base Case study: Disney World Need to consider those who are:
disabled, illiterate, hard of hearing, errors in usage (two touch
points), etc.
Slide 21
INDIRECT-CONTROL POINTING Pros: Reduces hand-fatigue Reduces
obscuration problems Cons: Increases cognitive load Spatial ability
comes more into play
Slide 22
INDIRECT-CONTROL POINTING - MOUSE Pros: Familiarity Wide
availability Low cost Easy to use Accurate Cons: Time to grab mouse
Desk space Encumbrance (wire), dirt Long motions arent easy or
obvious (pick up and replace) Consider, weight, size, style, # of
buttons, force feedback
Slide 23
INDIRECT-CONTROL POINTING Trackball Pros: Small physical
footprint Good for kiosks Joystick Easy to use, lots of buttons
Good for tracking (guide or follow an on screen object) Does it map
well to your app? Touchpoint Pressure-sensitive nubbin on laptops
Keep fingers on the home position
Slide 24
INDIRECT-CONTROL POINTING Touchpad Laptop mouse device Lack of
moving parts, and low profile Accuracy potentially low for those
with motor disabilities Graphics Tablet Comfort Good for CAD,
artists Limited data entry
Slide 25
COMPARING POINTING DEVICES Direct pointing Study: Faster but
less accurate than indirect (Haller 84) Lots of studies confirm
mouse is best for most tasks for speed and accuracy Trackpoint <
Trackballs & Touchpads < Mouse Short distances cursor keys
are better (experts use keyboard for movement more) Disabled prefer
joysticks and trackballs If force application is a problem, then
touch sensitive is preferred Vision impaired have problems with
most pointing devices Use multimodal approach or customizable
cursors Read Vanderheiden 04 for a case study Designers should
smooth out trajectories Large targets reduce time and
frustration
Slide 26
EXAMPLE Five fastest places to click on for a right-handed
user?
Slide 27
EXAMPLE What affects time?
Slide 28
Fitts Law Recreation.5 1 2
Slide 29
FITTSS LAW Paul Fitts (1954) developed a model of human hand
movement Used to predict time to point at an object What are the
factors to determine the time to point to an object? D distance to
target W size of target Just from your own experience, is this
function linear? No, since if Target A is D distance and Target B
is 2D distance, it doesnt take twice as long What about target
size? Not linear there either T = a + b log 2 (D/W + 1)
http://www.lynda.com/Web-User-Experience-
tutorials/Understanding-Fittss-Law/103677/119792- 4.html
http://www.lynda.com/Web-User-Experience-
tutorials/Understanding-Fittss-Law/103677/119792- 4.html
Slide 30
FITTSS LAW T = a + b log 2 (D/W + 1) T = mean time a = time to
start/stop in seconds (empirically measured per device) b =
inherent speed of the device (empirically measured per device)
[time/bit or ms/bit] Ex. a = 300 ms, b = 200 ms/bit, D = 14 cm, W =
2 cm Ans: 300 + 200 log 2 (14/2 + 1) = 900 ms Question: If I wanted
to half the pointing time (on average), how much do I change the
size? Proven to provide good timings for most age groups Newer
versions taken into account Direction (we are faster horizontally
than vertically) Device weight Target shape Arm position (resting
or midair) 2D and 3D (Zhai 96)
Slide 31
EXAMPLES T = a + b log 2 (D/W + 1)
Slide 32
EXAMPLES T = a + b log 2 (D/W + 1)
Slide 33
EXAMPLES
Slide 34
FITTSS LAW T = a + b log 2 (D/W + 1) T = mean time a = time to
start/stop in seconds (empirically measured per device) b =
inherent speed of the device (empirically measured per device)
[time/bit or ms/bit] First part is device characteristics Second
part is target difficulty
Slide 35
VERY SUCCESSFULLY STUDIED Applies to Feet, eye gaze, head
mounted sights Many types of input devices Physical environments
(underwater!) User populations (even mentally handicapped and
drugged) Drag & Drop and Point & Click Limitations
Dimensionality Software accelerated pointer motion Training
Trajectory Tasks (Accot-Zhai Steering Law is a good predictor and
joins Fitts Law) Decision Making (Hicks Law)
Slide 36
VERY SUCCESSFULLY STUDIED Results (what does it say about)
Buttons and widget size? Edges? Popup vs. pull-down menus Pie vs.
Linear menus iPhone/web pages (real borders) vs. monitor+mouse
(virtual borders) Interesting readings:
http://particletree.com/features/vis ualizing-fittss-law/
http://particletree.com/features/vis ualizing-fittss-law/
http://www.asktog.com/columns/0 22DesignedToGiveFitts.html
http://www.asktog.com/columns/0 22DesignedToGiveFitts.html
http://www.yorku.ca/mack/GI92.ht ml
http://www.yorku.ca/mack/GI92.ht ml Using Fitts Law to slow people
down Using Fitts Law to slow people down
Slide 37
PRECISION POINTING MOVEMENT TIME Study: Sears and Shneiderman
91 Broke down task into gross and fine components for small targets
Precision Point Mean Time = a + b log 2 (D/W+1) + c log 2 (d/W) c
speed for short distance movement d minor distance Notice how the
overall time changes with a smaller target. Other factors Age (Pg.
369) Research: How can we design devices that produce smaller
constants for the predictive equation Two handed Zooming
Slide 38
AFFORDANCE Quality of an object, or an environment, that allows
an individual to perform an action. Gibson (77) perceived action
possibilities Norman The Design of Everyday Things
FOR YOUR PROJECT Look at the interface What will people assume
they can do with it? Write it down.
Slide 43
TRADEOFF FOR NEW INTERFACES Consider a military training
simulator How would you allow a user to user a gun in the
simulator? Engineered Device High Affordance High Cost Low
Reusability Standard Device Low Affordance Low Cost High
Reusability
Slide 44
NOVEL DEVICES Themes: Make device more diverse Users Task
Improve match between task and device Improve affordance Refine
input Feedback strategies Foot controls Already used in music where
hands might be busy Cars Foot mouse was twice as slow as hand mouse
Could specify modes Xk-a-75-r pedal switch
Slide 45
NOVEL DEVICES Eye-tracking Either worn by user or in the
environment (e.g. Tobii) Accuracy.4 degrees Selections are by
constant stare for 200-600 ms How do you distinguish w/ a selection
and a gaze? video games, studying user behavior (1:41), design
evaluation (pause 0:24) video gamesstudying user behaviordesign
evaluation Multiple degree of freedom devices Logitech Spaceball
and SpaceMouse Ascension Bird Polhemus Liberty and IsoTrack
Polhemus
Slide 46
NOVEL DEVICES Boom Chameleon Pros: Natural, good spatial
understanding Cons: limited applications, hard to interact (very
passive). Not in production Large simulators DataGlove Pinch glove
Gesture recognition American Sign Language Music Pros: Natural
Cons: Size, hygiene, accuracy, durability
Slide 47
NOVEL DEVICES Haptic Feedback Why is resistance useful?
SensAble Technologys Phantom, Novint Falcon Cons: limited
applications, computational complex (1 kHz update rate) Sound and
vibration can be a good approximation Rumble pack Two-Handed input
Different hands have different precision Myron Kruger novel user
participation in art (Lots of exhibit art at siggraph) Myron
Kruger
Slide 48
UBIQUITOUS COMPUTING AND TANGIBLE USER INTERFACES Interacting
with physical objects https://www.youtube.com/w atch?v=Rik8Z_TaxDw
https://www.youtube.com/w atch?v=Rik8Z_TaxDw Which sensors could
you use? Elderly, disabled Research: Smart House http://
www.linuxjournal.com/files/linuxjournal.com/linuxjournal/articles/030/3047/3047f2.png
Slide 49
NOVEL DEVICES Paper/Whiteboards Video capture of annotations
Record notes (special tracked pens Logitech digital pen) Handheld
Devices Smartphones/PDA Universal remote Help disabled Read LCD
screens Rooms in building Maps Interesting body-context-sensitive.
Ex. hold phone by ear = phone call answer.
Slide 50
NOVEL DEVICES Miscellaneous Shapetape reports 3D shape.
Shapetape Tracks limbs
Slide 51
SPEECH AND AUDITORY INTERFACES Theres the dream Theres the
dream Then theres reality Then theres reality Practical apps dont
really require freeform discussions with a computer Goals: Low
cognitive load Low error rates Smaller goals: Speech Store and
Forward (voice mail) Speech Generation Currently not too bad, low
cost, available
Slide 52
SPEECH AND AUDITORY INTERFACES Ray Kurzweil (87) first
commercial speech recognition software Bandwidth is much lower than
visual displays Ephemeral nature of speech (tone, etc.) Difficulty
in parsing/searching (Box 9.2)
http://www.kurzweiltech.com/raybio.html
Slide 53
SPEECH AND AUDITORY INTERFACES Types Discrete-word recognition
Continuous speech Voice information Speech generation Non-speech
auditory If you want to do research here, review research in: Audio
Audio psychology Digital signal processing
http://www.kurzweiltech.com/raybio.html
Slide 54
DISCRETE-WORD RECOGNITION Individual words spoken by a specific
person Command and control 90-98% for 100-10000 word vocabularies
Training Speaker speaks the vocabulary Speaker-independent Still
requires Low noise operating environment Microphones Vocabulary
choice Clear voice (language disabled are hampered, stressed)
Reduce most questions to very distinct answers (yes/no)
Slide 55
DISCRETE-WORD RECOGNITION Helps: Disabled Elderly Cognitive
challenged User is visually distracted Mobility or space
restrictions Apps: Telephone-based info Study: much slower for
cursor movement than mouse or keyboard (Christian 00) Study:
choosing actions (such as drawing actions) improved performance by
21% (Pausch 91) and word processing (Karl 93) However acoustic
memory requires high cognitive load (> than hand/eye) Toys are
successful (dolls, robots). Accuracy isnt as important Feedback is
difficult
Slide 56
CONTINUOUS SPEECH RECOGNITION Dictation Error rates and error
repair are still poor Higher cognitive load, could lower overall
quality Why is it hard? Recognize boundaries (normal speech blurs
them) Context sensitivity How to wreck a nice beach Much training
Specialized vocabularies (like medical or legal) Apps: Dictate
reports, notes, letters Communication skills practice (virtual
patient) Automatic retrieval/transcription of audio content (like
radio, CC) Security/user ID
Slide 57
VOICE INFORMATION SYSTEMS Use human voice as a source of info
Apps: Tourist info Museum audio tours Voice menus (Interactive
Voice Response IVR systems) Use speech recognition to also cut
through menus If menus are too long, users get frustrated Cheaper
than hiring 24 hr/day reps Voice mail systems Interface isnt the
best Get email in your car Also helps with non-tech savvy like the
elderly Potentially aides with Learning (engage more senses)
Cognitive load (hypothesize each sense has a limited bandwidth)
Think ER, or fighter jets
Slide 58
SPEECH GENERATION Play back speech (games) Combine text
(navigation systems) Careful evaluation! Speech isnt always great
Door is ajar now just a tone Supermarket scanners Often times a
simple tone is better Why? Speech involves cognitive load Competes
w/ human-human communication Thus cockpits and control rooms need
speech
Slide 59
SPEECH GENERATION Ex: Text-to-Speech (TTS) Latest TTS uses
multiple syllabi to make generated speech sound better Latest TTS
Robotic speech could be desirable to get attention All depends on
app Thus dont assume one way is the best, you should user test
Apps: TTS for blind, JAWS Web-based voice apps: VoiceXML and SALT
(tagged web pages). Good for disabled, and also for mobile devices
Use if Message is short Requires dynamic responses Events in time
Good when visual displays arent that useful. When? Bad lighting,
vibrations (say liftoff)
Slide 60
NON-SPEECH AUDITORY INTERFACE Audio tones that provide
information Major Research Area Sonification converting information
into audio Audiolization Auditory Interfaces Browsers produced a
click when you clicked on a link Increases confidence Can do tasks
without visual cognitive load Helps figure out when things are
wrong Greatly helps visually impaired
Slide 61
NON-SPEECH AUDITORY INTERFACE Terms: Auditory icons familiar
sounds (record real world sound and play it in your app) Earcons
new learned sounds (door ajar) Earcons Role in video games is huge
Emotions, Tension, set mood To create 3D sound Need to do more than
stereo Take into account Head- related transfer function (HRTF)HRTF
Ear and head shape New musical instruments Theremin New ways to
arrange music
Slide 62
DISPLAY TECHNOLOGY Monochrome displays (single color) Low cost
Greater intensity range (medical) Color Raster Scan CRT LCD thin,
bright Plasma very bright, thin LED large public displays
Electronic Ink new product w/ tiny capsules of negative black
particles and positive white Braille refreshable cells with dots
that rise up
Slide 63
WALL DISPLAYS 1024*768 = 786k, 1080p = 2m, large displays =
20m+ Visual images of gigapixels Informational Control rooms,
military, flight control rooms, emergency response Provides System
overview Increases situational awareness Effective team review
Interactive Require new interaction methods (freehand sketch,
handheld) Local and remote collaboration Art, engineering
Slide 64
LARGE DISPLAYS Multiple Desktop Displays Multiple CRTs or Flat
panels for large desktops Cheap Familiar Spatial divide up tasks
Comparison tasks are easier Too much info? Eventually -> Every
surface a pixel
Slide 65
MOBILE DEVICE DISPLAYS Personal Reprogrammable picture frames
Digital family portrait (GaTech) Medical Monitor patients Research:
Modality Translation Services (Trace Center University of
Wisconsin) As you move about it auto converts data, info, etc. for
you
Slide 66
MOBILE DEVICE DISPLAYS GUIDLINES Bergman 00, Weiss, 02 Industry
led research and design case studies (Lindholm 03) Typically short
in time usage (except handheld games) Optimize for repetitive tasks
(rank functions by frequency) Research: new ways to organize large
amounts of info on a small screen Study: Rapid Serial Visual
Presentation (RSVP) presents text at a constant speed (33%
improvement Oquist 03) Searching and web browsing still very poor
performance Promising: Hierarchical representation (show full
document and allow user to select where to zoom into)
Slide 67
3D PRINTING Create custom objects from 3D models Create custom
objects from 3D models Create physical models for Design review
Construction