Embed Size (px)
Citation preview
1
Jennifer MankoffCoC & GVU Center
Georgia Tech
Programming Support for
Natural Interaction
2
Acknowledgements
Gregory Abowd & Scott Hudson FCE Group & GVU NSF & IBM
3
Natural Interaction
Mouse + Keyboard work great for killer apps of the GUI world
Problems Arise… Off the desktop (Ubiquitous computing) People with disabilities
My interest: Providing general support for interaction design in this space of problems
4
Outline of Talk
Output Augmenting physical objects and
spaces Input
Recognition in the face of errors Motivation and Design Space Contribution: Toolkit-level support Validation: 2 Toolkits, Problem areas
Low bandwidth, ambiguous input
5
Output: Augmenting physical objects and spaces PALplates
User study Exploration
Domisilica Dual augmentation General infrastructure
6
PALplates (CHI’97)
Based on study of paper prototype Point of need Dynamic UI
Identity Location
(FX-Pal)
7
Domisilica
Dual Augmentation Augment virtual world with
information about real world state Augment real world with virtual
services, info (recipes, notes) General Infrastructure Example app: Cyberfridge (CVE’98)
8
Contents: an Instant Jello Pudding:French Vanilla an Instant Jello Pudding:French Vanilla a 16oz Ferrara Capellini-Angel Hair a Hamburger Helper: Beef Noodle...
9
Outline of Talk
Output Augmenting physical objects and
spaces General support for exploring new
domains Input
10
Input
Recognition Motivation and Design Space Contribution: Toolkit-level support Validation: 2 Toolkits, Problem areas
Low Bandwidth, Ambiguous Input
11
Recognition
Recognition is becoming ubiquitous
Recognition is difficult to use A range of interface problems
result OOPS toolkit helps solve them
12
Research Method
Evaluators Study individual problems/solutions Design space of possible solutions
Builders Facilitate solutions to sets of problems
(Architectural / toolkit solutions) Design space exploration by a larger
community
13
Input
Recognition Motivation and Design Space Contribution: Toolkit-level support Validation: 2 Toolkits, Problem areas
Low Bandwidth, Ambiguous Input
14
Definitions Recognizer
interprets user input creates ambiguity
Error mistake from user’s perspective represented with ambiguity
Mediation dialogue between user and computer used for resolving ambiguity
15
SILK (Landay, 1996)
16
Burlap
17
Design Space of Mediation Techniques
Multimodal Suhm (98) McGee et al
(98) Handwriting
Huerst, Yang & Waibel (98)
Speech Ainsworth &
Pratt (92) Baber & Hone
(93) IUIs
Horvitz (99) Many Others…
18
Design Space of Mediation Techniques
Two major classes Repetition Choice
19
Design Space of Mediation Techniques
Two major classes Repetition Choice
20
Input
Motivation: Alternatives to Keyboard/Mouse
Recognition Motivation and Design Space Contribution: Toolkit-level support Validation: 2 Toolkits, Problem areas
Low Bandwidth, Ambiguous Input
21
OOPS Toolkit (CHI’00)
Toolkit-level support for handling ambiguity in recognition Library of mediators Architectural support
Based on subArctic
22
Architectural Support
INDEPENDENT of any specific toolkit
Separation of mediators, recognizers, and application
Communication by a common internal model (ambiguous hierarchical events)
Maintains ambiguity indefinitely
23
Related Work
GUI Garnet/interactors (Myers ‘90) X toolkits (Rosenberg et al. ’90)
GUI+Recognition SATIN (Hong et al ’00) Extended Amulet
(Landay & Myers ’93) PPSM (Hudson&Newell ’92) Artkit (Henry et al. ’90)
Context Context Toolkit
(Dey et al ’00) PARCPad, etc
(Schilit et al. ’94) Multimodal
OAA (Cohen et al ’94/Oviatt ’99)
PAC-Amodeus (Nigay& Coutaz ’95)
24
Three key pieces
Ambiguous hierarchical events Mediation subsystem Changes to event dispatch
25
down drag up• • • • • •
s
Ambiguous Hierarchical Events
strokestroke
down drag up• • • • • •
s c
Myers & Kosbie ‘96
26
Mediation Subsystem
Ambiguity is identified automatically Presence of multiple interactor nodes
Hierarchy is passed to mediators Recognizers, recipients informed of
accept/reject decisions Accept/reject modifies hierarchy
Application selects mediators from library
27
med1med2med3med4medn
rec1rec2rec3rec4recn
Event Dispatch
1. A sensed event arrives
eventInput
handler
2. It is dispatched to all recognizers3. It is mediated
•Problem: all recipients must support ambiguity
28
Modified Event Dispatch
1. A sensed event arrives 2a. It is dispatched to ambiguity
unaware components2b. It is dispatched to all recognizers3. It is mediated4. Any accepted leaf nodes are
dispatched to ambiguity unaware components
29
Comparison
stroke stroke
interactors
application
recognizers
application
Input handler
inter
recogs mediators
Input handler
meds
s c
s c
OOPS 1990’s GUI
30
Input
Recognition Motivation and Design Space Contribution: Toolkit-level support Validation: 2 Toolkits, Problem areas
Low Bandwidth, Ambiguous Input
31
Validation
Doing what’s been done Burlap (complex app) ~15 Mediators (based on design space)
Architecture: 2 Toolkits “direct” hookup in GUI world Context Toolkit extensions (Dey)
Making new things possible: 4 problem areas
32
Problem Areas (UIST ’00)
Errors & Ambiguity rejection errors target ambiguity
Mediation adding alternatives occlusion
33
Target Ambiguity
Problem: There may be multiple targets of a user action
Example: clicking Kabbash (95) Worden (97)
34
Target Ambiguity
Problem: There may be multiple targets of a user action
Example: Clicking Solution: Give the
user a choice of all of the targets
35
Target Ambiguity
Problem: There may be multiple targets of a user action
Example: Clicking Solution: Give the
user a choice of all of the targets
Analysis: Any interface
involving mouse press/release
Requires separation of concerns
Works with all interactors
36
Adding alternatives
Problem: The correct choice isn’t always present
37
Adding alternatives
Problem: The correct choice isn’t always present
Example: word-prediction
38
Adding alternatives
Problem: The correct choice isn’t always present
Example: word-prediction
Solution: Allow the user to add choices
39
Adding alternatives
Problem: The correct choice isn’t always present
Example: word-prediction
Solution: Allow the user to add choices
Analysis: Closely related
choices (e.g. URL prediction)
Requires extensible mediators
Benefits from recognizer API
40
Contributions (Recognition) Resolution of ambiguity in
recognition through mediation Architectural support
(2 toolkits) Exploration of domain of mediators
41
Future Work (Recognition)
Testing Implicit Input (Current) Intelligent Mediation
“Lazy” alternative generation Meta-intelligence
General support for varied inputs
42
Input
Recognition Motivation and Design Space Contribution: Toolkit-level support Validation: 2 Toolkits, Problem areas
Low Bandwidth, Ambiguous Input
43
Brain-computer interface (Current) Locked-in Syndrome Neural Mouse Control
44
Low-Bandwidth Input
No alternative modalities available Need for efficient error handling
Challenge: interpret brain signal in as rich a form as possible
45
Automatic Conversion for Low Bandwidth Input
Web Browser Example: Automatic URL
extraction Preview Area
Generalization –Next step for natural interaction
46
Conclusions Interaction design for ubiquity, disability
Dual augmentation Ambiguity in recognition Low bandwidth or error-prone input
Exploration of new interaction techniques Four mediation problems Low Bandwidth input
General, re-usable solutions Domisilica OOPS toolkit Extended Context Toolkit
48
What about Alternative Explosion?
Fairly independent, discrete interactions
Some stuff stays in recognizers Recognition API helps
Only cache ambiguous events Work to be done…
49
How general is “general”?
Two Toolkits Two complex applications Library of existing mediators Explorations of 4 problem areas
(UIST 00)
50
Is subArctic doing the work here?
No, our minimal requirements are common in today’s toolkits: An event-based toolkit An input-handling module that delivers
events to the appropriate places A library of interactors/widgets Access to source code (OOPS is not just
a library!)
51
Comparing SILK and Burlap
SI LK BURLAP
Modes Four One Combined
Only Draw Mode ThroughoutMediationSeparate Window I ntegrated
Editing Yes No
Storyboarding Yes No
52
Alternative Forms of Input
Cirrin (UIST 98)
53
Alternative Forms of Input
Cirrin (UIST 98) Cerebral Palsy
(Cursor Activity Recognition; Current)
54
Alternative Forms of Input
Cirrin (UIST 98) Cerebral Palsy
(Cursor Activity Recognition; Current)
Recognition
55
Interaction design for ambiguity: Supporting Creativity
Converting a the brain signal to music
Example
Two possible experiences Trying to play a piano with ones feet Having an artistic voice no one else can
reproduce
56
Library of mediators
Design space based on survey Generic and re-usable Three major classes
Repetition Choice Automatic
if (result is “dos”)reject it
elsedo nothing
57
58
Comparison
stroke stroke
interactors
application
recognizers
application
Input handler
inter
recogs mediators
Input handler
meds
s c
s c
OOPS 1990’s GUI
