Mobile Devices for ControlMobile Devices for Control

Brad A. Myersbam@cs.cmu.edu

http://www.cs.cmu.edu/~pebbles

Brad A. Myersbam@cs.cmu.edu

http://www.cs.cmu.edu/~pebbles

Human Computer Interaction Institute Human Computer Interaction Institute

School of Computer ScienceSchool of Computer Science

Carnegie Mellon UniversityCarnegie Mellon University

Brad Myers 2

Mobile Devices

Mobile Devices for communication As a “Personal Information Manager” As a “Personal Digital Assistant” As a PC replacement

“PocketPC”

For Remote Control of other devices Remote Control of PCs Remote Control of appliances

Mobile Devices for communication As a “Personal Information Manager” As a “Personal Digital Assistant” As a PC replacement

“PocketPC”

For Remote Control of other devices Remote Control of PCs Remote Control of appliances

Brad Myers 3

Problem

Appliances are too complex Appliances are too complex

Brad Myers 4

Problem

Too many remotes Too many remotes

Brad Myers 5

Problem

User Interfaces of Mobile Devices themselves are too hard to use

User Interfaces of Mobile Devices themselves are too hard to use

Brad Myers 6

Problem

PC’s have only a keyboard and mouse as input devices

PC’s have only a keyboard and mouse as input devices

Brad Myers 7

Approach

Use a personal Mobile Device as an Interface to the PC and to the Appliances

Use a personal Mobile Device as an Interface to the PC and to the Appliances

Brad Myers 8

History

ParcTab – continuous communication

Early “organizers” – no communication

Newton – extra cost for synchronization

Palm – HotSync once a day

WindowsCE/PocketPC –ActiveSync once a day

ParcTab – continuous communication

Early “organizers” – no communication

Newton – extra cost for synchronization

Palm – HotSync once a day

WindowsCE/PocketPC –ActiveSync once a day

Brad Myers 9

Now, Handhelds will be Communicating

802.11 (“Wi-Fi”)

BlueTooth

Cell-phone network (G3)

(Infrared)

(Wires or cradle)

802.11 (“Wi-Fi”)

BlueTooth

Cell-phone network (G3)

(Infrared)

(Wires or cradle)

                                                            

Toshiba e740

HP iPaq 3870

Microsoft SmartPhone

Brad Myers 10

Premises of our Research

“With the coming wireless technologies, connecting the PCs and PDAs together will no longer be an occasional event

for synchronization. Instead, the devices will frequently be in

close, interactive communication.”

“With the coming wireless technologies, connecting the PCs and PDAs together will no longer be an occasional event

for synchronization. Instead, the devices will frequently be in

close, interactive communication.”— Brad Myers, “Using Hand-Held Devices andBrad Myers, “Using Hand-Held Devices and

PCs Together,” PCs Together,” Comm. ACMComm. ACM,,Vol. 44, No. 11. Nov., 2001. pp. 34 - 41.Vol. 44, No. 11. Nov., 2001. pp. 34 - 41.

Brad Myers 11

Pebbles Project

Performed as part of the Pebbles Project Overall goal: Use of multiple devices at the

same time Palm and desktop PC Mobile Phone and “Smart Home” Multiple handhelds in a meeting Pocket PC and appliances

Multiple users with their devices Single user with multiple devices

Performed as part of the Pebbles Project Overall goal: Use of multiple devices at the

same time Palm and desktop PC Mobile Phone and “Smart Home” Multiple handhelds in a meeting Pocket PC and appliances

Multiple users with their devices Single user with multiple devices

Brad Myers 12

Pebbles is:

PEBBLES

PEBBLES

DAs forDAs forntry ofntry ofothothytes andytes andocations fromocations fromxternalxternalources.ources.

DAs forDAs forntry ofntry ofothothytes andytes andocations fromocations fromxternalxternalources.ources.

http://www.pebbles.hcii.edu/http://www.pebbles.hcii.edu/

Controlling a PCControlling a PC Remote Commander Slide Show Commander “Semantic Snarfing” Scrolling Shortcutter

Remote Commander Slide Show Commander “Semantic Snarfing” Scrolling Shortcutter

http://www.pebbles.hcii.edu/http://www.pebbles.hcii.edu/

Brad Myers 14

Important Work by Others PARCTab [Want 95], [Weiser, CACM’93] Abowd: eClass using Handhelds [CHI’98] Greenberg: public and private data

[Personal Technologies, 3(1), 1999] Jun Rekimoto: [UIST’97, CHI’98, CHI’99, …]

Pick and drop;handhelds withbig displays

PARCTab [Want 95], [Weiser, CACM’93] Abowd: eClass using Handhelds [CHI’98] Greenberg: public and private data

[Personal Technologies, 3(1), 1999] Jun Rekimoto: [UIST’97, CHI’98, CHI’99, …]

Pick and drop;handhelds withbig displays

Brad Myers 15

Remote Commander

Allow PDAs to control a PC Can be used with any application Uses the standard (single) cursor Don’t have to jump up and

grab mouse All mouse and keyboard functions

Use PDA like touchpad Graffiti or our own pop-up

keyboard Word prediction and completion

Allow PDAs to control a PC Can be used with any application Uses the standard (single) cursor Don’t have to jump up and

grab mouse All mouse and keyboard functions

Use PDA like touchpad Graffiti or our own pop-up

keyboard Word prediction and completion

Brad Myers 16

For People with Muscular Disabilities

Using handhelds as interface to PCs People with Muscular Dystrophy

have fine-motor control but losegross motor control

Difficulties with mouse and keyboard,but stylus OK

So use Remote Commanderas PC’s keyboard and mouse

Using handhelds as interface to PCs People with Muscular Dystrophy

have fine-motor control but losegross motor control

Difficulties with mouse and keyboard,but stylus OK

So use Remote Commanderas PC’s keyboard and mouse

                           

Brad Myers 17

PocketPC version Get PC’s screen onto PocketPC Full view, or one-to-one zooming

Scroll with iPaq’s buttons Control or scribble

Get PC’s screen onto PocketPC Full view, or one-to-one zooming

Scroll with iPaq’s buttons Control or scribble

Brad Myers 18

Power Point Control Use PC to give the presentation Use hand-held to control the PC

Two-way communication Hand-held shows picture of slide,

notes, timer

Use PC to give the presentation Use hand-held to control the PC

Two-way communication Hand-held shows picture of slide,

notes, timer

Brad Myers 19

SlideShow Commander, cont.

See list of slide titles Scribble Tap on on-screen buttons

and links Preview other

slides Control

demonstrations

See list of slide titles Scribble Tap on on-screen buttons

and links Preview other

slides Control

demonstrations

Brad Myers 20

Control at a Distance

Controlling a PC fromacross the room Meetings, etc.

Use a laser pointer? Studies Difficult due to:

Jitter of hand (+/- 10 pixels) No button on the laser pointer People not accurate at where

beam will go on and off

Controlling a PC fromacross the room Meetings, etc.

Use a laser pointer? Studies Difficult due to:

Jitter of hand (+/- 10 pixels) No button on the laser pointer People not accurate at where

beam will go on and off

Brad Myers 21

Idea:“Semantic Snarfing” Grab contents to handheld

Picture “Semantic” because

reformat based on the meaning of the contents Menus Text Re-visualization

Grab contents to handheld Picture

“Semantic” because reformat based on the meaning of the contents Menus Text Re-visualization

Brad Myers 22

Command Post of the Future Large DARPA funded project Make commanders more effective Private drill-down of public information

Large DARPA funded project Make commanders more effective Private drill-down of public information

Brad Myers 23

Classroom Investigations

Investigate improving large lectures with frequent in-class “concept tests” Provide immediate feedback to instructor

Easy for instructor

Investigate improving large lectures with frequent in-class “concept tests” Provide immediate feedback to instructor

Easy for instructor

Brad Myers 24

Scrolling Desktop Applications

Scrolling using the non-dominant hand Parallel and efficient use of both hands Generates Windows

scrolling events Study showed can be

faster for some tasks

Scrolling using the non-dominant hand Parallel and efficient use of both hands Generates Windows

scrolling events Study showed can be

faster for some tasks

Brad Myers 25

Shortcutter

User-created panels of controls Create custom interfaces and

extensions to PC applications And then take them with you

Direct manipulationfor edit, then setproperties

PocketPC or Palm

User-created panels of controls Create custom interfaces and

extensions to PC applications And then take them with you

Direct manipulationfor edit, then setproperties

PocketPC or Palm

Brad Myers 26

Shortcutter Widgets

Buttons Virtual Joy stick Virtual Knob Sliders Mouse pad Gesture panel

Buttons Virtual Joy stick Virtual Knob Sliders Mouse pad Gesture panel

Brad Myers 27

Shortcutter Actions

Send any keyboard key, mouse button, scrolling action or string to PC

Open a file or URL Run an application Invoke any PC menu or button

Windows message Recorded

Switch to a different Shortcutter panel Control the Mouse

Send any keyboard key, mouse button, scrolling action or string to PC

Open a file or URL Run an application Invoke any PC menu or button

Windows message Recorded

Switch to a different Shortcutter panel Control the Mouse

Brad Myers 28

Shortcutter Actions, cont. Control external devices

through PC’s serial port Directly (e.g., projectors) X-10 for electrical devices

Macro Can be multi-application

Application-specific Same button, different messages Useful for application sets:

browsers, compilers

Control external devicesthrough PC’s serial port Directly (e.g., projectors) X-10 for electrical devices

Macro Can be multi-application

Application-specific Same button, different messages Useful for application sets:

browsers, compilers

Brad Myers 29

More Scenarios of Use

Lean-back mail reading

Controlling media players

… and many others

Lean-back mail reading

Controlling media players

… and many others

Controlling AppliancesControlling Appliances

PhD research of Jeffrey Nichols PhD research of Jeffrey Nichols

http://www.pebbles.hcii.edu/puc/http://www.pebbles.hcii.edu/puc/

Problem

Too many complex devices, eachwith its own idiosyncratic interface Stereo system Telephones ATM Fax machine Photocopier Hotel alarm clock

Increasingly computerized Low usability

Too many complex devices, eachwith its own idiosyncratic interface Stereo system Telephones ATM Fax machine Photocopier Hotel alarm clock

Increasingly computerized Low usability

Brad Myers 32

Problem

Brad Myers 33

Existing “universal” controls

Pre-programmed at the factory with a subset, Or, Laboriously hand-programmed by the user

Pre-programmed at the factory with a subset, Or, Laboriously hand-programmed by the user

Brad Myers 34

Important Work by Others

Xweb [Olsen Jr. 2000] INCITS V2 standardization effort: Alternative

Interface Access Protocol (AIAP) [Zimmermann, CHI’02]

Stanford iRoom, iCrafter[Ponnekanti, UBICOMP 2001]

Xweb [Olsen Jr. 2000] INCITS V2 standardization effort: Alternative

Interface Access Protocol (AIAP) [Zimmermann, CHI’02]

Stanford iRoom, iCrafter[Ponnekanti, UBICOMP 2001]

Brad Myers 35

Our Approach Handheld is a “Personal Universal Controller” (PUC) Two-way communication

Appliance describes its functions Handheld PUC

Automatically creates interface Controls the appliance Displays feedback about appliance status

Handheld is a “Personal Universal Controller” (PUC) Two-way communication

Appliance describes its functions Handheld PUC

Automatically creates interface Controls the appliance Displays feedback about appliance status

Specifications

Control

Feedback

Brad Myers 36

Automatic Generation of UIs

Benefits All interfaces consistent for a user

With conventions of the handheld Even from multiple manufacturers

Addresses hotel alarm clock problem Can take into account user preferences Multiple modalities (GUI + Speech UI)

A Hard Problem Previous automatic systems have not

generated high quality interfaces

Benefits All interfaces consistent for a user

With conventions of the handheld Even from multiple manufacturers

Addresses hotel alarm clock problem Can take into account user preferences Multiple modalities (GUI + Speech UI)

A Hard Problem Previous automatic systems have not

generated high quality interfaces

Brad Myers 37

Research Challenges

1. Automatic Design of Graphical User Interfaces

2. Automatic Design of Speech User Interfaces

3. Connection with real devices Through various protocols X-10, AV/C, HAVi, UPnP, etc. Also, custom hardware and software

1. Automatic Design of Graphical User Interfaces

2. Automatic Design of Speech User Interfaces

3. Connection with real devices Through various protocols X-10, AV/C, HAVi, UPnP, etc. Also, custom hardware and software

Brad Myers 38

Hand-Generated Graphical Interfaces

First, Hand-Designed PocketPC interfaces: AIWA Shelf Stereo (Tape,CD,Tuner) AT&T Telephone/Answering

Machine

Used Embedded Visual Basic

Ensured quality with heuristicanalysis and think-aloud studies

Compared with manufacturer’sinterfaces

First, Hand-Designed PocketPC interfaces: AIWA Shelf Stereo (Tape,CD,Tuner) AT&T Telephone/Answering

Machine

Used Embedded Visual Basic

Ensured quality with heuristicanalysis and think-aloud studies

Compared with manufacturer’sinterfaces

Brad Myers 39

Results of Comparison

Using PUC, users took 50% less time & made 50% fewer errors

All differences are significant (p < 0.05)

Using PUC, users took 50% less time & made 50% fewer errors

All differences are significant (p < 0.05)

Brad Myers 40

Discussion of Comparison

Our hand designed interfaces succeeded for several reasons Good organization Each button has one function Good labels Only available functions are active

Others, hidden on tabs or grayed out Better feedback and error messages

Our hand designed interfaces succeeded for several reasons Good organization Each button has one function Good labels Only available functions are active

Others, hidden on tabs or grayed out Better feedback and error messages

Brad Myers 41

Current PUC Specification Language

XML Full documentation for the

specification language and protocol:

http://www.pebbles.hcii.cmu.edu/puc/

Contains sample specification for a stereo

XML Full documentation for the

specification language and protocol:

http://www.pebbles.hcii.cmu.edu/puc/

Contains sample specification for a stereo

Brad Myers 42

Properties of PUC Language

State variables & commands Each can have multiple labels

Useful when not enough room Typed variables

Base types: Boolean, string,enumerated, integers,fixed-point, floating-point, etc.

Optional labels for values Hierarchical Structure

Groups

State variables & commands Each can have multiple labels

Useful when not enough room Typed variables

Base types: Boolean, string,enumerated, integers,fixed-point, floating-point, etc.

Optional labels for values Hierarchical Structure

Groups

Brad Myers 43

Dependency Information

Crucial for high-quality interfaces Expressed as <active-if> clauses

Operations: Equals, Less-Than,

Greater-Than Combined Logically

AND, OR Used for:

Dynamic graying out Layout Widget selection

Crucial for high-quality interfaces Expressed as <active-if> clauses

Operations: Equals, Less-Than,

Greater-Than Combined Logically

AND, OR Used for:

Dynamic graying out Layout Widget selection

Brad Myers 44

Specifications

Have working specifications for: Audiophase stereo X-10 lights control Sony CamCorder Windows Media Player Audio ReQuest hardware MP3 player WinAmp Media Player Elevator

Have working specifications for: Audiophase stereo X-10 lights control Sony CamCorder Windows Media Player Audio ReQuest hardware MP3 player WinAmp Media Player Elevator

Brad Myers 45

Examples of Generated GUIs

Stereo and X-10 Stereo and X-10

Brad Myers 46

More Examples

Elevator Elevator

Brad Myers 47

Generating Speech Interfaces “Universal Speech Interface” (USI) project

Prof. Roni Rosenfeld of CMU http://www.cs.cmu.edu/~usi

Creates grammar, language model and pronunciation dictionary from PUC specification Pronunciation from labels using phonetic rules Can provide other pronunciations as labels for fine-tuning

Will use dependency information to help with disambiguation and explanation

Supports queries and spoken feedback Paraphrases as confirmation

“Universal Speech Interface” (USI) project Prof. Roni Rosenfeld of CMU http://www.cs.cmu.edu/~usi

Creates grammar, language model and pronunciation dictionary from PUC specification Pronunciation from labels using phonetic rules Can provide other pronunciations as labels for fine-tuning

Will use dependency information to help with disambiguation and explanation

Supports queries and spoken feedback Paraphrases as confirmation

Brad Myers 48

PUC Architecture

Brad Myers 49

Adaptors

“Adaptors” provide the interface to existing (and future) appliances If do not support specification language directly

Custom hardware Audiophase Stereo

Custom software ARQ MP3 player SIMA MP3 player (future)

X-10 Light switches, etc.

“Adaptors” provide the interface to existing (and future) appliances If do not support specification language directly

Custom hardware Audiophase Stereo

Custom software ARQ MP3 player SIMA MP3 player (future)

X-10 Light switches, etc.

AV/C (standard protocol) Sony CamCorder PlayStation, etc. (future)

HAVi (current work) Mitsubishi HDTV 65” TV Mitsubishi HDTV VCR

UPnP (future) ??

AV/C (standard protocol) Sony CamCorder PlayStation, etc. (future)

HAVi (current work) Mitsubishi HDTV 65” TV Mitsubishi HDTV VCR

UPnP (future) ??

Brad Myers 50

Adaptor using Custom Hardware for Stereo

Pretends to send IR codes Reads LED panel signals to decode state Created by Pittsburgh company: Maya Design

Pretends to send IR codes Reads LED panel signals to decode state Created by Pittsburgh company: Maya Design

Brad Myers 51

Adaptor for AV/C

Hand-written spec of AV/C

devices

PUC Java Proxy

Our Java device code

Our C++ code

Microsoft DirectShowlibrary

AV/C protocol

Firewire cable(IEEE 1394)

PUC Java Library

AutomaticGUI

generation

Digital VideoCamera or VCR

Multiple devicecontrollersMultiple devicecontrollers

AutomaticSpeech UIgeneration

Brad Myers 52

Demonstration

X-10 Camcorder through AV/C protocol for

IEEE 1394 (Firewire) Two way communication:

When state changes from appliance, GUI is updated and can query state with speech

X-10 Camcorder through AV/C protocol for

IEEE 1394 (Firewire) Two way communication:

When state changes from appliance, GUI is updated and can query state with speech

Brad Myers 53

For More Information

Many papers on the Pebbles web site On PUC and other work

Most programs available for free downloading:

SlideShow Commander available commercially

Many papers on the Pebbles web site On PUC and other work

Most programs available for free downloading:

SlideShow Commander available commercially

http://www.cs.cmu.edu/~pebbleshttp://www.cs.cmu.edu/~pebbles

http://www.slideshowcommander.comhttp://www.slideshowcommander.com

Brad Myers 54

Conclusions

Study The Big Picture: How mobile devices fit into users’ entire

information and control space

As more and more electronics are computerized, mobiles devices can help control them

Mobile devices can improve the user interfaces of everything else Not just be another gadget to be learned

Study The Big Picture: How mobile devices fit into users’ entire

information and control space

As more and more electronics are computerized, mobiles devices can help control them

Mobile devices can improve the user interfaces of everything else Not just be another gadget to be learned

Brad Myers 55

Thanks to Our Sponsors!The Pebbles research project is supported by

grants from:DARPAMicrosoftNSFPittsburgh Digital GreenhouseGeneral Motors

And equipment grants from:Hewlett PackardLucent TechnologiesMitsubishiPalm ComputingSymbol Technologies

The Pebbles research project is supported by grants from:DARPAMicrosoftNSFPittsburgh Digital GreenhouseGeneral Motors

And equipment grants from:Hewlett PackardLucent TechnologiesMitsubishiPalm ComputingSymbol Technologies

IBMSMART TechnologiesVividLogicSynergy SolutionsHandango

IBMSMART TechnologiesVividLogicSynergy SolutionsHandango

Mobile Devices for ControlMobile Devices for Control

Brad A. Myersbam@cs.cmu.edu

http://www.cs.cmu.edu/~pebbles

Brad A. Myersbam@cs.cmu.edu

http://www.cs.cmu.edu/~pebbles

Human Computer Interaction Institute Human Computer Interaction Institute

School of Computer ScienceSchool of Computer Science

Carnegie Mellon UniversityCarnegie Mellon University