Ieee Dist Rib Sensing Public

8/2/2019 Ieee Dist Rib Sensing Public

1/56

1

Ken Hinckley

[email protected]

Microsoft Research

Nov. 20th

, 2003

Distributed Sensing

Techniques for Ubiquitous

Mobile Devices


2/56

2

Who am I anyway?

Research Scientist, PhD in Comp Sci

Background in Human-Computer Interaction

Interests Input Devices

Interaction Techniques

Sensors Mobile Devices

Ubiquitous Computing


3/56

3

Some Recent Projects

Meat & Potatoes

Mouse Scrolling wheel accelerated

scrolling


4/56

4

Accelerated

Scrolling Wheel

0

1

2

3

4

5

0

0.02

0.04

0.06

0.08

0.1

0.120.14

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Wheel Speed (dt)

AccelerationGain

Distributionof WheelSpeed

How do users roll the wheel?

(1) trying to get somewhere fast, or

(2) reading

Roll faster move further

No loss of control The user does not

have to learnanything new

Exp. study: Does no harm, users can scroll faster


5/56

5


Meat & Potatoes


scrolling Office Keyboard


6/56

6

Navigation controls on left

Scrolling - wide wheel Web - Forward / Back]

Windows - AppToggle

Cut, Copy, Paste keys on left

RH: Compound selection / IPLH: articulation of command

MS Office KeyboardHugh McLoone, Ken Hinckley, Ken Fry, Steve Bathiche


7/56

7

Uncanny Coincidence #1

October 2003 Office Keyboard Winner of HFES 2003 User

Centered Consumer Product Design Award!

Office Keyboard discontinued by Microsoft


8/56

8


Meat & Potatoes


scrolling Office Keyboard bimanual navigation

controls

More far-ranging research Sensing Techniques for Mobile Interaction


9/56

9

Sensors:

Tilt 2 axis accelerometer

Touch

Proximity

inexpensive, low power

What new features mightinexpensive, ubiquitoussensors enable?

VIDEO

ACM UIST 2000 Best Paper Award!

Sensing Techniques for Mobile

Interaction
http://research.microsoft.com/users/kenh/papers/KENHUIST2000.mpghttp://research.microsoft.com/users/kenh/papers/KENHUIST2000.mpg


10/56

10

Uncanny

Coincidence #2

11/20/2003Ken advocates coming ubiquity oftilt & other sensors sensors

11/18/2003Conveniently to make Kens point,Toshiba announces PortegeM200 Tablet PC Tilt fwd/back to scroll

Tilt L/R to flip pages Orient to change

Portrait / Landscape


11/56

11


Meat & Potatoes

Mouse Scrolling wheel acceleratedscrolling

Office Keyboard bimanual navigationcontrols

More far-ranging research Sensing Techniques for Mobile Interaction

Distributed Sensing Techniques for

Ubiquitous Mobile Devices


12/56

12

Still an immature technology

Slow, flaky

Do people really want to read web pages on theircell phones?

Sensor nets - applications for mobile devices?

Odd phrase: not what it is, but what it lacks In 100 years Wireless Network = ???

Wireless Networks


13/56

13

Horseless Carriage?


14/56

14

2003: Isolated Digital Islands


15/56

15

Wireless Fills in Gaps between

Devices: But Which Ones?


16/56

16

Users need techniques to intuitively formbridges between devices

How do users name the devices to connect?

What is type / purpose of the connection?

Parameters? (Who copies what, to where?)

Name That Device


17/56

17

User activity that is sensed by two ormore devices in a distributed system

Shared with nearby participating devicesvia wireless networking

System synchronizes time & puts the

pieces back together

Our Strategy:

Synchronous Gestures


18/56

18

Shaking Hands


19/56

19

Bowing


20/56

20

Making a Toast


21/56

21

VIDEOPerceptive TabletPCs
http://research.microsoft.com/users/kenh/papers/UIST_Talk_Video1.wmvhttp://research.microsoft.com/users/kenh/papers/UIST_Talk_Video1.wmv


22/56

22

Any bumps while devices not being heldare ignored

Example Bump Data


23/56

23

Bumping does more than name the devices:

Edges to join: Left, Right, Top, Bottom 2-axis tilt sensor gives this information

Direction (hierarchy in connection)

One tablet is base, one tablet is connecting

Without this, unclear which tablet is boss

Touch sensor: which device is held?

Bumping has Implicit

Parameters


24/56

24

How Synchronous

Gestures Enable

Wireless Matchmaking


25/56

25

Participating Devices

Connect to ServerSync Gesture

Server Server may be

hosted inenvironment,

or on a tablet.


26/56

26

Send Sensor Data to

ServerSync Gesture

Server


27/56

27

Time Sync

Sync GestureServer


28/56

28

Bump Recognized!

Sync GestureServer


29/56

29

Server Gives Partners

Each Others IP AddressSync Gesture

Server

Alternatively can have private connection via Sync

Gesture Server


30/56

30

Connect to Peer &

Link DisplaysSync Gesture

Server


31/56

31

Cant synchronize with everyone: n(n-1)/2

limited resources / power

false positives through sheer chance Ideally, should sync only w/ co-located devices

Possible scoping mechanisms

peer-to-peer:anyone in radio range syncs

proximity server:discover nearby devicesthrough 802.11 signal strengths (John Krumm)

Open Problem:

Who to Synchronize With?


32/56

32

Wireless signal strength provides proximity

Bumping offers explicit step of intentionality

that user has control over selection done in real world via manipulation

Bumping provides additional information

edges to join, type of connection, direction, etc.

Proximity provides list of devices tosynchronize with

Why Not Just Use

Proximity Sensing?

Oth W t


33/56

33

Sharing informationmutual or one way

Multiple tablets Face-to-face

collaboration

VIDEO #2

Other Ways to

Use Bumping
http://research.microsoft.com/users/kenh/papers/UIST_Talk_Video2.wmvhttp://research.microsoft.com/users/kenh/papers/UIST_Talk_Video2.wmv


34/56

34

Bumping works well, intuitive, fun

But requires hardware that is not (yet ) in

common useA few operations possible, but seems hard

to add further variations or semantics

Started thinking about generalizing theissues we encountered

The result was

What Next?


35/56

35

Ken HinckleyGonzalo Ramos (Intern)

Francois Guimbretiere (Visiting Researcher)

Patrick BaudischMarc Smith

Work currently submitted to CHI 2004

Stitching: Pen Gestures that

Span Multiple Displays

Stit hi


36/56

36

Movethe pen

Cross overbezel

Finish stroke

on nearbytablet

System infers

connection

Yields versatileframework for connecting

pen-operated devices

Stitching

path takenby the pen

transferredpictures


transferredpictures


transferredpictures


37/56

37

VIDEO
http://research.microsoft.com/users/kenh/papers/Stitching.mpghttp://research.microsoft.com/users/kenh/papers/Stitching.mpg


38/56

38

1. How is a connection established?

2. What type of connection is required?

3. What information is shared?

4. What is the spatial relationship btwdevices?

5. How do stitching gestures coexist with

other uses for the pen?6. How do users share physical space?

Implemented a photo sharing app

Central Design Questions


39/56

39

1. Establishing a Connection

There is nothing special about the pen!

Recognizes timing & dynamics of pen trajectory

Pen exits from edge of one screen &

enters at edge of another within a timeout

Tablet 1 Tablet 2

t


40/56

40

2. Type of Connection

User selects a multi-device command

e.g., Pie menu at end of stitching gesture


41/56

41

3. What Information to Share?

Many commands may need operands

Lasso to select, continue to other display to stitch.

4 Wh t i S ti l R l ti hi


42/56

42

4. What is Spatial Relationship:

Automatic Screen Calibration

Direction of Stitching

Device #1

Device #2

PA

PBp0

p1

p10p11

} offset

0

1

= (0 + 1) / 2

first half ofgesture

secondhalf of

gesture

Direction of StitchingDirection of Stitching

Device #1

Device #2

PA

PBp0

p1

p10p11

} offset

0

1

= (0 + 1) / 2

first half ofgesture

secondhalf of

gesture

Fits a line to absolute coords of users pen stroke

5 C i t f Stit hi


43/56

43

Stitching can occur in Trackingor Dragging

Trackingis separate layer on top of GUI

Draggingis only option for PDAs

Must undo drags that turn out to be stitches

5. Coexistence of Stitching

with Traditional Pen Use

6 How Do Users Share


44/56

44

6. How Do Users Share

Physical Space?

SociologyProxemics:How people use space Invisible bubble that surrounds an individual



45/56

45


Physical Space?


Homework assignment: Sit right next to someone at airport

(when it is not necessaryto do so)

Time how long it is before the other

person leaves



46/56

46


Physical Space?



47/56

47

Proxemic Lessons


48/56

48

Dont requirecontact

Do they have to be right next to one another?: 10-40cm

wide screen would be nice for collaboration

no two faces trying to peek at only one screen

Proxemic Lessons

for Stitching

Proxemic Lessons


49/56

49

Dont requirecontact

Do they have to be right next to one another?: 10-40cm

wide screen would be nice for collaboration

no two faces trying to peek at only one screen

Establish and Relax

Users want to Establisha connection, but then Relaxincreasing social tension by quickly exiting

Observed lean back, short stitches; transporter popular

One argument against remote menus

Flexibility: Allow users to be involved or not

Proxemic Lessons

for Stitching


50/56

50

Intimate 0-18

Personal 18- 30

Social: 4 7 Public12 or more

Distances vary by culture Touchingis taboo in non-contact cultures

E. T. Halls Distances in Man

Within arms reach}

Beyond arms reach}


51/56

51

Features of Stitching


52/56

52

Stitching supports Intimate, Personal & SocialDistances

Gives users flexibility to be involved, or not

Intimate Spaces:Combining screens. For closecollaboration with friend or trusted colleague

Personal Spaces:Tablets can be separated byup to 30 yet still possible to stitch to give files

to colleague, etc. Social Spaces:Once connected, transporter

can be used to give files to a user beyond armsreach

Features of Stitching

Correspond to Halls Distances

Proxemics:


53/56

53

Proxemics:

Orientation of Spaces

6 conversations3

1

0

Communication patternschange wrt orientation

Stitching shouldsupport: Shoulder-to-shoulder

Face-to-face

At 90 anglesRelative frequency:

Alternative Proxemics


54/56

54

Tablet 1 Tablet 2PDA

P

DA

Porches

Tablet 1 Tablet 2

PDA

The Candy Dish

Alternative Proxemics

Using PDAs

PDA Stitching not yet impl. but should be feasible


55/56

55

Combining heterogeneous devices Tablet + PDA? PDA + keyboard?

Jeff Pierce @ Ga. Tech: Opportunistic Annexing

+ Cell phone? + Watch?

+ Furby ?!?

Multiple Devices& Orientations

Future Directions

User 1

User2

User3

U

ser4


56/56

Questions?

Documents

Ieee Dist Rib Sensing Public