1 Ken Hinckley Gonzalo Ramos (Intern – U. Toronto) Francois Guimbretiere (Visiting Researcher -UMD) Patrick Baudisch Marc Smith Microsoft Research May

1

Ken HinckleyGonzalo Ramos (Intern – U. Toronto)

Francois Guimbretiere (Visiting Researcher -UMD)

Patrick BaudischMarc Smith

Microsoft Research May 26th, 2004

Stitching: Pen Gestures that Span Multiple Displays

2

• Still an immature technology – • Flaky, hard for users to configure, slow, …• Lots of research has focused on small-

screen presentation of web pages • But what are the completely new & unique

affordances of wireless connectivity?

• Odd phrase: not what it is, but what it lacks

Wireless Networks

3

Wireless Network 2004 = Horseless Carriage 1904?

• Are Wireless Networks really just wired networks without the wires?

• Or are they something completely different?

4

Wireless Fills in Gaps between Devices: But Which Ones?

5

• Name that Device• Users need techniques to intuitively form

bridges between devices• How do users name the devices to connect?• What is type / purpose of the connection?• Parameters? (Who copies what, to where?)

• Stitching offers unique approach that combines all of these steps using the pen

Spontaneous Device Sharing Problem:

6

• Move the pen

• Cross over bezel

• Finish stroke on nearby tablet

• System infers connection

• Standard Pen: No new hardware required

How to Stitch

path taken by the pen

transferred pictures





7

VIDEO

http://research.microsoft.com/users/kenh/papers/Stitching.mpg

8

1. Connection: How is a connection established?2. Command: What type of connection is required?3. Operands: What information is shared?4. Geometry: What is the spatial relationship btw devices?5. Coexistence: How do stitching gestures coexist with other

uses for the pen? 6. Proxemics: How do users share physical space?

Implemented a photo sharing app

Central Design Questions

9

1. Establishing a Connection

• There is nothing special about the pen!• Recognizes timing & dynamics of pen trajectory

• Pen exits from edge of one screen & enters at edge of another within a timeout

Tablet 1 Tablet 2

Δt

10

2. Type of Connection

• User selects a multi-device command• e.g., Pie menu at end of stitching gesture

11

3. What Information to Share?

• Many commands may need operands• Lasso to select, continue to other display to

stitch.• Sender/receiver (direction to send) naturally

specified

12

4. What is Spatial Relationship: Automatic Screen Calibration

Direction of Stitching

Device #1

Device #2

PA

PB

p0

p1

p10

p11

} offsetα0

α1

α = (α0 + α1) / 2

first half of gesture

second half of gesture

Direction of StitchingDirection of Stitching

Device #1

Device #2

PA

PB

p0

p1

p10

p11

} offsetα0

α1

α = (α0 + α1) / 2

first half of gesture

second half of gesture

• Fits a line to absolute coords of user’s pen stroke

13

• Stitching can occur in Tracking or Dragging

• Tracking is separate layer on top of GUI• Dragging is only option for PDA’s

• Must undo drags that turn out to be stitches• Works best full-screen; otherwise menus in way

5. Coexistence of Stitching with Traditional Pen Use

14

6. How Do Users Share Physical Space?• SociologyProxemics: How people use

space• Invisible bubble that surrounds an individual

15



• Homework assignment:• Sit right next to someone at

airport (when it is not necessary to do so)

• Time how long it is before the other person leaves

16



17

18

19

• Don’t require contact• Touching is taboo in many cultures• “Do they have to be right next to one another?”

• Participants separated devices by 10-40cm• People clearly view their device as an extension of

their physical body

• But users also see clear need for adjacent devices:• “wide screen would be nice for collaboration”• “no two faces trying to peek at only one screen”

Proxemic Lessons for Stitching (1)

20

• Give users flexibility to be involved (or not)• E. T. Hall’s Distances in Man

• Within Arm’s Length: Intimate, Personal


21

Intimate

22

Personal

23


• Within Arm’s Length: Intimate, Personal • Beyond Arm’s Length: Social, Public


24

Social

25

Public

26


• Within Arm’s Length: Intimate, Personal • Beyond Arm’s Length: Social, Public

• Stitching supports:• Intimate Spaces: Combining screens. For close

collaboration with friend or trusted colleague• Personal Spaces: Tablets can be separated by up

to 30” yet still possible to stitch to give files to colleague, etc.

• Social Spaces: Once connected, “transporter” can be used to give files to a user beyond arm’s reach


27


6 conversations

3

1

0

• Support varying relative body orientation

• Stitching supports:• Shoulder-to-shoulder• Face-to-face• At 90 angles

• VIDEO

Relative frequency:

http://research.microsoft.com/users/kenh/papers/AVI_Stitch_PocketPC_Orientations.wmv

28

Usability Testing

• 12 users in pairs, many had not used pens• Very positive comments & reactions overall

• All users got making pen gesture across displays• Simple way to share information• Ability to easily combine devices

29

Usability Testing

• 12 users in pairs, many had not used pens• Very positive comments & reactions overall

• All users got making pen gesture across displays• Simple way to share information• Ability to easily combine devices

• Problems• Stitch starting at edge of screen• User’s don’t want to be forced into contact• Security / Privacy a common concern• Various small issues, e.g. prefix vs. postfix menus

30

• Combining heterogeneous devices• Tablet + PDA?• “The Candy Dish”

Future Directions

Tablet 1 Tablet 2

PDA

31


Future Directions

Tablet 1 Tablet 2

PDA

32


Future Directions

Tablet 1 Tablet 2

PDA

33


Future Directions

Tablet 1 Tablet 2

PDA

34


Future Directions

Tablet 1 Tablet 2

PDA

35


• Cooperative Stitching

Future Directions

User 1

User 2

Use

r 3U

ser 4

36


• Cooperative Stitching•other users complete

the gesture

Future Directions

User 1

User 2

Use

r 3U

ser 4

37

Questions?

38

• Can’t synchronize with everyone: n(n-1)/2 • limited resources / power • false positives through sheer chance• Ideally, should sync only w/ co-located devices

• Possible scoping mechanisms• proximity server: discover nearby devices through

802.11 signal strengths (John Krumm)• peer-to-peer: anyone in radio range sync’s

Who to Synchronize With?

39

• Wireless signal strength provides proximity• Stitching offers explicit step of intentionality that

user has control over• selection done in real world via manipulation

• Stitching provides additional information• edges to join, type of connection, direction, etc.

• Proximity provides list of devices to synchronize with

Why Not Just Use Proximity Sensing?

40

Why not Infrared Port (IrDA)?

• Only one geometry: Relative orientation fixed

• Proximity does not imply desire to connect devices• “Do you want to share files with Joe Blow in seat

8A?”• Stitching, Bumping give user explicit control

• Lacks additional parameters (how to combine, operands, geometery of displays, sender vs. receiver, etc.)

• Mainly for sharing files, not about combining devices

• Still difficult to use and set up

41

42

• <patrick – following slides are some images of the most closely related systems – which you may want to briefly mention at some point in the talk, or just leave as extra slides in the case of questions…>

• <Maybe even do a drive-by-shooting just rapidly showing the related systems without touching on all the details of limitations and differences from stitching…>

• <I see stitching as blending ideas from Connectables, Pick-and-Drop, and SyncTap (although we implemented stitching before this last one was even published)>

• Tandler, P., T. Prante, C. Müller-Tomfelde, N.A. Streitz and R. Steinmetz. Connectables: dynamic coupling of displays for the flexible creation of shared workspaces. in UIST 2001. 2001.

• Rekimoto, J. Pick-and-Drop: A Direct Manipulation Technique for Multiple Computer Environments. in Proc. ACM UIST'97 Symp. on User Interface Software & Technology. 1997. Banff, Alberta, Canada.

• Rekimoto, J., Y. Ayatsuka and M. Kohno. SyncTap: An Interaction Technique for Mobile Networking. in Mobile HCI 2003. 2003: Springer.

• Rekimoto, J. and M. Saitoh. Augmented Surfaces: A Spatially Continuous Work Space for Hybrid Computing Environments. in CHI'99. 1999.

Related Work

43

Tandler - ConnecTables

• Displays mustbe within ~1 inch

• Single orientation• Single operation

44

Rekimoto – Pick & Drop

Copyright © Sony Computer Science Laboratories, Inc.Copyright © Sony Computer Science Laboratories, Inc.

• Pen with unique ID “stores” information• Devices don’t have to be placed together• Application: sharing files (copy/paste)

45

Rekimoto - SyncTap

Images Copyright © Sony Computer Science Laboratories, Inc.

• Brings up window for other device

• Which one sends?• Spatial relationship?

46

Rekimoto- Augmented Surfaces


47

Aug. Surfaces – “Hyperdragging”


48

49

How Do Synchronous Gestures Enable Wireless Matchmaking?

50

Participating Devices Connect to Server

Stitching Server

• Server may be hosted in environment,or on a tablet.

51

Send Sensor Data to ServerStitching Server

52

Time Sync

Stitching Server

53

Bump Recognized!

Stitching Server

54

Server Gives Partners Each Other’s IP Address

Stitching Server

• Alternatively can have “private” connection via Stitching Server

55

Connect to Peer & Link Displays

Stitching Server

56

• Users want to Establish a connection, but then Relax increasing social tension by quickly exiting

• Observed lean back, short stitches; transporter popular

• One argument against remote menus

Establish and Relax

Documents

1 Ken Hinckley Gonzalo Ramos (Intern – U. Toronto) Francois Guimbretiere (Visiting Researcher -UMD) Patrick Baudisch Marc Smith Microsoft Research May