Hybrid User Interfaces for Augmented Reality

Hybrid User Interfaces for Augmented Reality

Alex Olwal, Ph.D. olwal.com

Human-Computer Interaction, KTH

Alex Olwal, Ph.D.

2003-2010 KTH Stockholm

2006 Microsoft Research Redmond

2005 University of California Santa Barbara

2002-2003 Columbia University New York

startup Rule Communication Stockholm www.rule.se

startup Harmonic Solutions Stockholm www.harmonic.se

Display Systems

Sensing

Interaction Techniques

WIKITUDE

Layar

Augmented Reality (AR) on mobile phones

X: The Man with the X-Ray Eyes (1963)

RoboCop (1987)

Terminator 3 (2003)

Augmented Reality (AR)

• Empower our senses → enhance perception

• Supplement real

• Seamless

AR Fuse virtual objects with real environment

• View

• Perspective

• Real-time

Head-worn

Ivan Sutherland 1968 5dt HMD Nomad ProView XL40/50 STm AddVisor 150

Head-worn

Handheld, Situated & Hybrid

Everywhere Displays [Pinhanez et al. 2001], iLamp Projectors [Raskar et al. 2003],On-board Mixed Reality Projector [Karitsuka & Sato 2003], Transflective Surfaces [Bimber et al. 2000], Cell phone [Möhring & Bimber 2004], NaviCam [Rekimoto 1997], PDA [Wagner & Schmalstieg 2003]

Ubiquitous computingWeiser 1993

Ubiquitous

• Displays

• Sensing

• Connectivity

Devices

• Tabs, pads & boards

Interactive surfaces & tangible UIs

• Display 2D graphics

• Sense on & above surface

• Detect & augment objects

DigitalDesk [ Wellner 1993 ]

Tangible Bits [ Ullmer & Ishii & Buxton 1997 ]

Augmented Surfaces [ Rekimoto & Saitoh 1999 ]

PlayAnywhere / PlayTogether [ Wilson 2005 / Wilson & Robbins 2006 ]

• Tracked display

• Context-sensitive

• Focus + context

Chameleon [ Fitzmaurice 1993 ]NaviCam [ Rekimoto 1995 ]Augmented Notebook [ Mackay et al. 2002 ]Focus + Context displays [ Baudisch et al. 2002 ]VITA [ Benko et al. 2004 ]Ubiquitous graphics [ Sanneblad & Holmquist 2006 ]

Spatially aware displays

14

NaviCamRekimoto 1995

Hybrid UIs for AR

1 Unobtrusive

2 Sensing in environment

AR, UbiComp, Mobile, Interactive Surfaces, …

Shopping window Microsoft Surface

SurfaceFusionSensing & augmenting everyday objects

GI 2008 [Olwal & Wilson]Proc. Graphics Interface

LightSenseDynamic augmentations of printed media

ISMAR 2006 [Olwal]Proc. IEEE & ACM International Symposium on Mixed & Augmented Reality

Ericsson Demo CenterStockholm 2008

LUMARHybrid 2D + 3D handheld AR

ICAT 2007 [Olwal & Henrysson]Proc. International Conference on Artifical Reality & Teleexistence

Judge’s Choice Award in Nokia “Calling All Innovators” Challenge 2009Mobile World Congress, Barcelona, 13-16 February, 2009

LUMARHybrid 2D + 3D handheld AR

Printed media

Floor plan

2D graphics

Photographs

3D graphics

3D model

Spatially aware handheldsEnhancing interaction with large displays

TEI 2009 [Olwal & Feiner]Proc. International Conference on Tangible & Embedded Interaction

INTERACT 2009 [Olwal]Proc. IFIP TC13 Conference on Human-Computer Interaction

Ericsson Trade Show Events in 2009 / 2010Barcelona, Las Vegas, Boston, Galway, Stockholm, Paris, Amsterdam, San Francisco

352 px

288

px

408 mm

306

mm

800 px

600

px

71 px

95 p

x

camera view

projection & touch input

mobile device 2704 mobile px

2021

mob

ile p

xdevice tracking region

improved visual fidelity & touch input resolution

projected pixels

mobile pixels

240 px

36 mm

48 m

m

320

px

217 px

163

px

px/mm2

0

20

40

60

80

100

120

140

160

180

Projector32"

1024×768

LCD21"

1600×1200

Apple iPhone3.5"

320×480

Sony Ericsson G9002.4"

240×320

Nokia 5800XM3.2"

360×640

Google Nexus One3.7"

480×800

Apple iPhone 43.5"

640x960

ResolutionDenser pixels & sensors on handhelds

Augmenting surface interaction Mobile devices ・ expanded capabilities

INTERACT 2009 [Olwal]Proc. IFIP TC13 Conference on Human-Computer Interaction

Ericsson Trade Show Events in 2009 / 2010Barcelona, Las Vegas, Boston, Galway, Stockholm, Paris, Amsterdam, San Francisco

• Need HybridSurface slide

Augmenting surface interactionCollaboration for multiple users, devices & locations

TEI 2009 [Olwal & Feiner]Proc. International Conference on Tangible & Embedded Interaction

INTERACT 2009 [Olwal]Proc. IFIP TC13 Conference on Human-Computer Interaction

Ericsson Trade Show Events in 2009 / 2010Barcelona, Las Vegas, Boston, Galway, Stockholm, Paris, Amsterdam, San Francisco

Hybrid UIs for AR

1 Unobtrusive

2 Sensing in environment

3 Novel display technology

AR, UbiComp, Mobile, Interactive Surfaces, …

Shopping window Microsoft Surface Microvision HUD Heliodisplay, Mid-air

ASTORTransparent window with autostereoscopic 3D overlays

SPIE 2008 [Olwal, Gustafsson & Lindfors]Proc. SPIE Electronic Imaging, Vol 6804

ISMAR 2005 [Olwal, Lindfors, Gustafsson, Kjellberg & Mattson]Proc. IEEE & ACM Symposium on Mixed & Augmented Reality

SIGGRAPH 2004 Sketches [Olwal, Lindfors & Gustafsson]International Conference on Computer Graphics & Interactive Techniques

workspace

process data

workspace & process data

Unencumbered 3D interactionSupporting walk-up-and-use for public displays

NordiCHI 2008 [Olwal]Proc. Nordic Conference on Human Computer Interaction

eye tracker

touch overlay

transparent 3D display ( holographic optical element )

3D camera

Unencumbered 3D interactionManipulate 3D data using mobile, gestures & touch

NordiCHI 2008 [Olwal]Proc. Nordic Conference on Human Computer Interaction

3DTV Book 2008 [DiVerdi, Olwal, Rakkolainen & Höllerer]Three-Dimensional Television: Capture, Transmission & Display, Springer

VR 2006 [Olwal, DiVerdi, Rakkolainen & Höllerer]Proc. IEEE Virtual Reality

SIGGRAPH 2005 Emerging Technologies [Rakkolainen, DiVerdi, Olwal, Candussi & Höllerer]International Conference on Computer Graphics & Interactive Techniques

Immaterial displaysFace-to-face, reach-through & walk-through interaction

Hybrid UIs for AR

1 Unobtrusive

2 Sensing in environment

3 Novel display technology

4 Hybrid approaches → sensing / displays / interaction

AR, UbiComp, Mobile, Interactive Surfaces, …

BiDi: Flat depth-sensing displaysHirsch, Lanman, Holtzman & Raskar 2009

SecondLightIzadi, Hodges, Taylor, Rosenfeld, Villar, Butler & Westhues 2008

Interactions in the airHilliges, Izadi, Wilson, Hodges, Mendoza & Butz 2009

Nintendo WiimoteCamera + accelerometers + gyros + “sensor” bar

PlayStation MovePS Eye camera + Motion controller

Microsoft KinectDepth camera

Mobile projection

AR marketing

• Relevant

• Content consumption

• Interactivity

• Activity analysis

• projects@rule.se

Rule Communication – www.rule.se

HybridSurface – www.harmonic.se

• Hybrid interaction: Touch surface + mobile devices

• Ultra-thin design

• Mobile World Congress 2009 (Barcelona), CTIA Wireless 2009 (Las Vegas), Ericsson Volvo Ocean Race VIP Pavilion 2009 (Boston, Galway, Stockholm), Ericsson Experience Center

• Next generation for new Maritime Museum Exhibit 2011

Acknowledgements

Organizers

• Disruptive Media

Collaborators

• Department of Production Engineering, KTH

• Center for Medicine, Health and Technology, KTH / KI / KS

• FunkIS (HCI + radiology research)

• Department of Mechatronics

• VITA, Linköping University

• Computer Graphics & UI lab, Columbia University

• ilab, USCB

Funding

• KK foundation

• Swedish Institute for Assistive Technologies

• Engineer’s Science Academy / Innovation Bridge

• Blanceflor Foundation

• Swedish Research Council

Equipment, donations & funding

• Ericsson, Nokia, Microsoft Research, Mitsubishi, Doro, …

Hybrid User Interfaces for Augmented Reality

Alex Olwal, Ph.D.olwal.com

research: olwal.come-marketing: rule.se

new user interfaces : harmonic.se

Hybrid User Interfaces for Augmented Reality

Alex Olwal, Ph.D.olwal.com

research: olwal.come-marketing: rule.se

new user interfaces: harmonic.se