COSC 426 Lect. 5 - Mobile AR

Lecture 5: Mobile ARLecture 5: Mobile AR

Mark Billinghurstk billi h t@hitl bmark.billinghurst@hitlabnz.org

HIT Lab NZ University of Canterbury

AR on mobile phonesLow cost, widely spread platform

Billions of phones deployedPeople know how to use themStrong demand from commercial sideHuge chance for AR!

Target practical applicationsEasy to useQuality graphicsRobust tracking15-30 Hz overall frame rate

Why would you use phones?Robust and fool-proof

People know how to use their mobile phones…

Variety of supported devicesSelf-contained operationSelf contained operation

Enough processing power to do everything natively

Ultra mobileUltra-mobileLow cost devices

!Even better: people already own the target hardware!

A very unique chance for bringing AR to the masses

Why would you not use phones?Compared to PC-based setups

Less processing power and memoryp g p yHarder to program, debug and deploy

Hardware difficult or impossible to extendHardware difficult or impossible to extendSmall number of available librariesT i ll l li l i i h Typically only little experience in research groupsSo many different devices and operating systems

Other limitations of handheld ARUsual limitations in mobile HCI

Small screen- Less information possible to display- Less immersion

Limited inputLimited input…

Other limitationsOther limitationsYou see through the camera and not through the phoneSwitch attention between background and phoneSwitch attention between background and phoneStrain factor of holding the phone upSocial issues with pointing the phone at peoplep g p p p

Mobile AR HistoryMobile AR History

E l f M b l AREvolution of Mobile AR

Camera phoneCamera phone

Thin client ARWearable AR Camera phone

- Self contained AR

WearableComputers

- Thin client AR

Handheld AR Displays

Self contained ARPDAs-Thin client AR

p yPDAs-Self contained AR

1995 1997 2001 2003 20041995 1997 2001 2003 2004

Handheld DisplaysTethered Applications

Fitzmaurice Chameleon (1994)( )Rekimoto’s Transvision (1995)Tethered LCDPC Processing and Tracking

Handheld AR Display - Tethered

1995, 1996 Handheld ARARPad CameleonARPad, CameleonRekimoto’s NaviCam, TransvisionTethered LCDTethered LCDPC Processing and Tracking

Mobile AR: Touring Machine (1997)Columbia University

Feiner, MacIntyre, Höllerer, Webster

Combines See through head mounted displayGPS t kiGPS trackingOrientation sensorBackpack PC (custom)p ( )Tablet input

MARS View

Vi t l t l id th l ldVirtual tags overlaid on the real world“Information in place”

Backpack/Wearable AR

1997 Backpack ARFeiner’s Touring MachineAR Quake (Thomas)Tinmith (Piekarski)MCAR (Reitmayr)Bulky, HMD based

Mobile AR - Hardware

GPSAntenna

RTK correction AntennaRTK correction AntennaRTK correction AntennaRTK correction Antenna Example self-built workingsolution with PCI-based 3D graphics

PCI 3D Graphics BoardHMDHMD

ControllerControllerHMDHMD

ControllerController

solution with PCI based 3D graphics

PC104 Sound Card

TrackerController

DC to DCCPU

PC104 PCMCIA

DC to DCConverter

Battery

WearableComputer

Hard Drive

Serial

GPS RTK correction

Radio

Ports

Columbia Touring Machine

1997 Philip Kahn invents camera phoneSharp J-SH04

1997 Philip Kahn invents camera phone1999 First commercial camera phone

Millions of Camera Phones

1000

1200

800

1000

DSC

400

600DSCPhone

0

200

2002 2003 2004 2005 2006 2007 2008 2009 2010

Handheld AR – Thin Client

2001 BatPortal (AT&T Cambridge)PDA used as I/O deviceWi l i k i Wireless connection to workstation Room-scale ultrasonic tracking (Bat)

2001 AR-PDA (C Lab)PDA thin graphics clientRemote image processingwww.ar-pda.com

Mobile Phone AR – Thin Client

2003 ARphone (Univ. of Sydney)Transfer images via Bluetooth (slow – 30 sec/image)g ( g )Remote processing – AR Server

Early Phone Computer Vision Apps2003 – Mozzies Game - Best mobile gameOptical motion flow detecting phone orientationSi SX1 S bi 120Mh VGA CSiemens SX1 – Symbian, 120Mhz, VGA Camera

2005 – Marble Revolution (Bit-Side GmbH)Wi f N ki ' S i 60 Ch ll 2005 Winner of Nokia's Series 60 Challenge 2005

2005 – SymBall (VTT)

Computer Vision on Mobile PhoneCameras and Phone CPU sufficient for computer vision applications Pattern Recognition (Static Processing)g ( g)

QR CodeShotcode (http://www.shotcode.com/)

M i Fl (2D I P i )Motion Flow (2D Image Processing)GestureTek

- http://www.gesturetekmobile.com/

TinyMotion

3D Pose CalculationAugmented RealityAugmented Reality

Handheld AR – Self Contained2003 PDA-based AR

ARToolKit port to PDAStudierstube ported to PDAStudierstube ported to PDAAR Kanji Educational App.Mr Virtuoso AR characterWagner’s Invisible Train

- Collaborative AR

Mobile Phone AR – Self Contained2004 M bil Ph AR2004 Mobile Phone AR

Moehring, BimberHenrysson (ARToolKit)Henrysson (ARToolKit)Camera, processor, display together

Location Aware Phones

Nokia NavigatorMotorola Droid

Real World Information OverlayTag real world locations Tag real world locations

GPS + Compass inputOverlay graphics data on live videoOverlay graphics data on live video

Applications Travel guide, Advertising, etc

Eg: Mobilizy Wikitude (www.mobilizy.com)Android based, Public API released

Other companiesLayar, AcrossAir, Tochnidot, RobotVision, etc

Layar – www.layar.com

HIT Lab NZ Android AR PlatformArchitectural ApplicationLoads 3D models

a OBJ/MTL format

Positions content in spacepGPS, compass

Intuitive user interfaceIntuitive user interfacetoolkit to modify the model

Connects to back end model databaseConnects to back end model database

HIT Lab NZ Mobile Outdoor AR

History of Handheld and Mobile AR1995 Handheld Display: NaviCam, AR-PAD, Transvision

1997 Wearable AR: Touring Machine, AR Quakeg

2001 Handheld AR – Thin Client: AR-PDA, Bat Portal

2003 Handheld AR Self contained: Invisible Train2003 Handheld AR – Self contained: Invisible Train

2003 Mobile Phone – 2D Vision: Mozzies, Symball

2003 Mobile Phone – Thin Client: ARphone

2004 Mobile Phone – Self contained: Moehring, Symbian

Mobile AR by Weight1996

2003 2007

Scale it down:Vesp‘R [Kruijff ISMAR07]:

Scale it down more:S t h $500

Backpack+HMD:5 8k

Vesp R [Kruijff ISMAR07]:…Sony UMPC 1.1GHz…1.5kg

till >$5K

Smartphone…$500…All-in-one…0.1kg

…5-8kg …still >$5K …billions of units

2011 S f h A2011 State of the ArtHandheld Hardware available

PDA, mobile phones, external camerasSensors: GPS, accelerometer, compass

Software Tools are AvailableTracking: ARToolKitPlus, QCARGraphics: OpenGL ESGraphics: OpenGL ESAuthoring: Studierstube, Layar, Wikitude, Unifye

What is needed:High level authoring toolsContent development toolsNovel interaction techniquesUser evaluation and usability

Mobile AR CompaniesMobile AR

GPS + compass

M C iMany CompaniesLayarWikitudeWikitudeAcrossair PressLiteYelpRobot visionEEtc..

$2 million USD in 2010$732 million USD in 2014

Handset ManufacturersQualcommQualcomm

$100 million USD investment

N kiNokia25+ people in NRC

SamsungExploring the space

Apple586 AR Applications on App Store

GoogleGoogle goggles/Android AR Applicationsg g gg pp

Qualcomm

Acquired Imaginationq gOctober 2010 - Releases free Android AR SDKComputer vision tracking - marker, markerless p gIntegrated with Unity 3D rendererhttp://developer.qualcomm.com/arp p q

Rock-em Sock-em

Sh d AR DShared AR DemoMarkerless tracking

Apple

iPhone 4 SDK supports direct camera accessppLaunches AR theme on App Store

> 500 AR apps on App Store

Developing AR applications

Mobile AR TechnologyInvolves

Tracking gContent Loading

Rendering/3D graphicsg g pUser Interface Application Design Application Design Evaluation

Scientific challengesAR requires (unlike related disciplines)AR requires (unlike related disciplines)

Strict real time operation (30Hz)- Unlike Ubicomp or mobile information systemsp y

High spatial precision (1cm, 1 degree)- Unlike location-based services

Robustness for operation by human user- Unlike many computer vision methods in automation etc.

M bil h AR i (i ddi i )Mobile phone AR requires (in addition)No thin client!S l l f f d k ARSame level of performance as desktop AR- New algorithms must be orders of magnitude more efficient

No unrealistic assumptions about HWNo unrealistic assumptions about HW

How does a basic AR application work?Main loopMain loop

Get a video frame from the cameraEstimate the position and the orientation

- computer vision, sensor input (GPS, compass)p p ( p )

Render the augmented scene (video + virtual content))Render GUIProcess user inputProcess user inputUpdate application status

Studierstube ES Framework

Typical AR application framework

Applications

Developed at TU Graz

StudierstGraz tube Software Staack

Platform

End User Application

Programming Lib i

pp

Libraries

OS/Low Level API

Hardware

Th S d b ES f kThe Studierstube ES frameworkA

pplicatioonsStudieContent

User Interface - Application

erstube Software S

Graphics

Content

StackTracking

Platform

Platform

Platform

What are the challenges?Experiences with embedded platforms requiredM l f ( i )Many platforms (operating systems)Slow CPUs (low clock rates, often no FPU)

Difficulties with trackingDifficulties with visualizations that require a lot of data

iprocessing

Slow memory accessNo or weak hardware 3D acceleration

Difficulties with graphics

Processing power of mobile phones

Weak processing power~1GHz Single core~1GHz, Single coreOften no floating point unit- Floating point code ~40x slower than integer codeoat g po t co e 0 s owe t a tege co e- Fixed point problematic for many algorithms- Requires good math library

Code optimized for phones runs 5-10x slower on ahigh-end phone than on an average PCNot going to change dramatically due to battery power

So what are the common problems?Bad camera quality under low lighting

Noise, motion blur,Strongly varies with different phones

Small memorySmall memoryKeeping large databases in memory is problematic

Slow memorySlow memoryLow clock rate- Processing large memory areas is prohibitiveg g y p- Typical CV building blocks (SVD, image processing) are too demanding

Slow data transfers between CPU and GPU

Pl f f h dh ld ARPlatforms for handheld ARPros Cons

Windows Mobile

Easy to program and debugLots of devices

Camera drivers are not always good

SymbianLargest installed basisGood devices

Hard to program and debug, SDK changes often, usually slow CPUs

Very nice hardware Camera API only with OS4iPhone

Very nice hardwareHype factor

Camera API only with OS4Objective C as main language

AndroidIncreasing number of devices

Java as main languageAndroidHype factor

Java as main language

Linux (LiMo, M )

Full Linux support, limitedh d f AR

Large set of different librariesMaemo) handsets  for AR

Large set of different libraries

RIM Blackberry

Widely spread in the US Java onlyBlackberry

Palm WebOS

Nice hardwareOnly very few devices so farNo native SDK so far

Worldwide smartphone market share

1Q101Q09

Symbian

RIMRIM

iPhoneiPhone

Android

Source: Gartnerhttp://www.gartner.com

2011 US Market Share

What makes a device interesting for AR?

Open and easy to programGood cameraGood cameraFast CPU, FPU is a plusGood H/W 3D supportLarge installed basisg

Easy access to devices

GPS, accelerometer, compassGPS, accelerometer, compassEnough memory/storage

Typical Smart Phone HardwareCPUCPU

300-800+ Mhz

GPUGPUNone, or Power VR Chip (OpenGL ES1.0/2.0)

InputpTouch screen, keyboard, keypad

SensorsGPU l (1 3 5 b )GPU, compass, accelerometer, camera (1.3-5mb+)

NetworkingBluetooth Wifi 3GBluetooth, Wifi, 3G

Screen320x240 up to 800x480

HTC HD2HTC HD2Windows MobileWindows MobileFast CPU (1GHz)Big screen

4.3”, 800x480

GPS, compass and accelerometerGood cameraGood camera

Depends on lighting conditions

Hardware 3DHardware 3DSlow texture upload: slow video background renderingslow video-background rendering

HTC DHTC DesireAndroidAndroidFast CPU (1GHz)Smaller screen

3.7”, 800x480

Multi-touchHardware 3DHardware 3DGPS, compass and accelerometeraccelerometer

Ph 4iPhone 4Apple iOS 4 0Apple iOS 4.0Faster CPU (1.2GHz)Hi h l iHigh screen resolution

3.5”, 960x640

(Finally) camera APIMulti-touchHardware 3DGPS compass accelerometerGPS, compass, accelerometerand gyroscope

Hardware SensorsC ( l i f )Camera (resolution, fps)

Maker based/markerless trackingVid lVideo overlap

GPS (resolution, update rate)Outdoor locationOutdoor location

Compass Indoor/outdoor orientationIndoor/outdoor orientation

AccelerometerMotion sensing relative tiltMotion sensing, relative tilt

Programming Environments

Mobile Development EnvironmentsNot like developing for desktops

Wide range of different OSLimited CPU, low memory, poor graphics, no floating point

Popular Mobile OSS bi (S bi C++ C bid IDE)Symbian (Symbian C++, Carbide IDE)iPhone (Objective C)Android (Java, Native NDK wrapper) (J pp )Windows Mobile (C#, C++, Visual Studio)

OtherPalm OS, Blackberry, Linux

Programming Windows MobileVery similar to desktop Windows

Almost identical APIU d f lUnicode functions only

Development toolsEmbedded Visual C++ C#Embedded Visual C++, C#

- Deprecated, not suggested

Visual Studio 2005Visual Studio 2008

- Required for FPU usage

F i f b l k tFor overview of camera bugs look athttp://studierstube.org/handheld_ar/camera_phones.php

Programming SymbianDevelopment tools

Carbide.c++Commercial version required for on-device debugging (important since emulator is bad…)SDK f dSDK appropriate for your device

Many quirksC i l d C++ tCrippled C++ supportWriting to static variables not allowed/recommendedCleanup StackpAPI includes ~1500 classes

Moving to Qtg

Programming iPhoneHarsh restrictions from Apple

Apps have to go through the apps store

X d IDE f d lXcode IDE for developmentNice development tools

Objective CObjective-CRequired for application developmentCan call into C/C++ codeCan call into C/C code

Camera API support in iOS 4+Can overlay on live videoy

AndroidHardware creatorsHardware creators

HTC, LG, Samsung, Motorola

Wid l il bl hWidely available phoneDifferent form factors – tablets, phones, PC, etc

Multiple versions and fragmentationAndroid 1.0, 1.6Android 2.0, 2.1

Free Tools Eclipse DevelopmentApp Integrator

Mobile Graphics

Computer Graphics on Mobile PhonesSmall screen, limited input optionsLimited support for accelerated graphics

Most phones have no GPU Most phones have no GPU

Mobile Graphics LibrariesOpenGL ES (1.0, 2.0)p ( , )

- Cross platform, subset of OpenGL- C/C++ low level library

Java M3GJava M3G- Mobile 3D graphics API for J2ME platform - Object importer, scene graph library- Support from all major phone manufacturerspp j p

O GL ESOpenGL ESSmall footprint subset of OpenGLSmall-footprint subset of OpenGL

OpenGL is too large for embedded devices!

P f l l l l API f ll f i li f 3D Powerful, low-level API, full functionality for 3D gamesCan do almost everything OpenGL can (but only one way)A l bl ll k l fAvailable on all key platformsSoftware and hardware implementations available

F ll iblFully extensibleExtensions like in OpenGL

No redundancy!Convenience functions removed

OpenGL ESSLIDE 68

OpenGL ES vs. OpenGL (1.x)OpenGL OpenGL ES

glBegin/glEnd 11

Primitive Types all no quads & polygons

Data Types float double int etc float fixedData Types float, double, int, etc… float, fixed

glDraw/Read Pixels glReadPixels only

T tTextures 1D, 2D, 3D, cube 2D

Stencil optional

Window Bindings WGL, GLX, etc… EGL

1: Except for Security Critical profile

OpenGL ESSLIDE 69

OpenGL ES on mobile devices

Java Applications C++ Applications

S h API G MiddlScenegraph APIM3G (JSR 184)

GameEngine

Middleware Engines

OpenGL ES Performance

( )Faster graphics (esp. hardware accelerated)Longer batter performance (> 10%)

VVersionsTwo major tracksTwo major tracks

Not compatible, parallel rather than competitive

OpenGL ES 1 xOpenGL ES 1.xFixed function pipelineSuitable for software implementationsSuitable for software implementationsAll 1.x are backwards compatible

OpenGL ES 2 xOpenGL ES 2.xVertex and pixel shaders using GLSL ESAll 2 x will be backwards compatibleAll 2.x will be backwards compatible

F d F (1 )Fixed Function (1.x)

http://www.khronos.org/opengles/2_X/

P bl (2 )Programmable (2.x)

http://www.khronos.org/opengles/2_X/

OpenGL ES 1.x vs 2.0

Tracking

Mobile Augmented Reality’s goal

Create an affordable, massively multi-user, widespread platform

© Tinmith, U. of South Australia

Tracking is…Estimating the device‘s pose (position and orientation) Strictly in real time (30Hz)Strictly in real time (30Hz)

With high spatial precision (1cm, 1 degree)Robustly for operation by human userRobustly for operation by human userNo unrealistic assumptions about HWLeaving enough power to other tasks (interaction graphics)Leaving enough power to other tasks (interaction, graphics)

Tracking requirements for AR on phonesFast and efficientFast and efficientForm factor: light and robustTrack simultaneously

A large number of objectsBy a large number of users

Requires little or no …qDevice modificationManual calibration (targeting non-technical users)( g g )Instrumentation of the physical environment

Low costsLow costs

Tracking on mobile phonesVision-based tracking

Marker-based trackingModel-based natural feature trackingNatural feature tracking in unknown environments

Sensor trackingGPS, inertial compass, gyroscope

Tracking for Handheld ARSLIDE 80

Backpack-based 1.

Höllerer et al. (1999), Piekarski & Thomas al. (2001), Reitmayr & Schmalstieg (2003)( ), ( ), y g ( )

Laptop, HMDEnhanced GPS (DGPS / RTK) + inertial sensor for viewpoint trackingEnhanced GPS (DGPS / RTK) + inertial sensor for viewpoint trackingHand tracking w/ fiducial markers

Tracking for Handheld ARSLIDE 81

Backpack-based 2.

Kalkusch et al., 2002Video see-through HMD w/ cameragViewpoint Tracking w/ inside-out computer vision using markersARToolKit markers on walls installed and surveyed manuallyy y

Tablet PC / UMPC-based 1.

Schall et al., 2006Hybrid tracking on UMPC

Camera fiducial marker trackinggWhen no marker in view inertial sensor + UWB tracking

Tablet PC / UMPC-based 2.CAMERA

LEDs

Klein & Drummond, 2004Combining outside in (LED tracking for low accuracy robust pose) & Combining outside-in (LED tracking for low accuracy, robust pose) & inside-out (edge-based tracking for high accuracy) computer vision

PDA-based 1.

BatPortal (Newman et al., 2001) SHEEP (MacWilliams et al., 2003)( , )

PDA as thin client (rendering & tracking on server + VNC)

Tracking by ART (external IR cameras + retroreflective target)

Ultrasonic tracking Projection-based AR environ.

PDA-based 2.

Signpost on PDA (Wagner & Schmalstieg, 2003)

First “truly” handheld AR platform: PDA + cameraStandalone self-contained AR systemStandalone, self contained AR systemOptimized fiducial marker tracking library

History of non-AR Tracking on Phones (1)

AR-PDA (Gausemeier et al., 2003)

Model based tracking

Kick Real (Paelke et al., 2004)

Edge detection of real foot + collisionModel-based trackingPDA = thin client

tracking off-loaded to serverNot real-time

Edge detection of real foot + collision detection w/ virtual ball2D tracking and limited interaction(tailored to game)

History of non-AR Tracking on Phones (2)

PhoneGuide (Bruns et al., 2005) LightSense (Olwal, 2006)

Neural network for recognizing visual features of museum artifactsCombined w/ BT “tracker”O

External camera tracks cell phone LEDSingle user, only coarse position tracking, no orientationBorder case of AROnly object recognition Border-case of AR

History of non-AR Tracking on Phones (3)

Mosquito Hunt (Siemens, 2003)Marble Revolution (BitSide, 2004)Pi i (VTT 2006)Pingis (VTT, 2006)Game control w/ optical flow techniques

TinyMotion (Wang et al., 2006)GUI control & input on cell phones

w/ image differencing & block correlationflow techniques w/ image differencing & block correlation

ARToolKit Tracking (Kato)

ARToolKit - Computer vision based marker tracking libraries

http://artoolkit.sourceforge.net/

History of AR Tracking on Phones (1)

2003ARToolKit on PDAARToolKit on PDAWagner et at.

200420043D Marker on PhoneMöhring et al.g

2005ARToolKit on SymbianHenrysson et al.

Tracking for Handheld ARSLIDE 91

Fiducial marker tracking on handhelds

Möhring et al., 2004 Henrysson et al., 2006Wagner et al., 2003

Rohs, 2006Bucolo et al., 2005

History of AR Tracking on Phones (2)

20052005Visual CodesRohs et atRohs et at.

2008Advanced Marker TrackingAdvanced Marker TrackingWagner et al.

2008Natural Feature TrackingWagner et al.

What can we do on today‘s mobile phones?

Typical specs600+ MHz~5MB of available RAM160x120 - 320x240 at 15-30 Hz camera

Possible to doMarker tracking in 5-15msNatural feature tracking in 20-50ms

Handheld AR Interfaces

Handheld HCI

Consider your userFollow good HCI principlesAdapt HCI guidelines for handheldsAdapt HCI guidelines for handheldsDesign to device constraintsRapid prototyping

User evaluation

Consider Your User■ They are probably mobile■ They are probably mobile

able to use the interface with one hand

■ They want quick access to application content■ They want quick access to application content.Want enhanced interaction with the real world

Interaction with the real world is the main focusInteraction with the real world is the main focus

■ They expect to be able to multitaskstart phone call, use another application, etcp pp

Norman’s Principles of Good PracticeEnsure a high degree of visibility

- allow the user to work out the current state of the system and the f ti ibl range of actions possible.

Provide feedback- continuous, clear information about the results of actions.continuous, clear information about the results of actions.

Present a good conceptual model- allow the user to build up a picture of the way the system holds

h h l i hi b i diff d h together, the relationships between its different parts and how to move from one state to the next.

Offer good mappingsg pp g- aim for clear, natural relationships between actions the user performs

and the results they achieve.

Hi h L l D i G id liHigh Level Design GuidelinesFrom Shneiderman’s 8 desktop design guidelines:p g g

Enable Frequent Users to Use ShortcutsOffer Informative FeedbackDesign Dialogs to Yield Closure

G d T i h’ id liGong and Tarasewich’s guidelines:Design for Small Devices Design for Limited and Split AttentionDesign for speed and recovery Design for speed and recovery Allow for personalizationDesign for EnjoymentDesign for Enjoyment

UI Device Constraints

Comparing Desktop to Handheld Interfaces

Screen Size Input Operation Multimedia Connectivity

Desktop > 1024 x 768 MouseKeyboard

Two handedStationary

Millions of coloursGraphics accel.5.1 Audio

WiredAlways On

Handheld < 640 x 480 StylusTouchButtons

One handedMobile

65K coloursNo graphics accel.Stereo audio

WirelessMaybe On

E m l O2 A ti MExample: O2 Active Menu

Highly visualUse PDA buttons for inputLarge icons and easy to read textVisually indicate which tabs are scrollableApplication UI looks different from device UIApplication UI looks different from device UI

iPhone Guidelines

Minimize required user input. Avoid unnecessary interactivity.y yProvide feedback when necessaryProvide fingertip sized target areasProvide fingertip-sized target areas.Avoid clutter and busy backgrounds. Express essential information succinctly.Make it obvious how to use your content.y

iPhone Interface

Designing for Device ConstraintsInput Device

Touch, stylus, keyboard, buttons, keypady y yp

ScreenSize, resolution,

SensorsCamera – frame rate image resolutionCamera frame rate, image resolutionGPS – resolution, coverageCompass - accuracyCo pass accu acy

Sample Handheld AR InterfacesCleanLarge Video ViewLarge Icons Large Icons Text Overlay

Twitter 360

www.twitter-360.comiPhone applicationppSee geo-located tweets in real worldTwitter com supports geo taggingTwitter.com supports geo tagging

Wikitude – www.mobilizy.com

BlahBlahBl h

BlahBlah

Bl h

BlahBlahBlah

BlahBlah

Bl h

Blah

Blah Blah

Blah

BlahBlahBlah

Blah

BlahBlahBlah

BlahBlah

BlahBlah

Information Filtering

Information Filtering Information Filtering (Julier et al ’00)Information Filtering (Julier et al. ’00)

• Remove clutter by goal- and distance based filtering • User’s task is route finding: Sniper and relevant buildings are displayed; bj t hi h d t i d t b dobjects, which are determined to be unnecessary, removed

W bl AR

HMD vs Handheld AR InterfaceHandHeld ARWearable AR

Output:Output:Display Input &

Output

Input

Handheld Interface MetaphorsTangible AR Lens Viewing

Look through screen into AR scene Interact with screen to interact with AR Interact with screen to interact with AR content

- Eg Invisible Train

Tangible AR Lens ManipulationSelect AR object and attach to device Use the motion of the device as input

- Eg AR Lego

Handheld Display vs Fixed Display

Experiment comparing handheld moving, to handheld button input, small fixed display, desktop display, large plasmaUsers performed (1) navigation task, (2) selection taskUsers performed (1) navigation task, (2) selection taskMoving handheld display provided greater perceived FOV, higher degree of presence, faster completion time

J. Hwang, J. Jung, G. Kim. Hand-held Virtual Reality: A Feasibility Study. In proceedings of VRST 2006

Search Task Completion Time

FOV, Presence and ImmersionPerceived FOV and Actual FOV (deg. marked by subjects)

5852

64 606070

33 31

52

30 30 30

45

2030405060

Perceived FOVActual FOV

01020

Motionbased hh

Buttonbased hh

Smallscreen

17' screen 42' screen5.45.76

7

based hh based hh screen4.3 4.1 4.3

4.74.5 4.34.7 4.9

3

4

5

PresenceImmersion

0

1

2

0Motion

based hhButton

based hhSmall

screen17' screen 42' screen

Rapid Prototyping

Speed development time by using quick hardware mockupsp p y g q phandheld device connected to PCLCD screenUSB phone keypadCamera

Can use PC development toolsCan use PC development toolsFlash, Visual Basic, etc

Mobile Physical PrototypingBug Labshttp://www.buglabs.net/

Open source hardware modules, each producing one or more services. p g

Modules snap together physically and the i h l i ll services connect together logically to

enable users to easily build applications.

Software PrototypingPython Symbian (HIT Lab NZ)

stbTracker wrapperstbTracker wrapperAccess to SMS, Bluetooth, GPSRapid developmentRapid development

import e32import appuifwfrom gles import *

if e32.s60_version_info>=(3,0):import imp

t i l d d i ('M t' ' \\ \\bi \\M t d')magnet=imp.load_dynamic('Magnet', 'c:\\sys\\bin\\Magnet.pyd')else:

import Magnetfrom Magnet import

#Define Model

def frameback(num_markers):if (num markers > 1):if (num_markers > -1):

glMatrixMode(GL_PROJECTION)#Draw graphics…

appuifw.app.orientation = 'landscape‘ # Use full frameSetCameraCallback(frameback) # Register callbackcreateCamera() # Define cameraInitGLES() # Start Open GLInitGLES() # Start Open GLTrackerInit() # Start trackerInitCamera() # Start camera

Design GuidelinesApply handheld HCI guidelines for on screen contentApply handheld HCI guidelines for on-screen content

- large buttons, little text input, etcDesign physical + virtual interface elementsPick appropriate interface metaphorpp p p

- “handheld lens” approach using handheld motionTangible AR for AR overlay- Tangible AR for AR overlay

Build prototypesContinuously evaluate application

AR BrowsersAR Browsers

AR BrowsersCommercial outdoor AR applications

Junaio, Layar, Wikitude, etc

All have their own language specificationsWikitude – ARMLJunaio - XML

Need for common standardBased on existing standards for geo-located content etcSupport for dynamic/interactive contentEasier to author mobile AR applicationsEasy to render on AR browsers

Layar

Junaio

Hello World Example" " " " "echo "<?xml version=\"1.0\" encoding=\"UTF-8\"?>

<results><poi id=\"1\" interactionfeedback=\"none\">

<name><![CDATA[Hello World POI]]></name><description><![CDATA[[This is my first POI.]]></description> <l>48.1385,11.5750,0</l>, ,<o>0,0,0</o>

<mime-type>text/plain</mime-type><thumbnail>http://www junaio com/publisherDownload/tutorial/icon jpg</<thumbnail>http://www.junaio.com/publisherDownload/tutorial/icon.jpg</thumbnail><icon>http://www.junaio.com/publisherDownload/tutorial/icon_map.png</icon> icon> </poi></results>"

KHARMA + Argon: A KML/HTML Architecture and Browser for AR Applications and Games

Bl i M I d Al HillBlair MacIntyre and Alex HillSchool of Interactive Computing,

145Georgia Institute of Technology 145

KHARMAKHARMA: KML/HTML Augmented Reality Mobile Architecturemedia hackersresearch tools {

skilled computationalistsour past focus {savvy technical designers

general public

{{

breadth of adoption

general publiccurrent focus{Ygrasil IDE

Designers AR Toolkit

Unity AR Toolkit

146146Designers AR Toolkit

KHARMA

KHARMA: KML/HTML Augmented Reality Mobile ArchitectureProblem: limited authoring tools for mobile ARProblem: limited authoring tools for mobile AR

limited expression (coord, name, desc, link) vs higher hurdle of 3Dno accepted standard and proprietary client protocollimited client-side interactivity is more akin to Web 1 0limited client-side interactivity is more akin to Web 1.0

Solution: HTML with KML combines What? and How? with Where?

• allows extensive client side (albeit 2D) interactivity and expressivityallows extensive client side (albeit 2D) interactivity and expressivity• two the most broadly adopted standards for presentation and geo location• HTTP server distribution, CSS and Javascript allow for true Web 2.0 content

++KML

147

HTML

KHARMA Architecture with four componentsArchitecture with four components

Channel servers- delivering individual channels of AR content,

Tracking servers Tracking servers - providing content related to location

Tracking infrastructure serversTracking, infrastructure servers- delivering information about the physical environment,

Mobile client - for generating the resulting augmentations

KML already supports HTML• description tag accepts CDATA enclosed markup

- but no global styling of scripting support• no control over balloon styling• no control over balloon position and orientation• no relative positioning <Placemark id="culc_center">

<name>CULC Visualization</name><description><![CDATA[

<di id " l i ">G i T h><div id="culc_image">Georgia Tech><img src="http://www.culc.net/culc.png"></div>

</description><Balloon>

<location><location><latitude>34.0</latitude><longitude>--84.5</longitude>

</location><orientation><orientation>

<heading>31</heading></orientation>

</Balloon></Placemark></Placemark>

<Placemark id="culc_center"><name>CULC Visualization</name><description><![CDATA[

KARML extension to KML

• added undecorated displayMode <description><![CDATA[<div id="culc_image"><img src="http://www.culc.net/culc.png"></div>

</description><Style>

<BalloonStyle>

• added undecorated displayMode

• added Balloon element<displayMode>undecorated</displayMode>

</BalloonStyle></Style><Balloon>

<locationMode targetId=”culc geospot”>relative</locationMode>

added Balloon element- similar in nature to Model element

g _g p<location>

<latitude units="meters">4.0</latitude><longitude units="meters">-2.5</longitude>

</location><orientation>

• relative locationMode- accepts “units” attribute

<orientation><heading>31</heading>

</orientation></Balloon>

</Placemark>

151151

GPS Located Content<Placemark>

GeospotSurveyed location

<Placemark><name>GeoSpot</name><description>a surveyed GeoSpot </description><Camera> <!-- camera element for GeoSpot -->

Moves camera to fixed location

p<longitude>-84.394135</longitude> <!- GPS coordinates --><latitude>33.76083</latitude><altitude>0</altitude><TimeStamp> <! when GeoSpot was surveyed ><TimeStamp> <!-- when GeoSpot was surveyed -->

<when>1997-07-16T10:30:15+03:00</when></TimeStamp><Icon>

<href>http://geospot.imtc.gatech.edu/image/03_icon.png</href></Icon>

</Camera><Point> <! location displayed on map or within browser view ><Point> <!-- location displayed on map or within browser view -->

<coordinates>-84.394135,33.76083</coordinates></Point>

</Placemark>

For more information, please visit:http://www argon gatech edu/http://www.argon.gatech.edu/

Developing AR Experiences

Sony CSL © 2004

Game Case Study

Resources

ResourcesBooksBooksMobile Interaction Design

M J d G M dMatt Jones and Gary MarsdenDesigning for Small Screens

Studio 7.5Handheld Usability

Scott WeissDesigning the Mobile User Experience

Barbara Ballard

Developer Guidelines

Palmhttp://www.access-company.com/developers/documents/docs/ui/UI_Design.html

Zen of Palm guidelineshttp://www.access-company.com/developers/documents/docs/zenofpalm.pdf

Motorolahttp://developer.motorola.com/docstools/developerguides/

iPhone Human Interface Guidelineshttp://developer.apple.com/documentation/iPhone/Conceptual/iPhoneHIG/

Handheld HCI Design WebsitesDo’s and Don’ts of PocketPC designhttp://www.pocketpcmag.com/_archives/Nov04/Commandements.aspx

U bili i l i h dh ld biliUsability special interest group – handheld usabilityhttp://www.stcsig.org/usability/topics/handheld.html

Usable Mobile websitehttp://www.smartgroups.com/groups/usablemobile

Mobile Coders Websitehttp://www mobilecoders com/Articles/mc-01 asphttp://www.mobilecoders.com/Articles/mc-01.asp

Univ of Waikato Handheld Grouphttp://www.cs.waikato.ac.nz/hci/pdas.html

Mobile Interaction Websitehttp://www.cs.waikato.ac.nz/~mattj/mwshop.html

Platform – Recommended readingLots of low level information on the complete ARM familyp yValuable tool for driver andframework developersframework developersNot that important for pure application developersapplication developers

Platform – Recommended readingVery low level and targeted for PCsMost information outdated on PCMost information outdated on PCEffective memory usage one of the most importantof the most importantoptimization strategies on mobile devices!on mobile devices!

Tracking – Recommended readingLots of the basics on theComputer Vision you will p yneed for AR trackingSeveral code and pseudo-codeSeveral code and pseudo codesnippets

Tracking – Recommended readingAll about the geometryyou will need for a tracking system

Camera modelsProjectionEpipolar geometryH hiHomographies…

Graphics – Recommended reading

Mobile 3D Graphics( ll b t O GL ES 1 )(all about OpenGL ES 1.x)OpenGL ES 2.0

GProgramming GuideOpenGL ES 2.0 Man Pageshttp://www.khronos.org/opengles/sdk/docs/man/ShaderX7Chapter on “ Augmented Reality on Mobile Phones”

OpenGL ES ResourcesKhronos Group OpenGL ES Page

http://www.khronos.org/opengles/

OpenGL ES 2.0 Bookhttp://www.opengles-book.com/

AMDs OpenGL ES 2.0 Emulator