AR & MR - Undergraduate Courses | Computer Science at Virginia

Augmented and mixed reality(AR & MR)

Doug BowmanCS 5754

Based on original lecture notes by Ivan Poupyrev

(C) 2008 Doug Bowman, Virginia Tech 2

AR/MR example


Definitions

Augmented reality: Refers to a system inwhich the user views and acts within anenhanced version of the real world. Theenhancements are virtual (computer-generated), and can include objects orinformation.

Mixed reality: Refers to a system thatcombines real and virtual objects andinformation.


Mixed reality continuum

Reality VirtualityAugmentedReality (AR)

AugmentedVirtuality (AV)

Mixed Reality (MR)

Milgram (1994)


AR/MR application areas

MaintenanceTrainingTourism / Cultural heritageDesign / constructionBattlefield information display


AR/MR technology - displays

See-through HMDs:Video see-throughOptical see-through

Handheld displays Projection


AR/MR technology - tracking

Optical / vision-basedtrackingAR toolkitEnsures portabilityLarge number of

tracked objects

Registration and lowlatency are crucial forAR systems


AR/MR technology - tracking

Sourceless inertial orientation trackingGPS position trackingEnables mobile outdoor AR


Mobile outdoor AR

“Backpack systems” User wears/carries:

ComputerHMD Inertial trackerGPS unit/antenna Input device(s)


AR video examples


Mixed Reality Interfaces

KARMA, Feiner, et al. 1993

Azuma (1997)combine real and virtual

objects interactive in real timevirtual objects are

registered in 3Dphysical world


Challenges in AR Interfaces

Conflict between realworld and virtualNot neatly separated

anymore Limitations of displays

Precise, fast registration &tracking

Spatially seamless display Limitations of controllers

Precise, fast registration &tracking

Spatially seamless interactivityImage Copyright Sony CSL


AR interfaces as 3D informationbrowsers (I)

3D virtual objects areregistered in 3D See-through HMDs, 6DOF

optical, magnetic trackers “VR in Real World”

Interaction 3D virtual viewpoint

control Applications

Visualization, guidance,training State, et al. 1996


AR interfaces as context-basedinformation browsers (II) Information is registered to real-world

context Hand held AR displaysHand held AR displays

Video-see-through (Video-see-through (RekimotoRekimoto,,1997) or non-see through1997) or non-see through((Fitzmaurice, et al. 1993Fitzmaurice, et al. 1993))

Magnetic trackers or computerMagnetic trackers or computervision basedvision based

Interaction Manipulation of a window

into information space Applications

Context-aware information displays Rekimoto, et al. 1997


AR Info Browsers (III):Pros and Cons

Important class of ARinterfacesWearable computersAR simulation, training

Limited interactivityModification and

authoring virtual contentis difficult Rekimoto, et al. 1997


3D AR Interfaces (I)

Virtual objects are displayed in 3Dspace and can be alsomanipulated in 3D See-through HMDs and 6DOF

head-tracking for AR display 6DOF magnetic, ultrasonic, or other

hand trackers for input Interaction

Viewpoint control 3D user interface interaction:

manipulation, selection, etc.Kiyokawa, et al. 2000


3D AR Interfaces (II):Information Displays

How to move information in ARcontext dependent informationbrowsers?

InfoPoint (1999)Hand-held deviceComputer-vision 3D trackingMoves augmented data

between marked locationsHMD is not generally

needed, but desired sincethere are little displaycapabilities

Khotake, et al. 1999


3D AR Interfaces (III):Pros and Cons

Important class of AR interfacesEntertainment, design, training

AdvantagesSeamless spatial interaction: User can interact with 3D

virtual object everywhere in physical spaceNatural, familiar interfaces

DisadvantagesUsually no tactile feedback and HMDs are often required Interaction gap: user has to use different devices for virtual

and physical objects


Tangible interfaces andaugmented surfaces (I)

Basic principlesVirtual objects are projected on

a surface back projection overhead projection

Physical objects are used ascontrols for virtual objects Tracked on the surface Virtual objects are registered to

the physical objects Physical embodiment of the

user interface elementsCollaborative

Digital Desk. 1993


Tangible Interfaces and AugmentedSurfaces (II)

Graspable interfaces, Bricks system(Fitzmaurice, et al. 1995) and Tangibleinterfaces, e.g. MetaDesk (Ullmer’97): Back-projection, infrared-illumination computer

vision tracking Physical semantics, tangible handles for virtual

interface elements

metaDesk. 1997


Tangible Interfaces and AugmentedSurfaces (III) Rekimoto, et al.

1998Front projectionMarker-based trackingMultiple projection

surfacesTangible, physical

interfaces+ AR interaction withcomputing devices

Augmented surfaces, 1998


Tangible Interfaces and AugmentedSurfaces (IV)Advantages

Seamless interaction flow – user hands areused for interacting with both virtual andphysical objects.

No need for special purpose input devicesDisadvantages

Interaction is limited only to 2D surfaceSpatial gap in interaction - full 3D interaction

and manipulation is difficult


Orthogonal nature of AR interfaces(Poupyrev, 2001)

Nosame devices for

physical andvirtual objects

Yesseparate devices forphysical and virtual

objects

Interaction gap

Yesinteraction is onlyon 2D surfaces

Nointeraction iseverywhere

Spatial gap

Augmentedsurfaces

3D AR


Tangible AR interfaces (I)

Virtual objects are registered to markedphysical “containers” HMD Video-see-through tracking and registration

using computer vision tracking Virtual interaction by using

3D physical container Tangible, physical interaction 3D spatial interaction

Collaborative

Shared Space, 1999


Tangible AR (II): generic interfacesemantics

Tiles semantics data tiles operation tiles

menu clipboard trashcan help

Operation on tiles proximity spatial arrangements space-multiplexed Tiles, 2001


DO NOT REMOVE!!!THIS IS A PLACEHOLDER FOR EXTRANOTES PAGE!!!!!!!


Tangible AR (III):Space-multiplexed

Data authoring in Tiles (Poupyrev, et al. 2001). Left, outsideview of the system; right, view of the left participant.


Tangible AR (IV): Time-multiplexedinteraction

Data authoring in WOMARinterfaces (Kato et al. 2000). Theuser can pick, manipulate andarrange virtual furniture using aphysical paddle.


Tangible AR (V): AR - VR TransitoryInterfaces

Magic Book (Billinghurst,et al. 2001)3D pop-up book: a

transitory interfacesAugmented Reality

interfacePortal to Virtual

Reality Immersive virtual

reality experienceCollaborative

Augmented Reality

Virtual Reality


Tangible AR (VI):ConclusionsAdvantages

Seamless interaction with both virtual andphysical toolsNo need for special purpose input devices

Seamless spatial interaction with virtual objects3D presentation of and manipulation with virtual objects

anywhere in physical space

DisadvantagesRequired HMDMarkers should be visible for reliable tracking


Interfaces for mobile outdoor AR

Devices must behandheld

No tracking or limitedtracking for devices

Interaction at-a-distance

Tinmith project


Challenges in AR/MR

Occlusion and depth perceptionText display and legibilityVisual differences between real and virtual

objectsRegistration and trackingBulky HMDs and other equipment

Documents

AR & MR - Undergraduate Courses | Computer Science at Virginia