COSC 426 A d R liCOSC 426: Augmented Reality
Mark Billinghurst
July 2011
Lecture 1Lecture 1
Mark BillinghurstPhD Electrical EngineeringPhD Electrical Engineering
University of WashingtonInteraction DesignInteraction Design
Museum experiencesTools for designersTools for designers
Augmented RealityMobile AR Evaluation Mobile AR, Evaluation, Multimodal Interfaces, Collaborative
CollaborationCollaborationEnhanced FtF and remote collaborationSocial networkingSocial networking
OverviewOne tow hour lectures weekOne tow hour lectures week
Friday 9 – 11amYou will learnYou will learn
Introduction to Augmented RealityAugmented Reality technologyAugmented Reality technologyAR Interaction techniquesInteraction DesignInteraction DesignAR authoring toolsResearch directions in AR
Complete a simple project
C O lCourse OutlineWk 1 (July 15th): Introduction to AR(J y )Wk 2 (July 22nd): AR TechnologyWk 3 (July 29th): AR Developer Tools(J y ) pWk 4 (Aug 5th): AR Interaction TechniquesWk 5 (Aug 12th): AR ApplicationsWk 5 (Aug 12 ): AR ApplicationsWk 6 (Aug 19th): Outdoor and Mobile ARWk 7,8 (Aug 26th, Sept 2nd): Holidays Wk 7,8 (Aug 26 , Sept 2 ): Holidays Wk 9 (Sept 9th): Collaborative AR Wk 10 (Sept 16th): Usability Testing Wk 10 (Sept 16 ): Usability Testing Wk 11 (Sept 23rd): AR research DirectionsWk 12 (Sept 30th): Final Project Presentations Wk 12 (Sept 30 ): Final Project Presentations
Assessment - Update
Research project – 40%Group work (2-4 people)Group work (2-4 people)Due Sept 30th
Two Class Assignments – 20 %Programming assignments, individual workg g g ,
Final Exam – 40%E k O 3 d 14thExam week Oct 3rd – 14th
IntroductionIntroduction
A Brief History of Time
T dTrendsmaller, cheaper, more functions, more intimate
T h l b blTechnology becomes invisibleIntuitive to use I f i lInterface over internalsForm more important than functionH t d d iHuman centered design
A B f H f CA Brief History of Computing
Trendsmaller, cheaper, faster, more intimate, intelligent objects
Computers need to become invisiblehide the computer in the real world
- Ubiquitous / Tangible Computing
put the user inside the computer Vi l R li- Virtual Reality
Invisible Interfaces
Jun Rekimoto, Sony CSL
Virtual Reality
19891989…
Cheap HMDs
Virtual Reality
Immersive VR Head mounted display, glovesSeparation from the real worldSeparation from the real world
VR Today
$3-5 Billion VR business (+ > $150 Billion Graphics Industry)Visualization, simulation, gaming, CAD/CAE, multimedia, graphics arts
1977 – Star Wars
2008 - CNN
Augmented Reality DefinitionD f Ch [A 97]Defining Characteristics [Azuma 97]
Combines Real and Virtual Images- Both can be seen at the same time
Interactive in real-time- The virtual content can be interacted with
Registered in 3Dg- Virtual objects appear fixed in space
Augmented Reality Examples
Put AR pictures here
AR vs VR
Virtual Reality: Replaces RealityScene Generation: requires realistic imagesq gDisplay Device: fully immersive, wide FOVTracking and Sensing: low accuracy is okayTracking and Sensing: low accuracy is okay
Augmented Reality: Enhances RealityScene Generation: minimal rendering okayDisplay Device: non-immersive small FOVDisplay Device: non immersive, small FOVTracking and Sensing: high accuracy needed
Milgram’s Reality-Virtuality continuum
Mixed Reality
Real Augmented Augmented VirtualEnvironment Reality (AR) Virtuality (AV) Environment
Reality - Virtuality (RV) Continuum
Metaverse
Neal Stephenson’s “SnowCrash”The Metaverse is the convergence of: The Metaverse is the convergence of:
1) virtually enhanced physical reality 2) h ll l 2) physically persistent virtual space
Metaverse Roadmapphttp://metaverseroadmap.org/
Metaverse Dimensions• Augmentation technologies that layer information onto
our perception of the physical environment.• Simulation refers to technologies that model reality • Intimate technologies are focused inwardly, on the Intimate technologies are focused inwardly, on the
identity and actions of the individual or object;• External technologies are focused outwardly towards • External technologies are focused outwardly, towards
the world at large;
Metaverse Components
Four Key ComponentsVirtual WorldsAugmented RealityMirror WorldsMirror WorldsLifelogging
Mirror Worlds
Mirror worlds are informationally-enhanced virtual models of the physical world.p y
Google Earth, MS Street View, Google Maps
LifeLogging
Technologies record and report the intimate states and life histories of objects and usersj
Nokia LifeBlog, Nike+, FitBits
Steve Mann - LifeLogginggg g
Gordon Bell: LifeLogging
1 TB to store 65 years of data1 TB to store 65 years of data
Summary
Augmented Reality has three key featuresCombines Real and Virtual ImagesgInteractive in real-timeRegistered in 3DRegistered in 3D
AR can be classified alongside other technologiesMilgrams Mixed Reality continuumMetaVerse MetaVerse
AR HistoryAR History
A Brief History of AR (1)1960’ S h l d / S ll’ 1960’s: Sutherland / Sproull’s first HMD system was see-throughthrough
A Brief History of AR (2)
1960 - 70’s: US Air Force helmet mounted displays (T Furness)displays (T. Furness)
A Brief History of AR (3)
1970 - 80’s: US Air Force Super Cockpit (T. Furness)p p ( )
A Brief History of AR (4)
990’ “A ” Early 1990’s: Boeing coined the term “AR.” Wire harness assembly application begun (T. Caudell, D. Mizell). E l t id 1990’ UNC lt d i li ti j tEarly to mid 1990’s: UNC ultrasound visualization project
A Brief History of AR (5)
1994: Motion stabilized display [Azuma]1994: Motion stabilized display [Azuma]1995: Fiducial tracking in video see-through [Bajura / Neumann]1996: UNC hybrid magnetic-vision trackery g
A Brief History of AR (6)
1996: MIT Wearable Computing efforts1998: Dedicated conferences beginL 90’ C ll b i d i iLate 90’s: Collaboration, outdoor, interactionLate 90’s: Augmented sports broadcasts1998 - 2001: Mixed Reality Systems Lab1998 - 2001: Mixed Reality Systems Lab
History Summary
1960’s – 80’s: Early Experimentation1980’s – 90’s: Basic Research1980 s 90 s: Basic Research
Tracking, displays
1995 – 2005: Tools/ApplicationsInteraction, usability, theoryy y
2005 - : Commercial ApplicationsG M di l I dGames, Medical, Industry
Gartner Hype Cycle
2007 - AR Reaches Mainstream
MIT Technology ReviewMarch 2007list of the 10 most exciting technologiesexciting technologies
EconomistDec 6th 2007Dec 6th 2007Reality, only better
Gartner’s top 10 disruptive technologies 2008-2012:Multicore and hybrid processors Virtualisation and fabric computing S i l k d i l f Social networks and social software Cloud computing and cloud/Web platforms Web mash s Web mashups User Interface Ubiquitous computing Ubiquitous computing Contextual computing Augmented reality Augmented reality Semantics
2009 - AR in Magazines
Esquire MagazineDec 2009 issue12 pages AR content
M OthMany OthersWiredColorsRed BullRed BullEtc
Google Searches for AR
2008 - Browser Based ARFlash + camera + 3D graphicsHigh impactg p
High marketing value
Large potential install baseLarge potential install base1.6 Billion web users
Ease of developmentEase of developmentLots of developers, mature tools
L f Low cost of entryBrowser, web camera
Impact of Web-based AR
Boffswana Living Sasquatchhttp://www.boffswana.com/news/?p=605p p
In first month100K 100K unique visits 500K page views6 minutes on page
Mobile Phone ARMobile Phones
cameracameraprocessord ldisplay
AR on Mobile PhonesSimple graphicsOptimized computer visionOptimized computer visionCollaborative Interaction
AR Advertising (HIT Lab NZ 2007)
T d l d AR li i (200K)Txt message to download AR application (200K)See virtual content popping out of real paper advertTested May 2007 by Saatchi and SaatchiTested May 2007 by Saatchi and Saatchi
2008: Location Aware Phones
Nokia NavigatorMotorola Droid
2009 - Outdoor Information OverlayM bil h b dMobile phone basedTag real world locations g
GPS + Compass inputOverlay graphics data on live videoOverlay graphics data on live video
ApplicationsTravel guide, Advertising, etc
Wikitude, Layar, Junaio, etc.., y , J ,Android based, Public API released
Layar (www.layar.com)
Location based dataGPS + compass locationpMap + camera view
AR L l ldAR Layers on real worldCustomized dataAudio, 3D, 2D content
Easy authoringEasy authoringAndroid, iPhone
$784 million USD in 2014
Some Commercial AR Companies
ARToolworks (http://www artoolworks com/)ARToolworks (http://www.artoolworks.com/)ARToolKit, FLARToolKit, SDKs
Metaio (http://www.metaio.com/)Marketing, Industry, SDKsMarketing, Industry, SDKs
Total Immersion (http://www.t-immersion.com/)Marketing, Theme Parks, AR Experiences
Qualcomm Qua co (http://developer.qualcomm.com/dev/augmented-reality)
Mobile AR QCAR SDKMobile AR, QCAR SDK
FLARToolKit (ARToolworks)
Flash version of ARToolKitFLARToolKit + Papervison3D = AR on the webFLARToolKit + Papervison3D = AR on the webExample - GE Ecomagination Website
1.2 million views on YouTubehttp://ge.ecomagination.com/smartgrid/#/augmented_realityp g g g g y
Software Platform (Metaio)
Unifeye software platformModular platform for building AR applicationsp g pp
Markerless Tracking (Total Immersion)
D’Fusion markerless tracking platform Turn key solutiony
Example - Topps baseball cards AR h f l b b ll dAR characters on top of real baseball cards
Summary
Augmented Reality has a long history going back to the 1960’sInterest in AR has exploded over the last two years and is being commercialized quicklyyears and is being commercialized quicklyAR is growing in a number of areas
Mobile ARWeb based ARWeb based ARAdvertising experiences
Sample AR ApplicationsSample AR Applications
Applications
MedicineManufacturingInformation overlayInformation overlayArchitectureMuseumMarketingMarketingGaming
Applications: medical“X-ray vision” for surgeonsAid visualization, minimally-invasive operations. Aid visualization, minimally invasive operations. Training. MRI, CT data.
Ul d j UNC Ch l HillUltrasound project, UNC Chapel Hill.
CourtesyUNCChapelHill
Medical AR TrialsSauer et al. 2000 at Siemens Corporate Research, NJStereo video see throughStereo video see through
F. Sauer, Ali Khamene, S. Vogt: An Augmented Reality Navigation System with a Single-Camera Tracker: System Design and Needle Biopsy Phantom Trial,Single Camera Tracker: System Design and Needle Biopsy Phantom Trial,MICCAI 2002
Assembly and maintenance © 1996 S. Feiner, B. MacIntyre, & A. Webster, Columbia University
© 1993 S F i B M I t &© 1993 S. Feiner, B. MacIntyre, & D. Seligmann, Columbia University
PS3 - Eye of Judgment (2007)C t Vi i T kiComputer Vision TrackingCard based battle gameC ll b ti ARCollaborative AROctober 24th 2007
AR Books – Markerless Tracking
AR Jam story bookAR Jam story book
AR A t tiAR Annotations
Columbia University
© 1993 S. Feiner, B. MacIntyre, M. Haupt, & E. Solomon,C l bi U i i
© 1997 S. Feiner, B. MacIntyre, T. Höllerer, & A. Webster,
Columbia University, ,
Columbia University
HRL
Applications: Annotating Environment
P bli d i iPublic and private annotationsAid recognition, “extended memory”g , y
The World = Your User Interface (Feiner)
E E h Al
© 2002, Scientific American
Everyone, Everywhere, Always
Application: Broadcast Augmentation
Adding virtual content to live sports broadcasts“First down” line in American footballFirst down line in American footballHockey puck trails, virtual advertisementsNational flags in swimming lanes in 2000 OlympicsNational flags in swimming lanes in 2000 Olympics
Commercial applicationPrinceton Video Image is one company
- http://www.pvi.tv/
Broadcast Examples
Interactive Museum ExperiencesBlackMagic
Virtual America’s Cup410,000 people in six months
MagicPlanetTeManawa science museumVirtual AstronomyCollaborative AR experience
AR VolcanoInteractive AR kioskScienceworks museum, Melbourne
Museum Archeology
LifePlus (2002-2004) Natural feature trackinggVirtual charactersMobile AR systemMobile AR system
Archeoguide (2000-2002)Cultural heritage on-site guideHybrid trackingHybrid trackingVirtual overlay
SummaryAR h l b d d l d AR technology can be used to develop a wide range of applicationsPromising application areas include
GamesGamesEducationEngineeringEngineeringMedicineMMuseumsEtc..
AR Experience DesignAR Experience Design
“The product is no longer The product is no longer the basis of value The the basis of value. The
experience is ”experience is.
Venkat RamaswamyThe Future of Competition.
Gilmore + Pine: Experience EconomyGilmore Pine: Experience Economy
E tiexperiences Emotion
services
productsproducts
components Function
Sony CSL © 2004
The Value of Good User Experience
5050c
$3.50
20c
Good Experience DesignReactrixReactrix
Top down projectionCamera based inputCamera based inputReactive GraphicsNo instructionsNo instructionsNo training
Using the N-gage
SideTalkinghttp://www.sidetalkin.com
Interaction Design
“Designing interactive products to support people in their everyday and working lives”everyday and working lives
Preece, J., (2002). Interaction Design
Design of User Experience with TechnologyHigher in the value chain than product designHigher in the value chain than product design
Interaction Design involves answering three questions:What do you do? - How do you affect the world?What do you feel? – What do you sense of the world?What do you know? – What do you learn?
Interaction Design Process
Who are your Users?
Everyone!y
Designing for Everyone
Designing for Everyone pleases No one
Understanding Specific Needs
Interaction Design is All About You
Users should be involved throughoutinvolved throughout the Design ProcessConsider all the needs Co s de a t e eedsof the user
B ildi C lli AR E iBuilding Compelling AR Experiences
experiencesUsability
applications Interaction
tools Authoringtools Authoring
components Tracking, Display
Summary
In order to build AR applications you need to focus on the user experiencepGreat user experience is based on
L l l AR h lLow level AR component technologyAuthoring toolsApplication/Interaction designUser experience textingUser experience texting