At Google I/O

Sang Chul Kang (sckang@sk.com) Corporate R&D Center/VE Technology Lab

T-AR for Project Tango

1

Contents

• VR vs AR, Tech Trend

• T-AR Platform

• Google Project Tango

• T-AR for Project Tango

2

VR vs AR, Tech Trend

3

Virtual Reality

Augmented Reality

1862 Pepper’s Ghost

1962 Sensorama

by Morton Heilig

1968 Ultimate Display by

Ivan Sutherland

1987 “Virtual Reality” coine

d by Jaron Lanier

1993 Sega announces

VR HMD 2003 “Second Life” by Linden Lab

2014 Facebook acquired Oculus

1993 “Augmented Reality” coined by Tom Caudell

1993 NaviCam by Jun Rekimoto

1999 ARToolkit by Hirokazu Kato and Mark Bilingh

ust 1997 MARS by Steve Feiner

2009 Layar

2012 Google Glasses

2015 Microsoft HoloLens

2014 Google invests Magic Leap

2014 Project Morpheus

by SONY

1993 1862 1962 1968 1997 1999 2012 2014 2015 1987

VR & AR Brief History

4

Source: Gartner

Hype Cycle for Emerging Technology 2015

5

Technical Difficulty Comparison

Display

Rendering

Tracking

Interaction

Easy Difficult Difficult

Contents

Virtual Reality Augmented Reality

6

AR/VR Technology Acquisition by Big Players

D T C T

T T T T

D I T T I

7

VR & AR Market Trend

(Source: Digi-Capital)

(Source: KZERO)

8

Why AR?

Entertainments - Games & Media Daily Use – Eye Phone VS Virtual Reality Augmented Reality

9

T-AR Platform

10

T-AR Platform

Authoring

Contents/Service Server

Contents

The origin of the Statue of Liberty project is

sometimes traced to a comment made by French law professor and politician Édouard

René de Laboulaye in mid-1865. In after-

dinner conversation at his home near Versailles, Laboulaye, an ardent supporter of the Union

in the American Civil War, .. moreService

AR Contents

Education

Tourism Advertisement

Museum

Location POI

O2O

Publishing

AR Services

Real World Augmented World

Service

Creation

AR Browser AR running on Device

HTML5 Image 3D Animation

Audio Video Text

11

The AR Browsers in the Market

JSON Architect(JS) AREL(XML+JS)

Non Standard !!

12

T-AR Platform

ARML v2.0 International Standard (Feb. 2015)

AR Browser + Authoring Tool

& Server

Location-based AR

Vision-based AR Service

MWC 2014

13

AR Interoperability based on ARML

JSON

Architect(JS)

AREL(XML+JS)

CARIF(Common AR Interchange Format) Subset of ARML v2.0

14

AR Authoring

인식 대상 등록 (사진 촬영 or 위치 선택) 컨텐트 배치

저작 정보 저장 (채널 정보)

Database

(ARML, Targets, Contents)

서비스/컨텐츠 서버

Browser

Browser 컨텐트 선택

AR 채널 데이터 전송

AR 실행 채널 선택

ARML Repository

AR 채널 데이터 요청

15

ARML v2.0 (Vision-based AR)

<arml xmlns=“…” xmlns:xlink=“…” xmlns:gml=“…">

<ARElements>

<Feature id=“apple">

<anchors>

<anchorRef xlink:href="#myApple" />

</anchors>

</Feature>

<Tracker id="sktar">

<uri xlink:href="https://araon.sk.com/" />

</Tracker>

<Trackable id="myApple">

<config> <tracker xlink:href="#sktar" />

<src>http://www.mysite.com/appleimage.zip</src>

</config>

<assets>

<Video id="myAsset">

<href xlink:href="/content.mp4"/>

</Video>

</assets>

</Trackable>

</ARElements>

<script type="text/javascript">

<![CDATA[

try {

… java scripts here …

}catch(e){ loge("### Exception = " + e.message “); }

]]>

</script>

<style type="text/css">

<![CDATA[ … CSS style here … ]]>

</style>

</arml>

Assets : • contents used in augmentation, such as text, image, 3d models, video, audio.. Etc.

Style : • styling the virtual object in the scene and set the attributes of elements

Script : • describe the behavior of digital assets and AR elements.

Anchor : • defines the link between the digital and the physical world

Feature : • physical object in real world

ARElements : • contains AR elements

Tracker : • tracks the trackable • ex) image tracking engine, motion tracking engine

Trackable : • the trigger that should be tracked in the scene • ex) image, motion, 3d space.. Etc (target)

Config : • set the tracking engine & the digital information of trackable for it

16

T-AR for Smartphone & HMD

T-AR Browser Engine was extended to support Stereoscopic Rendering for both AR & VR. Currently it supports Google Cardboard.

Mobile AR

17

AR Browser Engine Architecture

AR Browser Engine APIs (SDK)

T-AR Browser (or Application)

Authoring I/F

Tracker Manager

(Pluggable)

Location Manager

JavaScript Engine (V8)

CSS Handler

Sensor Manager

Android System

Audio

Html5 Viewer

(WebView)

Resource Manager

AR Renderer

3D Rendering Engine

Ogre 1.9

COLLADA Converter

Ogre Assimp

Video

Object Manager

AR Manager

Event Handler ARML DOM Manager

OpenGL ES

OpenSL ES

Free Type

zlib zzip

Free Image

ARML Parser

TinyXML2

TinyXPath

SKTAR 2D Tracking

Engine

Cloud-based Recognition

(TBD)

3D Tracking (TBD)

Rendering ARML Handling Tracking Reference: Reference:

<arml xmlns=“…” xmlns:xlink=“…” xmlns:gml=“…">

<ARElements>

<Feature id=“apple">

<anchors>

<anchorRef xlink:href="#myApple" />

</anchors>

</Feature>

<Tracker id="sktar">

<uri xlink:href="https://araon.sk.com/" />

</Tracker>

<Trackable id="myApple">

<config> <tracker xlink:href="#sktar" />

<src>http://www.mysite.com/appleimage.zip</src>

</config>

<assets>

<Video id="myAsset">

<href xlink:href="/content.mp4"/>

</Video>

</assets>

</Trackable>

</ARElements>

<script type="text/javascript">

<![CDATA[

try {

… java scripts here …

}catch(e){ loge("### Exception = " + e.message “); }

]]>

</script>

<style type="text/css">

<![CDATA[ … CSS style here … ]]>

</style>

</arml>

Locations Images

+

+

ARML

Targets

Contents

+

SKT AR Browser supports only Android devices. ICS(4.0), JB(4.1~4.3), KK(4.4) or above

Mode (+AVR, VR)

Stereo Rendering

18

Google Project Tango

19

Project Tango – H/W

Integrated

Depth Sensing

Motion Tracking

Camera

4MP 2um Pixel

Camera

NVIDIA Tegra K1

128GB SSD

4GB RAM

Intel RealSense Tango Phone, IDF 2015

Qualcomm Tango Phone, Google I/O 2015

20

Project Tango – S/W

Motion Tracking Depth Perception Area Learning

Core Technologies

21

Project Tango – Motion Tracking

Motion tracking allows a device to understand position

and orientation using Project Tango's custom sensors.

This gives you real-time information about the 3D motion

of a device.

22

Project Tango – Area Learning

Project Tango devices can use visual cues to help

recognize the world around them. They can self-correct

errors in motion tracking and re-localize in areas they've

seen before.

23

Project Tango – Depth Perception

Depth sensors can tell you the shape of the world

around you. Understanding depth lets your virtual world

interact with the real world in new ways.

24

T-AR for Project Tango

At Google I/O

25

T-AR for Tango, Service Concept

Target Registration Area Learning & Depth Perception

Authoring AR Scene Create AR Service

Enjoy AR with Tango (Museum, Exhibition.. Etc)

T-AR, AR Service Server

T-AR Browser for Tango T-AR Browser for Tango

Easy to Create Easy to Use

26

T-AR at Google I/O

27

Authoring & Viewing AR Concept

AR Service Authoring

공간학습

(Area Learning)

공간모델 생성(Point Cloud)

컨텐츠 선택

컨텐츠 배치

ARML&

Package

Generation

채널 패키지 업로드,

Database

Execute AR Service

Package Transfer

(ARML, ADF, Contents, Point Cloud)

1. ARML Load/Parse

2. ADF Import/Load

3. Space Scan

4. Motion Tracking with ADF

5. Augment Contents

채널 선택

서비스/컨텐츠

서버

Browser

Browser

Area Description &

Point Cloud

AR 실행

28

AR Browser Engine Architecture

AR Browser Engine APIs (SDK)

T-AR Browser for Tango

Authoring I/F

Tracker Manager

(Pluggable)

Location Manager

JavaScript Engine (V8)

CSS Handler

Sensor Manager

Android System

Audio

Html5 Viewer

(WebView)

Resource Manager

AR Renderer

3D Rendering Engine

Ogre 1.9

COLLADA Converter

Ogre Assimp

Video

Object Manager

AR Manager

Event Handler ARML DOM Manager

OpenGL ES

OpenSL ES

Free Type

zlib zzip

Free Image

ARML Parser

TinyXML2

TinyXPath

SKTAR 2D Tracking

Engine

Cloud-based Recognition

(TBD)

3D Tracking (TBD)

Rendering ARML Handling Tracking Reference: Reference:

<arml xmlns=“…” xmlns:xlink=“…” xmlns:gml=“…">

<ARElements>

<Feature id=“apple">

<anchors>

<anchorRef xlink:href="#myApple" />

</anchors>

</Feature>

<Tracker id="sktar">

<uri xlink:href="https://araon.sk.com/" />

</Tracker>

<Trackable id="myApple">

<config> <tracker xlink:href="#sktar" />

<src>http://www.mysite.com/appleimage.zip</src>

</config>

<assets>

<Video id="myAsset">

<href xlink:href="/content.mp4"/>

</Video>

</assets>

</Trackable>

</ARElements>

<script type="text/javascript">

<![CDATA[

try {

… java scripts here …

}catch(e){ loge("### Exception = " + e.message “); }

]]>

</script>

<style type="text/css">

<![CDATA[ … CSS style here … ]]>

</style>

</arml>

Locations Images

+

+

ARML

Targets

Contents

+

SKT AR Browser supports only Android devices. ICS(4.0), JB(4.1~4.3), KK(4.4) or above

Motion Tracking

Area Learning Depth Perception AR Authoring (+Edit)

Mode (+Tango)

Tango Core Tango SDK

Stereo Rendering

29

T-AR for Tango in Action

30

Dive Support

31

ARML v2.0 for Tango (Location-based AR)

<arml>

<ARElements ar:runmode="tango" ar:adfuuid="c7f5aac9-edae-4866-bfee-48549a145277">

<Feature id="tango_sample">

<anchors>

<anchorRef xlink:href="#_anchor_geo_001"/>

</anchors>

</Feature>

<Geometry id="_anchor_geo_001">

<assets>

<Model id="_model_001">

<href xlink:href="/fbf0dddce7883d8f3ec6a902e83622fc4308c09e.zip"/>

<orientationMode>absolute</orientationMode>

<orientation>

<roll>0.000</roll>

<tilt>-90.000</tilt>

<heading>0.000</heading>

</orientation>

<scale>

<x>0.721</x>

<y>0.721</y>

<z>0.721</z>

</scale>

</Model>

</assets>

<gml:Point gml:id="_p_001" gml:srsDimension="3">

<gml:pos>5.377 -1.453 -1.615</gml:pos>

</gml:Point>

</Geometry>

</ARElements>

</arml>

32

Q&A