© Copyright Khronos Group, 2012 - Page 1€¦ · Breakthrough apps embrace mobility’s strengths – not just treat phones as small PCs - will need complex, interoperating APIs

© Copyright Khronos Group, 2012 - Page 1


Mobile AR Forum Erik Noreke

Khronos Working Group Chair


Agenda for Today • Augmented Reality, Open Standards and Khronos

- Neil Trevett - Khronos President, VP of Mobile Content, NVIDIA

• PowerVR GPU: Accelerated Augmented Reality

- Kristof Beets - Business Development Manager, Imagination Technologies

• From AR-Capable to AR-Optimized Mobile Platforms

- Enrica Filippi - Platform Ecosystem Marketing Head, ST-Ericsson

• Enhancing the mobile AR experience with ARM

- Sri Kannan Iyer - AR Ecosystem Manager, ARM

• Taking AR to New Dimensions

- Steve Wainwright - VP of Sales and Marketing, GM for Europe, Middle East and

Africa, Freescale

• Panel Discussion – Get your questions ready!

- Christine Perey - Founder & Chairperson, International AR Standards Community


Augmented Reality, Open Standards and Khronos

Neil Trevett - Khronos President VP of Mobile Content, NVIDIA


Inventing the Future

Gartner Hype Cycle – 2012 Edition

"The best way to predict the future is to invent it.“

Alan Kay

This Forum is to explore how the silicon community is putting the processing power and standards in place to create an AR market

opportunity MUCH sooner


AR = Input AND Output Processing • Augmented Reality is uniquely challenging

• Needs sophisticated IN AND OUTPUT processing - in perfect harmony

Generating 3D Graphical

Augmentations

Sensing and tracking the context and the

scene around the user

Detecting user gestures and actions

Compositing real and synthetic elements

to delight and inform

Augmented Reality Application


Brief History of Mobile AR so far..

GPS and Compass

Planar Tracking 3D Tracking and Reconstruction

Layar

Qualcomm

Metaio


Visual-based Augmented Reality

Camera video stream sent to the compositor

3D Augmentation Rendering

3D augmentations composited with video stream

Camera Tracking

Camera images used to track the

camera’s location and

orientation

Camera-to-scene transform locks the 3D rendering to the real world


Khronos - Connecting Software to Silicon • Creating open, royalty-free API standards

- Focus on graphics, dynamic media, compute and sensor hardware

• Low-level - just above raw silicon

- “Foundation” functionality needed on every platform

• Safe forum for industry cooperation

- ‘By the industry for the industry’

- Open to any company to join

- IP framework to protect

members and industry

• The right APIs at the right time

enable efficient hardware acceleration

= 10X performance/power

APIs enable software developers to turn silicon

functionality into rich end user experiences


Board of Promoters

Over 100 members – any company worldwide is welcome to join

Apple

http://www.amd.com/

http://www.antixlabs.com/

http://www.altsoftware.com/

http://www.google.com/

http://www.hu1.com/

http://www.fujitsu.com/

http://global.mitsubishielectric.com/

http://www.nds.com/

http://www.s3graphics.com/

http://www.st.com/

http://www.sasken.com/

http://www.toshiba.com/

http://www.vivantecorp.com/

http://www.marvell.com/index.jsp

http://www.umu.se/umu/index_eng.html

http://www.gshark.com/


Khronos API Ecosystem Trends

As platforms diversify – mobile,

TV, embedded – HTML5 will

become increasingly important

as a universal app platform

Breakthrough apps embrace

mobility’s strengths – not just treat

phones as small PCs - will need

complex, interoperating APIs

e.g. Augmented Reality

Mobile is the new platform for apps

innovation. Mobile APIs are needed

to unlock hardware potential while

conserving battery life

High-end API technology is created on

high-end platforms


OpenGL ES Adoption • OpenGL for embedded and mobile devices

- Eliminates redundant and legacy features

- Adds extensions to make it mobile-friendly

• The dominant 3D API for mobile devices

- Widely adopted for STB, DTV, automotive,…

- Hundreds and hundreds of millions shipped

• OpenGL ES 2.0

- Fully programmable pipeline

• Runs high-end content and engines

- UE3, Unity, Unigine, Rage

© Copyright Khronos Group, 2012 - Page 13 Copyright Khronos 2009

OpenCL – Heterogeneous Computing • Framework for programming diverse

parallel computing resources in a system

• Platform Layer API

- Query, select and initialize compute devices

• Kernel Language Specification

- Subset of ISO C99 with language extensions

• Runtime API

- Execute compute kernels – gather results

• OpenCL has Embedded profile

- No need for a separate “ES” spec


OpenMAX AL • Enables key image, camera and video use cases

- Allows optimal hardware acceleration with app portability

• Create Media Objects to process images and video with AV sync - Connect to variety of input and output objects to PLAY and RECORD media

Advanced image capture and photography

HD content playback with robust DRM

HD video teleconferencing

Augmented Reality

OpenMAX AL Media Object

URI

Memory

Camera

Audio Input

URI

Memory

Audio Mix

Display Window DSrc DSnk

Analog Radio

Sources Sinks

EGLStream to ES Data Tap to CPU


OpenSL ES – Advanced Audio • OpenSL ES does for audio what OpenGL ES does for graphics

- Advanced audio functionality from simple playback to 3D audio

• Object-based native audio API for simplicity and high performance

- Reduces development time

• Same API regardless of underlying implementation

- Software or hardware accelerated

• Cross OS portability

- Preserves application investment


?

EGLStream – Video/Graphics Interop

Camera

File

URL Etc.

OpenMAX AL

MEDIA PLAYER

Object

OpenGL ES

GL_TEXTURE_EXTERNAL

OpenMAX AL Media Player

is the EGLStream

“Producer” and controls

production of frames.

OpenGL ES GL_TEXTURE_EXTERNAL

is the EGLStream “Consumer” and

converts video format into RGB

OpenGL ES texture

EGLStreams enables and hides details of

video frame transport. Enables multiple

buffering modes for different uses cases

eg: FIFO and explicit latch/release

EGLStream


StreamInput Connects Sensors to Apps

Advanced Sensors Everywhere Standard cameras, depth cameras

motion and position, touch, microphones wireless controllers

Apps Need Sophisticated Access to Sensor Data

Without coding to specific systems or sensor hardware

Universal Timestamps

Apps request semantic sensor information StreamInput defines list of possible semantic requests

“Am I in an elevator?” “Give me gestures and face position”

Sensor graph created to provide sensor information StreamInput defines graph creation API and node interconnects

Low-level sensor processing encapsulated in nodes – unleashes fusion innovation Apps gain ‘magical’ situational awareness

Standardized Node Intercommunication

Input Device

Input Device

Input Device

Filter Node

Filter Node

App Filter Node

SHOULD NOT FORCE APPLICATIONS TO

ACCESS INDIVIDUAL SENSORS


Current StreamInput Participants • Aiming for specification release in 2012


Khronos Computer Vision Working Group • Vision Hardware Acceleration Layer

- Enable hardware vendors to implement

accelerated imaging and vision algorithms

• CV HAL* can be used by high-level

libraries or applications directly

- Primary focus on enabling mobile and

embedded systems

• OpenCV is widely used open source

library for vision projects

- Future version will leverage CV HAL

- * CV Hal is placeholder name

OpenCV open source library

Application

CV HAL

Open source sample implementation?

Hardware vendor implementations

Other higher-level CV libraries


Augmented Reality Functionality

Camera Processing

3D Rendering and Video Composition

Audio Rendering

Application on CPU

Positional and GPS Sensor Data

Computer Vision and Tracking

Position

and Tracking

Semantics

Control Camera, Preprocess and generate

video streams

Video TAP to CPU

Synchronization and sensor

fusion

Video stream to GPU

Positional Sensors

Camera EGLStream

Much more flexibility than just “overlay augmentations

over background”

CV HAL


Android Native API Adoption

OpenGL ES OpenGL ES 2.0

Shipping - Android 2.2

OpenSL ES OpenSL ES 1.0

Shipping – Android 2.3

OpenMAX AL OpenMAX AL 1.0

Shipping - Android 4.0

EGL EGL 1.4

Shipping under SDK -> NDK

OpenCL Not yet adopted

StreamInput Working group will ship spec in 2012

CV HAL Working group will ship spec in 2012


Innovating with Native APIs on Android • OEMs can add new OPEN STANDARD APIs to Android NDK

- Before Google platform adoption

- Do not break/change existing Google APIs

- Evangelize Google to adopt proven, standard APIs into standard platform

• Extended APIs can be used by:

- Bundled apps, Market apps with API selection

- Multiple APKs behind single multi-APK SKU

Standard

Android SDK

and NDK

Extended NDK for

Key use cases

Evangelize Google and ISVs to

adopt enabling APIs

E.g. video, camera, imaging,

sensor and composition APIs


HTML5 – Cross OS App Platform • Increasing diversity of devices

creates a demand for a true cross OS

programming platform

• BUT need more than “more HTML”

Rich Experiential Processing Multi-core CPUs

Rich 2D and 3D GPU GPU Computing

Multiple HD cameras Image and vision processing

Video encode/decode Audio encode/decode

Inertial and positional sensors

How can the Browser rapidly assimilate such diverse functionality?

Traditional Web-content


Leveraging Native API Investment into HTML5 • HTML5 evolving into cross-platform programming platform

- Gradually exposing complete system capabilities

• Opportunity to synergize Web and native APIs development

- Leverage native API investments, reduce developer learning cycles

• Khronos and W3C creating close liaison

Native APIs shipping or working group underway

JavaScript API shipping or working group underway

WebVL? Vision

Processing

WebAudio Advanced JavaScript

Audio

WebMAX? Camera

control and video

processing

Possible future JavaScript APIs

Device and Sensor APIs

Device Orientation

Working Groups

Native

JavaScript

CV HAL


Hype Cycle -> Adoption Timeline

Industrial Beachhead applications becoming

possible today

Wearable displays will ignite the consumer market


In Summary • AR is coming – and its going to change how we interact with everything

• The silicon community through Khronos is putting the technology building

blocks in place right now – in an SOC near you!

• Industrial applications will be the beachhead – becoming possible today

• Enormous scope for innovation and differentiation in consumer markets

over next few years

• Khronos is driving open standards for hardware acceleration

Participate, change the industry AND get the inside edge for your products!

Connecting Software to Silicon


Mobile AR Forum Panel

Moderator – Christine Perey Founder & Chairperson, International AR Standards Community Neil Trevett, Khronos President, VP of Mobile Content, NVIDIA Enrica Filippi Platform Ecosystem Marketing Head, ST-Ericsson Sri Kannan Iyer, AR Ecosystem Manager, ARM Steve Wainwright, VP of Sales and Marketing, GM for Europe, Middle East and Africa, Freescale


Augmented Reality at MWC 2012

Hardware • Imagination – Hall 1 D45

• ST Ericsson – Hall 7 D45

• ARM – Hall 1 C01

• Freescale – AV27

• Qualcomm - Hall 8

• Hewlett Packard – Hall 7

• Nokia/NAVTEQ - Hall 7

• Texas Instruments – Hall 8

• Intel – Hall 8

Software • VTT Hall 2

• mCRUMBS and metaio Hall 2

• HealthAlert App Hall 2

• Augmented Reality Lab - Hall 2

• Blippar Avenue - AV35

• BRGR Media - Hall 2

• Pordiva - Hall 2

• wöwbile Mobile Marketing Hall 7

Documents

© Copyright Khronos Group, 2012 - Page 1€¦ · Breakthrough apps embrace mobility’s strengths – not just treat phones as small PCs - will need complex, interoperating APIs