CONNECTING PHYSICAL AND DIGITAL WORLDSMARIUS PREDA, MPEG 3DG CHAIR, INSTITUT MINES TELECOM

www.slideshare.net/MariusPreda

From centuries, we are building two different types of worlds

PhysicalInformational

Immersion, from a human user point of view, is making abstraction of the Physical world (remove all possible connection to it) and have stimuli only from the Informational one

Immersion, presence, stimuli, perception, …

stories

music

films

knowledge

In which world our users are?

The stimuli from physical worlds are familiar or at very low intensities

We intentionally block our interfaces with the physical world

Improving immersion feeling

Strategies for blocking the interfaces with the Physical world

Improving immersion feeling

Strategies for extending the space of the Informational world stimuli

Improving immersion feeling

How far we’ve gone with this strategy?

Relatively far …

Improving immersion feeling

Probably too far …

Improving immersion feelingHow far?

It’s the time to connect our worlds

Why now?

• For the first time, smartphones enable “everyone” to access Informational world while sensing also the Physical world

• They change our view of the environment• Enhance the rich history

and meaningfulness ofthe real world withpowerful digitalelements

Physical & Informational Augment The Reality

An invitation: become architects of a new kind of bridges

In AR, almost everything remains to be invented

Las Vegas

This place some years ago

In AR, almost everything remains to be invented

• Marketing• Tourism• Culture• Education• Serious and less serious games• …

• Hardware manufacturers• Software vendors

In AR, almost everything remains to be invented

We (and few others) already started and invite you to be part of

The AR Revolution

In AR, almost everything remains to be invented

Tidy City

Portal Hunt

Elements

ARQuiz

Augmented Books

Summer School games

TECHNOLOGY INSIGHTSMPEG-V AND MPEG ARAF

MPEG technologies related to AR

MPEG-1/2(AV content)

1992/4

VRML

1997

• Part 11 - BIFS: -Binarisation of VRML -Extensions for streaming -Extensions for server command -Extensions for 2D graphics - Real time augmentation with audio & video• Part 2 - Visual: - 3D Mesh compression - Face animation

1998

• Part 2 – Visual - Body animation

1999

MPEG-4 v.1

MPEG-4 v.2

First form of broadcast signal augmentation

MPEG technologies related to AR

MPEG-4

2003

•AFX 2nd Edition: - Animation by morphing - Multi-texturing

2005

• AFX 3rd Edition - WSS for terrain and cities - Frame based animation

2007

MPEG-4

MPEG-4

• Part 16 - AFX: - A rich set of 3D graphics tools - Compression of geometry, appearance, animation

• AFX 4th Edition - Scalable complexity mesh coding

2011

MPEG-4A rich set of 3D Graphics representation and compression tools

MPEG technologies related to AR

MPEG-4

2003

•AFX 2nd Edition: - Animation by morphing - Multi-texturing

2005

• AFX 3rd Edition - WSS for terrain and cities - Frame based animation

2007

MPEG-4

MPEG-4

• Part 16 - AFX: - A rich set of 3D graphics tools - Compression of geometry, appearance, animation

• AFX 4th Edition - Scalable complexity mesh coding

2011

MPEG-4

2009

• Part 25 - Compression of third-party XML (X3D, COLLADA)

MPEG-4

2004

• Part 16 - X3D Interactive Profile

MPEG-4

MPEG technologies related to AR

MPEG-V - Media Context and Control

2011

• 2nd Edition: - GPS - Biosensors - 3D Camera

201x

• Compression of video + depth

201x

MPEG-V

- 3D Video

• 1st Edition - Sensors and actuators - Interoperability between Virtual Worlds

• Feature-point based descriptors for image recognition

201x

CDVS

MPEG-U – Advanced User Interface

2012

A rich set of sensors and actuators

- 3D Audio

MPEG-H

MPEG technologies related to ARMPEG-V – Media Context and Control

ActuatorsLight Flash Heating Cooling Wind Vibration Sprayer Scent Fog Color correction Initialize color correction parameter Rigid body motion Tactile Kinesthetic

Global position command

SensorsLight Ambient noise Temperature Humidity Distance Atmospheric pressure Position Velocity Acceleration Orientation Angular velocity Angular acceleration Force Torque Pressure Motion Intelligent camera typeMulti Interaction point Gaze tracking Wind

Dust Body height Body weight Body temperature Body fat Blood type Blood pressure Blood sugar Blood oxygen Heart rate

Electrograph EEG , ECG, EMG, EOG , GSR Weather Facial expression Facial morphology Facial expression characteristics Geomagnetic

Global position Altitude Bend Gas

MPEG technologies related to ARMPEG-V – Media Context and Control

•All AR-related data is available from MPEG standards•Real time composition of synthetic and natural objects•Access to

• Remotely/locally stored BIFS/compressed 2D/3D mesh objects • Streamed real-time BIFS/compressed 2D/3D mesh objects

•Inherent object scalability (e.g. for streaming)•User interaction & server generated scene changes•Physical context

• Captured by a broad range of standard sensors• Affected by a broad range of standard actuators

Main features of MPEG AR technologies

MPEG vision on AR: the MPEG AR Browser

Point to a URL – no need to download new applications for each contextThe browser• Retrieves scenario from the internet• Starts video acquisition• Tracks objects• Recognizes objects from visual signatures• Recovers camera pose• Gets streamed 3D graphics• Composes new scenes• Gets inputs from various sensors• Offers optimal AR experience by constantly adapting interaction possibilities and

objects from a remote server.

Industry • Maximize number of customers through MPEG-compliant authoring tools and

browsers• No need to develop a new application for each use case and device platform

MPEG vision on AR

MPEG-4 + MPEG-7 + MPEG-21 ++ MPEG-U+ MPEG-V

= ARAFMPEG Player

CompressionAuthoring Tool

Produce

Download

Technology insights

Modeling the content chain– Creation– Consumption– The pivot : a representation formalism

Technology insights: the chain

ARAF Browser

MediaServers

ServiceServers

User

LocalSensors & Actuators

RemoteSensors & Actuators

MPEG ARAF

Local Real World

Environment

Local Real World

EnvironmentRemote

Real World Environment

Remote Real World

Environment

Auth. Tools

Technology insights: Infrastructure

ARAF Browser

MediaServers

ServiceServers

User

LocalSensors & Actuators

RemoteSensors & Actuators

MPEG ARAF

Local Real World

Environment

Local Real World

EnvironmentRemote

Real World Environment

Remote Real World

Environment

Auth. Tools

The ARAF browserARAF Browser

• Full scene graph representation (2D and 3D)• Static and dynamic objects and scenes• Programmable behavior (JS)• Access to sensors and cameras• Access to remote resources and services (http

request and web sockets)• Support for user interaction• Support for compressed resources

The ARAF browserARAF Browser

ARAF Browser

JavaScript Engine

MPEG Totem Services

WebSockets Server

Communication service

WebSockets Client

ARAF Browser

SC3DMC Decoder

BBA Decoder

ARAF File

BIFS Stream

SC3DMC Stream

BBA Stream

BIFS DecoderMPEG-4 Scene

IndexedFaceSet

SBVCAnimation

Camera

Image data

Reference Images

Image data

Image data

Image data

AFAR Browser

Reference Signal

Detection

Detected Images

MyMultimediaWorld.com

• Convert media assets in a common format• Describe, index, organize, search digital assets• Compressed assets, ready for transmission

A multimedia repository and processing platform supporting both creation and consumption Media

& ServiceServers

Technology insights: the formalism

ARAF Player

MediaServers

ServiceServers

User

LocalSensors & Actuators

RemoteSensors & Actuators

MPEG ARAF

Local Real World

Environment

Local Real World

EnvironmentRemote

Real World Environment

Remote Real World

Environment

Auth. Tools

ARAF Description Language

ARAF : an XML representation

Player

CompressionAuthoring Tool

Produce

Download

ARAF

A set of scene graph nodes defined in MPEG-4• Audio, image, video, graphics, programming, communication, user

interactivity, animation

• Map, MapMarker, Overlay, ReferenceSignal, ReferenceSignalLocation, CameraCalibration, AugmentedRegion

Connection to sensors defined in MPEG-V• Orientation, Position, Angular Velocity, Acceleration, GPS, Geomagnetic,

Altitude, Local camera(s)

Compressed media• Image, sound, video, 2D/3D graphics

Augmented Reality Application Format

ARAF

Technology insights: creation

ARAF Browser

MediaServers

ServiceServers

User

LocalSensors & Actuators

RemoteSensors & Actuators

MPEG ARAF

Local Real World

Environment

Local Real World

EnvironmentRemote

Real World Environment

Remote Real World

Environment

Auth. Tools

Authoring ARAF

• Specific tools• Generic tools

– TOTEM.Designer– TOTEM.Scout

Auth. Tools

Specific Authoring Tools

• Scenario driven approach– Tidy City, Portal Hunt, ARQuiz, AR Book, Treasure

Hunt

Auth. Tools

Towards a generic Designer

• Create templates for app. objects• Fill structure with data• Create GUI• Create application logic • Export data

On site designer

Conclusions

• AR – the next digital revolution• For making revolutions, one needs weapons:

– MPEG technologies represent an efficient and powerful set of weapons

• Current MPEG activities– Improve and enrich the weapons– Package them in consistent and easy to use tools