A Real-time Collaboration-enabled Mobile Augmented Reality System with Semantic Multimedia

Lehrstuhl Informatik 5(Information Systems)

Prof. Dr. M. JarkeI5-KAKl-1012-1 This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

A Real-time Collaboration-enabled Mobile Augmented Reality System with Semantic

Multimedia

Dejan Kovachev, Gökhan Aksakali & Ralf KlammaRWTH Aachen University, Germany

Advanced Community Information Systems (ACIS)[email protected]

http://creativecommons.org/licenses/by-sa/3.0/





http://dbis.rwth-aachen.de/cms/research/MiCIS


http://www.umic.rwth-aachen.de/


Prof. Dr. M. JarkeI5-KAKl-1012-2

Responsive Open

Community Information

Systems

Community Visualization

and Simulation

Community Analytics

Community Support

Web A

nalytics

Web

Eng

inee

ring

Advanced Community Information Systems (ACIS)

Requirements Engineering






Agenda

Motivation and scenarios Conceptual approach System design and implementation Evaluation Related Work Conclusions and Future Work






Motivating Scenario Documentation of Ghazni Wall






Digital Documentation in Cultural Heritage

Ghazni, Afghanistan, capital of Islamic Culture 2013

http://arch.rwth-aachen.de/cms/Architektur/Forschung/Forschungsprojekte/Cultural_Heritage/~cqdn/Ghazni_2013/lidx/1/






Local Workforce

On site Architecture Expert

Remote expert (Historian)

4-annotate multimedia

5-push updates

Metadata RepositoryMPEG-7,

CIDOC CRM

5-push updates

5-push updates

On-site DocumentationExpert

1- images and videos

CollaborativeMultimedia Cloud

3- store in cloud

2- tag basic metadata

7-push updates

7-push updates

6-correct annotation7-push updates

8-view repository

Workflow of Documentation






Ancient Helmet

Captured timeTaken locationCreation time

Provenance….

Metadata

Increase Awareness of Cultural Heritage

Multimedia overlays on video stream

Interaction via position and 3D movement

Still missing– real-time collab.– semantic

metadata






XMPP-base Mobile Multimedia Collaboration (XMMC)

Prototype: XMMC Mobile real-time multimedia collaboration system with

- integrated AR multimedia browser - standardized metadata for semantic multimedia and

annotations (MPEG-7)- Coordination, field note taking, instant geo-tagged multimedia

acquisition and sharing

Goals: How to enable Collaborative digital documentation of historical

sites with rich multimedia and metadata using inexpensive hardware

Increase of cultural heritage awareness






XMMC Use Cases

Acquire multimedia

Annotate multimedia

BrowseAR

Select a POIConsumer

<<extend>>

Display the details

Synchronize with other clients

Sharemultimedia

Content<<include>>

<<include>>

<<include>>

Call LAS MPEG-7 services

<<extend>>

<<include>>

Add POI

<<include>>

<<include>>

Producer

Collaborator

<<include>>

Chat






Concurrent Editing of XML

Most metadata standards use XML format– Used by standards like MPEG-7, CIDOC CRM, SVG

XML suited for concurrent editing of metadata– Hierarchal structure more efficient then linear

Concurrency Problems– Divergence, Causality Violation, Intention Violation

Approach Architecture Algorithms ExamplesPessimistic centralized locking, turn taking SVN,CVS

Optimistic P2P OT, treeOPT, CMAX [Gerlicher 2007]

Google Operational Wave Transformation, CEFX+






Consistency Maintenance Algorithm for XML (CMAX) [Gerlicher 2007]

Out-of-the-box lightweight concurrent editing of XML files– CEFX+ [Voigt 2009] based on CMAX

Optimistic approach mainly inspired from OT– First execute locally then propagate

Direct addressing with <cefx-uid> Concurrent editing of the same node

– Delegates conflict resolution to user State Vector

– Operation count for every participant History Buffer

– Previous operations






eXtensible Messaging and Presence Protocol (XMPP)

Classic SOAs like HTTP, SOAP are inadequate for real-time purposes

Good fit for real-time communication Easily customizable & extendable

– Open XML-based standard Existing extensions

– File transfer, multi-user chat, PubSub Library and server support

– Smack(Java), Openfire






XMMC Architecture

Client side Server Side

XmmC Server

Client Side

XMPP Server (Openfire)

Collab.Editing Service

LAS-MPEG7

Integration Service

AR Service

Media Store

Service

XMPP

Modules

XMPP Client(aSmack)

XmmC Client

Media Catalog Service

XMPP Connection Layer (XCL)

MultimediaModule

WLAN GPS Compass ACCELCamera

Relational DB

…

Collab.Annotation

Module

ARModule

CEFX+

MPEG-7 Services

previously availablemodifiedimplemented

Metadata Annotation

AR Browser

Media Acquisition

XMPP Client (Smack)

XMPP

XMPP






Technical Contributions Adaptation of CEFX+ for multimedia annotation

– New operations: editing and deleting of nodes– Later join, leave collaboration session

XMPP-based AR browser by extending Mixare [http://www.mixare.org/]– POI awareness, ARML-based, mobile content provider

Custom XMPP IQ stanzas for multimedia sharing and AR Integration with MPEG-7 metadata services

http://www.mixare.org/






Collaborative Annotation

Mobile Client-1

<jabber:iq:rpc> joinSession()

<jabber:iq:rpc> SessionData

Collaborative Annotation

Module

Collaborative EditingService

joinSession()

SessionData

MPEG-7 Integration

Service

…

Mobile Client-2

<jabber:iq:rpc> joinSession()

<jabber:iq:rpc> DocumentData

<groupchat> Message (Operation) executeOperation()updateSemantic BaseTypes()

Ack

Mpeg-7 Services

SemanaticbasetypeService()

Ack

<jabber:iq:rpc> loadDocument()

<jabber:iq:rpc> DocumentData

loadDocument()

DocumentData

<groupchat>Message (Operation)

<jabber:iq:rpc> leaveSession()<jabber:iq:rpc> leaveSession()

…

<message id="8Dpbh-26" to="[email protected]/Smack" from="[email protected]/test_1" type="groupchat"> <body> <basetype type="time" cefx:uid="101"> <name cefx:uid="201"> Second World War</name> <date cefx:uid="301">

1997-09-24T00:00:00:000F1000+01:00 </date> </basetype> </body> <x xmlns="jabber:x:cefx#ins" p="100" ci="1" sv="0,1" ba="1" fn="null"/></message>






XMPP-based Augmented Reality Browser

Contemporary mobile AR Browsers

XmmC

GPS data

Provider Server

Get POIs

DataSOAP/RESTful

3rd PartyContent Provider

Create POI

Wikipedia, Flicker,

Facebook, etc.

Get POIs

GPS dataXMPP Server

XmmC Server

Get POIs Get POIs

XMPP XMPP

Create POI Create POI

PubSub Node Data

Edit POI Metadata Edit POI Metadata

Publish EventEvent Message











Technical Evaluation

Collaborative Annotation Remote operation execution

– Average = 412ms– Stan. Derivation = 209ms

Conflict resolution– Resolved the possible conflicts

– with 900ms latency and1,2 KB/s bandwidth network– Still has open issues regarding undo and redo operations

AR Browser The positions of closed POIs are not very accurate due to

- Inaccuracy in GPS signal, compass, accelerometer- Presence of nearby metal objects causes inaccurate results






Technical Evaluation

AR browser consumes 4 times more battery than collaborative annotation

Energy consumption XMPP

– Generates verbose XML streams– Requires constant open sockets– Community works for

improvement for mobile networks Distribution of Stanza Types






User Evaluation

Field study with 7 participants– various mobile devices, Android versions and screen sizes

Tasks– acquisition of multimedia artifacts in a historical site– editing basic metadata– collaborative annotation– browsing with AR

Questionnaire – collect subjective qualitative measures and user experience






Evaluation Session Results

Majority of the participants observed real-time updates on the annotations (85%)

Participants mostly satisfied with the user interface and found it user-friendly and responsive

MPEG-7metadata service integration was successful The prototype increased cultural heritage awareness of the participants






Related Work

System MobilityBi-directional

Comm.Mobile AR

BrowserLight-weight

Real-time Collaborative

Editing

Multimedia Metadata

Mobilis(TU Dresden, 2009)

++ ++ - - ++ +Mobile Campfire (RWTH Aachen, 2010)

++ - - + - ++Junction (Stanford, 2010)

++ + - ++ - -Google Wave(Google Inc, 2009)

- ++ - - ++ -Novell Vibe Cloud(Novell, 2011)

- ++ - - ++ +CEFX+(London Arts U., 2007)

+ ++ - + ++ -Layar(Layar, 2009)

++ - ++ - - -Mixare(Mixare.org, 2010 )

++ - + ++ - -Wikitude Architect(Wikitude, 2009)

++ - + + - -

XmmC(This work) ++ ++ + ++ ++ ++






Conclusions XmmC is capable of storing and sharing various

multimedia types and sizes Concurrent XML file editing with mobile clients Collaborative annotation in digital documentation of

historical sites and increasing cultural heritage awareness

XMPP fits well for mobile real-time multimedia collaboration context

However, AR and XMPP consumes battery






Future Work

Further multimedia types– Videos, 3D objects, visual formats

Improved AR browser with 3D rendering and visual location mapping

Integration with real-time video streaming services, mobile cloud computing

Community support– User centric features, e.g. trust and security






Thanks for your attention!Q & A




Documents

A Real-time Collaboration-enabled Mobile Augmented Reality System with Semantic Multimedia