Addressing and Annotating Multimedia Fragments

Addressing and annotating multimedia fragments

Raphaël Troncy <raphael.troncy@eurecom.fr>

Use Case

10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 2

Use Case

Aidem received on her Facebook wall a status message containing a Media Fragment URI Use a ‘#’ ! Highlight a

video sequence

Highlight a region to pay attention to

What are Media Fragments?

t 0 20 35 temporal media fragment

spatial media fragment

track media fragment

Temporal Media Fragments

Fragment beginning Fragment end Playback progress

Original resource length

Spatial Media Fragments

semi-opaque overlay

highlighted fragment

Media Fragments Use Cases

Bookmark / Share parts (fragments) of audio/video content

Annotate media fragments

Mash-ups

Conserve bandwidth

http://www.w3.org/TR/media-frags-reqs/

Requirements

r01: Temporal fragments: a clipping along the time dimension from a start to an end time that

are within the duration of the media resource

r02: Spatial fragments: a clipping of an image region, only consider rectangular regions

r03: Track fragments: a track as exposed by a container format of the media resource

r04: Named fragments: A temporal media fragment that has been given a name through

some sort of annotation mechanism

Media URIs

Using URI query part:

Using URI fragment part:

Mixing both:

http://www.example.org/video.ogv?t=60,100

http://www.example.org/video.ogv#t=60,100

http://www.example.org/video.ogv?t=60,100#t=10,15

The media fragment URI syntax can be used for URI queries

We will focus on URI fragments

URI Fragments vs. URI Queries

#t=20,30 ?t=20,30 secondary resource,

notion of context primary resource,

no notion of context extraction needs to be

expressible in byte ranges no adaptation restrictions

no provisions for communi-cating

fragments to the server

key-value pairs are sent to the server

potentially cacheable not cacheable

Media Fragments Resolution

For the URI query part: The media file is only processed on server side The UA receives a new video file

For the URI fragment part: Smart UA may strip out the fragment definition and

encode it into custom http headers (Range header) (Media) Servers will handle the request, slice the media

content and serve just the fragment (corresponding byte ranges) … while old ones will serve the whole resource

Influence of Media Formats

Fragment extraction needs to be expressible in terms of byte ranges

Requirements for the different axes temporal: presence of intra-coded frames

(i.e., random access points) spatial: presence of independently coded spatial regions track: need to be identifiable by a name

Conclusion: temporal and track axes are realistic, spatial fragments can hardly be expressed in terms of byte ranges

Can I use Media Fragments now?

Firefox 9+ (or WebKit)

No bandwidth saving

Sysnote over YouTube

No bandwidth saving

Ninsuna

Bandwidth saving

Examining HTTP Traffic

HTTP request: retrieving URI parsing key=values pairs from fragment part setting Range header

HTTP response: checking Content-Type

and Content-Range-Mapping headers values attaching custom playback controls to page creating spatial dimension overlay (if specified)

Example: Requesting a Time Fragment

A web developer specifies a video source with a temporal fragment URI:

key=value pair is analyzed, fragment begin and end time are matched

Media Fragments Extension analyses the fragment part, retrieves beginning and end time and sets proper Range header value:

http://ninsuna.elis.ugent.be/DownloadServlet/mfwg/fragf2f.ogv#t=5,15

t=5,15

Range: t:npt=5-15

The NinSuna server responds with the 206 Partial Content response and Content-Range-Mapping header showing the mapped time ranges and media fragment in the message payload:

HTTP/1.1 206 Partial Content Content-Type: video/ogg Accept-Ranges: bytes, t, track, id Content-Range: bytes 629578-1690588/4055466 Content-Range-Mapping: {t:npt 4.8-14.8/0-38.33}={bytes 629578-690588/4055466}

... and it won’t work!

Because the player does not have the media file header

Player needs to be initialized, thus bare fragment is not playable

To fix this, we add ;include-setup to the Range header value:

Range: t:npt=5-15;include-setup

The response from the server is slightly different:

HTTP/1.1 206 Partial Content Content-Type: multipart/byteranges;boundary=End Content-Range-Mapping: {t:npt 4.8-14.8/0-38.33;include-setup} ={bytes 0-5998,629578-1690588/4055466} ... --End Content-Type: video/ogg Content-Range: bytes 0-5997/4055466 {binary data} --End Content-Type: video/ogg Content-Range: bytes 629578-1690588/4055466 {binary data} --End --End--

This is why we need to attach a Stream Listener to the HTTP channel:

HTTP/1.1 206 Partial Content Content-Type: multipart/byteranges;boundary=End ... --End Content-Type: video/ogg Content-Range: bytes 0-5997/4055466 {binary data} --End Content-Type: video/ogg Content-Range: bytes 629578-1690588/4055466 {binary data} --End --End--

HTTP/1.1 206 Partial Content Content-Type: video/ogg ... {binary data} {binary data}

Spatial Fragments

Without additional information send to server

Overlay is created by appending four additional DIVs to the webpage

They are styled to create the impression of semi-opaque layer over the video element

NinSuna: Overview

Fully integrated media adaptation and delivery platform media adaptation and media packaging core is

independent of media formats based on model-driven content adaptation & delivery technique

only high-level adaptation operations such as scene selection and frame rate scaling no transcoding is applied ideal candidate for MF implementation

more information: http://ninsuna.elis.ugent.be

Support for Media Fragment URI queries both HTTP and RTSP implementation try some URIs at http://ninsuna.elis.ugent.be/MediaFragmentsServer#Test http://ninsuna.elis.ugent.be/DownloadServlet/apple/10,000_BC_t

railer_2.mp4?track=5;6

Support for Media Fragment URI fragments i.e., support for the MF-specific HTTP headers

Combining Media Fragment URI queries and fragments e.g., http://foo.com/media.mp4?t=10,40#t=5,10

MF-specific Features of NinSuna

Updating existing infrastructure is not trivial web servers need to be extended with media extractors MF-specific HTTP communication

existing web caches are only effective with HTTP byte ranges specialized media caches need to be developed in the future to

cache the other cases

user agents need to be extended with a Media Fragment URI parser and interpreter MF-specific HTTP communication MF-specific visualization

Impact on the Existing Web Infrastructure

Goal: make existing media resources, served by generic HTTP Web servers, available as Media Fragments

Biggest problem for content providers willing to support Media Fragments media extractor (dependent on underlying media

formats)

Solution: an approach that works with existing HTTP Web servers (apache, IIS) works with existing Web caches works with not so smart user agents

Media Fragments Proxy

-get (only) the header info -try to find a fragment-to-byte mapping -construct the redirect response

Media Fragments Proxy

HTTP server

MF Proxy MF User

http://foo.com/video.ogv#t=11,19

GET /?url=http://foo.com/video.ogv HTTP/1.1 Host: MFProxy.com Accept: video/* Range: t:npt=11-19 Accept-Range-Redirect: bytes

HTTP/1.1 307 Temporary Redirect Location: http://foo.com/video.ogv Accept-Ranges: bytes, t, track Content-Length: 0 Content-Type: video/ogg Content-Range-Mapping: t:npt 10-20/0-50 Range-Redirect: 24000-32000 Vary: Accept-Range-Redirect

GET /video.ogv HTTP/1.1 Host: foo.com Accept: video/* Range: bytes=24000-32000

http://ninsuna.elis.ugent.be/MFProxy?url=<mediaURI>

Annotating Media Fragments

yuma.min.js

Annotating Media Fragments

Benchmarking: Sphinx, HTK, Julius

NER + full text index with the transcription

Interlinking with the Linked Data Cloud to enable semantic search

http://www.slideshare.net/troncy

10/04/2012 - - 32 Séminaire Muséologie - Le Web devient audiovisuel

Addressing and Annotating Multimedia Fragments

Technology

A URI-based approach for addressing fragments of media ...troncy/Publications/Troncy_Mannens-mtap12.pdf · A URI-based approach for addressing fragments of media resources on the

Highlighting VS Annotating

Annotating Abstract Syntax Tree

Annotating Texts

Annotating Search Results From

McCarthyism + Annotating

Annotating and deconstructing contents page

Annotating Retroelements

Annotating images

Annotating Posters

ANNOTATING MATERIAL. WHAT IS ANNOTATING? Annotating a text is a way to augment your understanding. It is a fairly pliable skill and therefore difficult

Annotating with Annotations - DPC UnCon

Annotating Search Results

Annotating UML with MOAL

Annotating Population Pyramids

Annotating Digital Documents

-Annotating Full Document

Annotating Texts S17

Annotating Java Bytecode

Annotating 2 Products