DAVVI and vESP: experimental systems for doing search

in multimedia collections

Pål Halvorsen

TF-Media meeting, March 2011University of Oslo

Video streaming is everywhere!!

TF-Media meeting, March 2011University of Oslo

Streaming systems today... View a video sequentially

Once produced, video never changes

Make and distribute the same video to everybody

Emerging:− topic based composition using search− personalized playlists− recommendation− integration with social networking− anytime, everywhere...

timeline

TF-Media meeting, March 2011University of Oslo

Topic based composition

Query:- premier league - April 2010 - Liverpool- Goal by Steven Gerrard- 30 seconds duration Users select events

from multiple sources,played out as one video

TF-Media meeting, March 2011University of Oslo

Personalized video:

Personalized ordering

Query:- premier league - April 2010 - Liverpool- Goal by Steven Gerrard- 30 seconds duration

Users select eventsfrom multiple sources, arbitrary ordered,played out as one video

TF-Media meeting, March 2011University of Oslo

Recommendations and social networking

Personalized video:

Enabling user (re)publishing:− create directory service

(user generated content)− social network

Recommendations:− recommend personalized content− user interest profile stored−match user profile against

interesting content

StreamingSolution

Segmented Adaptive HTTP Streaming

TF-Media meeting, March 2011University of Oslo

Torrent-like HTTP streaming For load-balancing and scaling

multiple servers, taking the best from several worlds….

Downloads segments

Tracker manages information about segment locations

The user contacts the tracker for segment locations

Users send HTTP GET requests to download video segments

Video object:

TF-Media meeting, March 2011University of Oslo

Torrent-like HTTP streaming

playout time

quality

Based on experiments, we use 2-second segments(2-hour movie 3600++ small, indexed videos)

FFMPEG encoded:− H.264 (GOP = IP48I)− MP3− Custom made container

To support adaptation to available resources, each segment is coded in many quality levels

DAVVIsearch and delivery of soccer events

TF-Media meeting, March 2011University of Oslo

DAVVI: Idea

Present a 2-minutes video of highlights from last month games combined from

• goals by Dirk Kuyt• sliding tackles • tip over the bar• …

• TV broadcasters, etc. have huge repositories of sports content

o full videos, short events, highlights etc. o should be searchable

• Multimedia search and delivery systemsstill lacks precision and flexibility

TF-Media meeting, March 2011University of Oslo

YouTube (or Google, Bing, …)

TF-Media meeting, March 2011University of Oslo

VG Live

TF-Media meeting, March 2011University of Oslo

Web-servers fromBBC, Yahoo, VG, …

video analysislive-text crawling

live commentary

DAVVI

search / recommendation

feedback

tran

scod

er/

chop

per

HTTP Get

Video segments

stor

age

web

-ser

vers

tracker

DAVVI: system architecture

TF-Media meeting, March 2011University of Oslo

Annotation: sports event analysis

Audio-video analysis is difficult - why so hard?− Video data at 25 fps, an event may last 2,000 frames− Variation among & within sport broadcasts− Complex video quality, camera angles and on screen graphics

− Many different events to detect, e.g., in the context of soccer• yellow cards / red cards / goals / penalties / free kicks / fouls / corners / throw-ins / tackles / headings / passes / player numbers...

− Identify the beginning and end of the event− Find all the events – not miss any, no false positives

− Computationally expensive

How can an event be identified??

TF-Media meeting, March 2011University of Oslo

Annotation: sports event analysis

Event identification figures:− Huang et al. (U. Illinois) – text & video analysis (2000): 57%− Hanjalic et al. (Delft U.)– audio based analysis (2002): 52%− Sadlier et al. (DCU) – audio & video analysis (2005): 64%

Initial evaluation of visual/aural approach we developed for iAD: 67% - 83%

TF-Media meeting, March 2011University of Oslo

Annotation: live text commentaries Many online TV-stations and newspapers

provide live text commentaries

DAVVI uses a semi-automatic live-text crawler and parser to improve the automatic annotations

uk.eurosport.yahoo.com:

news.bbc.co.uk:

TF-Media meeting, March 2011University of Oslo

Search and recommendation Solr/Lucene open-source search

engine which has indexed thevideos

Users can query for video clips using a rich set of keywords, specifying values for tags or as free text

Each result is returned as a playlist of video segments, and playlists can be combined to make an topic-based, personalized video

The playlist describing the personalized video can be submitted to the social network

TF-Media meeting, March 2011University of Oslo

DAVVI demo systemsearch box automatically generate an X-minute playlist

search results – horizontally

scrollable

video quality

indicator

player controls

playlist generated by drag-and-drop or automatically

generated

each clip can be

adjusted

textual description of

the event which can be

expanded

vESPsearch and delivery of talks/lectures

TF-Media meeting, March 2011University of Oslo

vESP: Idea

Present a video of explanations about TCP congestion control techniques combining slides from talks/lectures given by

• Van Jacobson• Vinton G. Cerf • Mark Allman• Jitendra Padhye• …

• Companies/Schools/Universities/etc. have huge repositories of presentation content

o presentations, training videos, etc. o must be searchableo part of the content is multimedia

• Enterprise multimedia search still lacks precision and flexibility

TF-Media meeting, March 2011University of Oslo

YouTube

TF-Media meeting, March 2011University of Oslo

Altus vPresenter / vSearch

TF-Media meeting, March 2011University of Oslo

TalkMiner

TF-Media meeting, March 2011University of Oslo

tran

scod

er/

chop

per

talk transcript indexing

HTTP Get

Video segments

stor

age

web

-ser

vers

tracker

search

video analysis

slide indexing

vESP platform

TF-Media meeting, March 2011University of Oslo

vESP platform

Slide number

Start End

1 0:00 2:34

2 2:34 5:43

Query: “Windows 7”Slides

Transcript

Slide timing

Video of presentation

Search index

Video serverCustom presentation

Search results

TF-Media meeting, March 2011University of Oslo

vESP custom video presentation

PPT-file A, slide 3 PPT-file D, slide 2

Video-file: A, seg. 161 - 198

PPT-file C, slide 4

Video-file: D, seg. 20 - 28 Video-file: C, seg. 40 - 61

TF-Media meeting, March 2011University of Oslo

vESP demo system

slide playlist generated by drag-and-drop or

automatically generated

video presentationselect slides for playlist

document preview

TF-Media meeting, March 2011University of Oslo

Summary

DAVVI and vESP are scalable prototypes that−

− give you a new way to access video content

− integrate video streaming, search, personalization and recommendation(with social networking potential)

− well evaluated by subjective assessment group

Our next generation systems aim for 3D and free-view video experiences

TF-Media meeting, March 2011University of Oslo

Questions?? Comments??

Contact information:

Pål Halvorsen

paalh@ifi.uio.nohttp://home.ifi.uio.no/paalh