"Video Killed the Radio Star": From MTV to Snapchat

“Video Killed the Radio Star” the path from MTV to Snapchat

Lora Aroyo

http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo

The CNN/YouTube Republican Debate on 2007-‐11-‐28






h;p://www.blogherald.com/2010/10/27/history-‐of-‐online-‐video/

massive amount of digital content to explore …


but at some point it all looks the same …


Massive Scale: A lifetime of video content is uploaded to YouTube everyday.

Granularity Mismatch: Searching for the relevant video fragments is still not possible.

Passive Engagement: Video is still primarily a linear net-time viewing activity

… people search & browse with some implicit relevance in mind


snapchat genera8on …


audiences feel disconnected & lost …


there is huge seman8c & cultural GAP


so=ware systems are ever more intelligent


but they don’t actually understand people

focus on human knowledge in machine-‐readable form


but there are types of human knowledge that can’t be captured by machines

classical AI involves human experts to manually provide training knowledge for machines


human expert-‐based ground truth does not scale for current demand for machines to deal with wide

ranges of real-‐world tasks and contexts

we need to be able to …. support of mulGple perspecGves



to provide an approach to capturing human knowledge in a way that is scalable & adequate to real-‐world needs

the key scien8fic challenge is

Goodbye Single Truth

Hello Multiple

Perspectives

humans accurately perform interpreta8on tasks


humans accurately perform interpreta8on tasks


can their effort be adequately harnessed in a scien8fically reliable manner that scales across tasks,

contexts & data modali8es?

Quan8ty is the new Quality


Human Computa8on adopts human intelligence at scale to improve purely machine-‐based systems

diversity of opinion Independent decentralized aggregated

James Surowiecki

“the wise crowd”


a novel approach to gather diversity of perspec8ves & opinions from the crowd, expand expert vocabularies with these and gather new type of gold standard for machines

L. Aroyo, C. Welty: Crowd Truth: Harnessing disagreement in crowdsourcing a rela?on extrac?on gold standard. ACM WebSci 2013.


L. Aroyo, C. Welty. The Three Sides of CrowdTruth, Journal of Human Computa?on, 2014

http://CrowdTruth.org http://data.CrowdTruth.org/ http://game.crowdtruth.org

Visual Content Domina8on

•  90% of informa8on transmiSed to the brain is visual (processed 60,000X faster in the brain than text)

•  Videos increase average page conversion rates by 86% •  Visuals are social-‐media-‐ready/friendly -‐ easily sharable •  Posts with visuals receive 94% more page visits •  Visuals are becoming easier and easier to create as photo / video ediGng tools

become more accessible

any piece of media can be the starting point to a world of compelling visual experiences.

turning “mute” images into content-aware images.

NEW JERSEYHUDSON RIVER

CENTRAL PARK

URBANIZATION

VERIZON

METLIFE BUILDING

SUNSET

EAST RIVER

NEW YORK CITY

SKYSCRAPER

UPPER EAST SIDE

turning “mute” images into content-aware images.any piece of media can be the starting point to a world of compelling visual experiences.

combining machine processing with

crowdsourcing for enriching, curating &

gathering metadata

quickly & cheaply — at scale.


CENTRAL PARK

URBANIZATION

VERIZON

METLIFE BUILDING

SUNSET

EAST RIVER

NEW YORK CITY

SKYSCRAPER

UPPER EAST SIDE


CENTRAL PARK

URBANIZATION

VERIZON

NEW YORK CITY

SKYSCRAPER

METLIFE

BUILDING

UPPER EAST SIDEEAST RIVER

MIDTOWN

MANHATTAN

PAN-AM BUILDING

PAN-AM AIRLINES HELICOPTER CRASH

AIR TRAVEL

ARCHITECTURE

turning “context-free” images in

relationship-aware images


CENTRAL PARK

URBANIZATION

VERIZON

NEW YORK CITY

SKYSCRAPER

METLIFE

BUILDING

UPPER EAST SIDEEAST RIVER

MIDTOWN

MANHATTAN

PAN-AM BUILDING

PAN-AM AIRLINES HELICOPTER CRASH

AIR TRAVEL

ARCHITECTURE

… not only images, but also for videos

YOUTUBE: NYC FROM THE EMPIRE STATE BUILDING

allowing viewers to explore relationships across themes, locations, characters, etc. — within a video.


h;p://www.adweek.com/socialGmes/millennials-‐love-‐video-‐on-‐mobile-‐social-‐channels-‐infographic/622313





BRIDGING THE GAP BETWEEN PEOPLE & THE OVERWHELMING

AMOUNT OF ONLINE MULTIMEDIA CONTENT

HyperVideos: Link video fragments in non-linear paths

Binging Engagement:Construct continuous and interactive experiences

Video Snacks: Break video down into snackable moments

SOLUTIONS

•  Decomposing & granular description of images & videos.

•  Constructing mediaGraph with rich media semantics.

•  Continuously enriching & consolidating machine, expert, & user content descriptions.













Machines & Crowds

http://waisda.nl

Crowdsourcing Video Tags @Sound and Vision

@waisda hSp://waisda.nl

Two Pilots

Results of First Pilot

– The first 6 months: •  44.362 pageviews •  12.279 visits (3+ min online) •  555 registered players (thousands anonymous players!)

– 340.551 tags added to 602 items – 137.421 matches

Results of First Pilot

11 PartcipaGng Museums 1,782 Works of Art in the Research 36,981 Tags collected 2,017 Users who tagged

First two years (2006-‐2008)

Q: Why did you tag?

0% 20% 40% 60% 80% 100%

don't remember

to connect with others

so that I could find works again later

other (please specify)

to learn about art

to improve search for other users

for fun

to help museums document art work

Public

MMA

Tags by Documentalists •  Tags describe mainly short segments •  Tags are oaen not very specific •  Tags not describe programmes as a whole •  User tags were useful & specific -‐-‐> domain dependent

user vocabulary 8% in professional vocabulary 23% in Dutch lexicon 89% found on Google

locations (7%)

engeland

persons (31%) objects (57%)

On the Role of User-‐Generated Metadata in A/V Collec?ons Riste Gligorov et al. KCAP Int. Conference on Knowledge Capture 2011

Crowd vs. Professionals

System MAP

All user tags 0.219

Consensus user tags only 0.143

NCRV tags 0.138 NCRV catalog 0.077

Captions 0.157

Captions + User tags 0.247

Captions + NCRV catalog 0.183

Captions + NCRV tags 0.201 NCRV tags + User tags 0.263 NCRV tags + NCRV catalog 0.150 All – User tags 0.208 All 0.276

All tags better than consensus only •  Improvement of 53% •  Consensus tags have

•  higher precision: 0.59 vs. 0.49 •  but lower recall: 0.28 vs. 0.42

WAISDA? Tags vs. Rest

System MAP

All user tags 0.219



Captions 0.157




All tags better than rest •  Individually

•  beat NCRV tags by 69% •  beat captions by 39%


System MAP

All user tags 0.219



Captions 0.157






•  Combined •  Improvement of 5%


System MAP

All user tags 0.219



Captions 0.157




All data performs best •  largely due to contribution of user tags – 33%


System MAP

All user tags 0.219



Captions 0.157




All tags better than consensus only •  Improvement of 53% •  Consensus tags have

•  higher precision: 0.59 vs. 0.49 •  but lower recall: 0.28 vs. 0.42



All data performs best •  largely due to contribution of user tags – 33%

•  Combined •  Improvement of 5%


Current Pilot

h;p://spotvogel.vroegevogels.vara.nl/

Accurator ask the right crowd, enrich your collection

hSp://annotate.accurator.nl

Crowdsourcing & Nichesourcing @Rijksmuseum

Rijksmuseum Amsterdam collection over 1 million artworks

only a small fraction of about 8000 items are currently on display

… online collection grows 125.000 artworks already available

another 40.000 are added every year

expertise of museum professionals is in describing & annotating collection with art-historical information, e.g. when they were

created, by whom, etc.

detailed information about depicted objects, e.g. which species the animal or plant belongs to,

is in most cases not available

annotated only with “bird with blue head near branch with red leaf”

species of the bird and the plant are missing

use crowdsourcing to get more annotations use nichesourcing, i.e. niches of people with the right expertise, to add more specific information

use sources like Twitter to find experts or groups of experts on certain areas, e.g. bird

lovers, ornithologists or people who enjoy bird-watching in their spare time

platform where users enter tags: (1) structured vocabulary terms or (2) free text

hSp://annotate.accurator.nl

for tasks that are too difficult: game in which players can carry out an expert

annotation task with some assistance

BIRDWATCHING RIJKSMUSEUMSunday October 4, 10.00 am - 14.00 pmCuypers Library Rijksmuseum

On World Animal Day, the Rijksmuseum will host a birdwatching day in collaboration with Naturalis Biodiversity Center, Wikimedia Netherlands and the COMMIT/ SEALINCMedia project.

We are looking for bird watchers to join an expedi-tion through the digital collections and help the museums identify bird species in works of art.

dive.beeldengeluid.nl

In Digital Hermeneu8cs

Event-‐centric Explora8on @Sound & Vision and Royal Library

3rd Price at the SemanGc Web Challenge 2014

OPENIMAGES.EU •  3000 videos •  NL InsGtute for Sound & Vision •  mostly news broadcasts

DELPHER.NL •  1.5 Million Scans of •  Radio bulleGns •  (hand annotated) •  1937 – 1984

Simple Event Model (SEM) OpenAnnota8on (OA) & SKOS

DIVE:MEDIA OBJECT SEM:EVENT

SEM:PLACE

SEM:TIME

SEM:ACTOR

SKOS:CONCEPT

OA:ANNOTATION

•  LINKS TO EUROPEANA (MULTILINGUAL) •  LINKS TO DBPEDIA

Digital Submarine UI

Infinity of Explora8on

Events Linking Objects

Crowd Bringing the Human Perspec8ves

Linked (Open) Data

En8ty & Event Extra8on with CrowdTruth.org

ENTITY EXTRACTION

EVENTS CROWDSOURCING AND LINKING TO CONCEPTS THROUGH CROWDTRUTH.ORG

SEGMENTATION & KEYFRAMES

LINKING EVENTS AND CONCEPTS TO KEYFRAMES


Erp, M. van; Oomen, J.; Segers, R.; Akker, C. van de; Aroyo, L.; Jacobs, G.; Legêne, S; Meij, L. van der;O ssenbruggen, J.R. van; Schreiber, G. AutomaGc Heritage Metadata Enrichment with Historic Events Museums and the Web 2011 h;p://www.museumsandtheweb.com/mw2011/papers/automaGc_heritage_metadata_enrichment_with_hi


engaging users

through event

narratives


“Digital HermeneuGcs: Agora and the online understanding of cultural heritage” In proc. of Web Science Conference, (ACM: New York, 2011)

Interpreta8on Support for Online CollecGons


Explora8ve Search


Engagement with Games

Links from the slides

On the Web •  http://waida.nl •  http://prestoprime.org •  http://agora.cs.vu.nl •  http://sealincmedia.wordpress.com •  http://dive.beeldengeluid.nl •  http://diveplu.beeldengeluid.nl •  http://annotate.accurator.nl •  http://accurator.nl •  http://crowdtruth.org •  http://data.crowdtruth.org •  http://game.crowdtruth.org •  http://www.adweek.com/socialtimes/

millennials-love-video-on-mobile-social-channels-infographic/622313

•  http://www.blogherald.com/2010/10/27/history-of-online-video/

•  http://wm.cs.vu.nl

On TwiSer @waisda @agora-‐project @sealincmedia @prestocenter @vistatv #CrowdTruth #Accurator


Lecture Reading Material

h;p://www.aaai.org/ojs/index.php/aimagazine/arGcle/view/2564 Truth Is a Lie: Crowd Truth and the Seven Myths of Human AnnotaGon

h;ps://www.wired.com/2006/06/crowds/ THE RISE OF CROWDSOURCING

h;ps://www.microsoa.com/en-‐us/research/project/algorithmic-‐crowdsourcing/

h;p://cci.mit.edu/publicaGons/CCIwp2011-‐04.pdf Programming the Global Brain

h;p://www.orchid.ac.uk/eprints/248/1/main.pdf

The ACTIVECROWDTOOLKIT: An Open-‐Source Tool for Benchmarking AcGve Learning Algorithms for Crowdsourcing Research


Technology

"Video Killed the Radio Star": From MTV to Snapchat