Proprietary and Confidential

Exploration is the New Search/ Lora Aroyo/ Vrije Universiteit Amsterdam/ IBM Netherlands (CAS)/ Tagasauris Inc/ http://lora-aroyo.org/ @laroyo

2

massive amount of online digital video content

3

online video content is 64% of Internet traffic

4

300h of video uploaded each min on YouTube

5

in 2020 it would take a person more than 5 million years to watch the videos shared online in a month

6

there are videos out there relevant to you …

7

but …

8

most online videos NOT DESCRIBEDCAN’T BE FOUNDWON’T BE SEEN

9

what can we do to describe videos better?

10

machine computation can do a lot

11

recognise objects & conceptsProprietary and Confidential

unlock rich semanticsProprietary and Confidential 12

Proprietary and Confidential

[setting] WESTERN[character] DOLORES

[story arc] FLASHBACK 7

[object] TRAIN

[action] SHOOTOUT[mood] SURREAL

[audience] SHOCKED

[extras] 22

[role] BYSTANDER

13

discover creative contexts

14

link across online videosProprietary and Confidential

15

put all that descriptive information together to know what’s in a video

on a second-by-second basis

6

so, we can …

Proprietary and Confidential

16

add also behavioral information to know how people watch & talk about videos

6

and, we can …

Proprietary and Confidential

Viewership

Channels profiles

Social Media Buzz

Audience Demographics

Audience Contexts

Platform profiles

17

all this will make video search better

18

but we need more …

19

SEARCH vs. EXPLORATION/ helps only when you know what to search for/ assumes you understand a topic/ accuracy driven/ click-through driven

/ helps when you don’t know what to search for // helps you understand & deepen in a topic // serendipity driven // focused on engagement /

20

what can we do to support video exploration?

21

human computation can do a lot

22

CrowdTruth.org

bring more semantics of the real world

23

diversityofopinions,sen)ments,contextscollectedinadecentralizedwayaggregatedviewsovercollec1ons

introduces perspectives

http://crowdtruth.org

Proprietary and Confidential 24

consolidateexperts&endusersperspec)vesforcon1nuousimprovingvideodescrip1ons

25

humans & machines can go a long way

Proprietary and Confidential

HUMAN-IN-THE-LOOP AI FOR VIDEO EXPLORATION/Machines help to break-down video into granular moments, i.e. shots & scenes /Machines generate multitude of paths within and across videos / Humans perform simple actions, e.g. watching, following and rating a path/Machines generalise from these actions using explicit semantics/Machines learn to evolve & improve exploration path

/ Orchestrate a continuous human and machine symbiosis/ The ultimate aim is to reach a tipping point for video exploration,

e.g. web search, speech recognition

27

to transform the online video experience

28

Multimedia Explorations for Smart Culture

29

Interac1veExplora)on&DiscoveryinContextlinkingobjectstoeventsanden&&esbuildingautoma1cstorylines(narra1ves)ontopicsorobjects

Accesstointegratedonlinemul)mediacollec1onsusingLinkedOpenDatatointegratemetadataofvariousheritagecollec1ons

DIVE+

Aggregatedviewsoverthecollec1oncollec1ngperspec&vesfromcrowds&niches

http://diveproject.beeldengeluid.nl/

30Erp,M.van;Oomen,J.;Segers,R.;Akker,C.vande;Aroyo,L.;Jacobs,G.;Legêne,S;Meij,L.vander;Ossenbruggen,J.R.van;Schreiber,G.AutomaDcHeritageMetadataEnrichmentwithHistoricEventsMuseumsandtheWeb2011hKp://www.museumsandtheweb.com/mw2011/papers/automaDc_heritage_metadata_enrichment_with_hi

31

types

32

filters

33

explore entities

related entities

34

entity exploration

35

object exploration

36

people filter

37

event filter

38

narrative

39

40

Video Explorations withMicro-moments

MOMENTS REDEFINE CURRENT VIDEO SEARCH

MOBILE/SOCIAL OPTIMIZED

RELEVANT SEARCH RESULTS

PREVIEW SPECIFIC MOMENTS OR WATCH FULL VIDEOS

DISCOVER ACTIONABLE MOMENTS WITHIN VIDEO

MOMENTS: PERSONAL VIDEO CHANNEL

SEARCH FOR ANYTHING, WITH ANYTHING

HYPERMEDIA PLAYER W/ LINKED MOMENTS & FULL VIDEOS

AI ACTIVELY LEARNS USER PREFERENCES

DISCOVERS MORE & MORE CONTENT

AI, DEEP LEARNING & NETWORK EFFECTS

Ensemble algorithmic processing

Video, audio & text

Extract & understand entities

Does what computers are good at

AI & DEEP LEARNING:

//

//

// Human assisted computing

Collective intelligence (incl. fans)

Waze effect

Does what humans are good at

NETWORK EFFECTS:

//

//

//

OBSERVE & LEARN

VERIFY & EXTEND

*U.S. Patent Application #13/863,751

Web-scale layer of structured, linked data.

////

DOMAIN-SPECIFIC DATA

MOMENTS

44

45

47

Proprietary and Confidential

HUMAN-IN-THE-LOOP AI FOR VIDEO EXPLORATION/Machines help to break-down video into granular moments, i.e. shots & scenes /Machines generate multitude of paths within and across videos / Humans perform simple actions, e.g. watching, following and rating a path/Machines generalise from these actions using explicit semantics/Machines learn to evolve & improve exploration path

/ Orchestrate a continuous human and machine symbiosis/ The ultimate aim is to reach a tipping point for video exploration,

e.g. web search, speech recognition

49

to explore strange new worlds …. to boldly WATCH what no-one has WATCHED before

Proprietary and Confidential

Exploration is the New Search/ Lora Aroyo/ Vrije Universiteit Amsterdam/ IBM Netherlands/ Tagasauris Inc/ http://lora-aroyo.org/ @laroyo