Finding Media Illustrating Events

Xueliang Liu <xueliang.liu@eurecom.fr>Raphaël Troncy <raphael.troncy@eurecom.fr> Benoit Huet <benoit.huet@eurecom.fr>

What are Events?

Experiences documented by Media

Events are observable occurrences grouping

People Places Time

19/04/2011 - - 2Finding Media Illustrating Events - ICMR 2011, Trento, Italy

http://www.flickr.com/photos/crsan/3697785107

Searching for an event

19/04/2011 - - 4Finding Media Illustrating Events - ICMR 2011, Trento, Italy

Searching for media about an event

19/04/2011 - - 5Finding Media Illustrating Events - ICMR 2011, Trento, Italy

There’s a lot of information out there…

19/04/2011 - - 6Finding Media Illustrating Events - ICMR 2011, Trento, Italy

http://www.flickr.com/photos/mwparenteau/43203978319/04/2011 - 7

Organize the mess

Scrape event directories Link the information Find media illustrating events Design the application Interface

EventMedia

19/04/2011 - - 8Finding Media Illustrating Events - ICMR 2011, Trento, Italy

Linking Open Data cloud diagram, by Richard Cyganiak and Anja Jentzsch. http://lod-cloud.net/

Róisín Murphy at Nouveau Casino

19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 9

350591

E0-001-005971169-9

Representing Events with

19/04/2011 - - 10Finding Media Illustrating Events - ICMR 2011, Trento, Italy

Interlinking

Linking Agents with Freebase, Dbpedia, MusicBrainz

Linking Venues with Geonames, Dbpedia, Foursquare (via Uberblic)

Linking Events with Last.fm, Upcoming, Eventful

Linking Categories with Facebook, Eventful, Upcoming, Zevents, LinkedIn,Eventbrite,

TicketMaster

Linking Users with Social Graph API

19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 11

Róisín Murphy at Nouveau Casino

19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 12

Media explicitly associated with the event

19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 13

1.7 million images over 108.000 events

Machine tags“lastfm:events”

APIs

4790 photos, 263 videos over 110 events

Representing Media with Media Ontology

19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 14

How much data is there?

19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 15

Event Agent Location Photos UserLast.fm 57,258 50,150 16,471 1,425,318 18,542Upcoming 13,114 0 7,330 347,959 4,518Eventful 37,647 6,543 14,576 0 0Total 108,019 56,693 38,377 1,773,277 23,060

1,248,021 geo-tagged photos by propagating information from events!

How fast media are uploaded?

19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 16

Finding more media that illustrate an event

A. Compute the bounding box area of a venue

B. Retrieve all media geo-tagged in this area

C. Retrieve all media with a similar title

D. Prune the results with visual analysis

E. Extend the result set with all media from the same uploader

19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 17

A. Bounding box of Nouveau Casino?

19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 18

B. 74 photos taken in this area this day

19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 19

C. 85 additional photos with a similar title

19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 20

D. 6 photos after visual pruning

19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 21

How is the visual pruning performed?

Model dataset: photo id + photo geo

Testing dataset: similar title

Low-level features used:Color moments, Gabor texture, Edge histogram

L1 distance on the K-nearest neighbors

ThresholdMin L1 distance between two model image pairsConservative approach

19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 22

E. 66 photos after uploader heuristics

19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 2319/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 23

cartoixa

hellerpop

13 photos

DustGraph / Stefan

46 photos

Same process for videos

19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 24

1 video (id)3 videos (geo)26 videos (title)

Visual pruning performed on key framesNb positive > 50%

How illustrated are the events?

19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 25

Query By ID Query By Geo Query By Title Visual Pruning Heuristic

Photos 5 74 (74) 85 (85) 6 (6) 66 (66)

Videos(title)

1 3 (0) 23 (0) 13 (0) -

Videos(title+venue)

10 (10)

20 eventsModel dataset: 785 photosTesting dataset: 1766 photos (1573 positive, 193 negative)Results: 439 photos (99% precision, 28% recall)

User Interface

19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 26

Conclusions

Method for finding media illustrating scheduledeventsSearch media with machine and geo tagsSearch media with title and normal tagsPrune visually and retrieve all media from confirmed usersChallenge: do not necessarily trust the geo-coordinates

Event-based approach for users to explore, annotate and share mediaUX can help semantics, semantics can help UXOutstanding challenges in interlinking and curating the dataDetecting events from social media activity

19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 27

19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 28

http://www.slideshare.net/troncy