Embed Size (px)
DESCRIPTION
As the fastest-growing type of content on the Internet, consumer produced videos are a wealth of information about the world that's essentially untapped. We present ICSI's research on the large-scale video search methods using an application that reveals the geo-location of a consumer- produced video based on its content. Gerald Friedland, University of California, Berkeley.
Citation preview
Finding the Needle in the Video Haystack
Dr. Gerald Friedland Director Audio and Multimedia Research
International Computer Science Institute Berkeley, CA [email protected]
The Internet is Multimedia
2
Multimedia in the Internet is Growing
3
Multimedia in the Internet is Growing
3
• YouTube alone claims 48 72 100 hours video uploads every minute.
Multimedia in the Internet is Growing
3
• YouTube alone claims 48 72 100 hours video uploads every minute.
• Youku (Chinese YouTube) claims 80k video uploads per day
Multimedia in the Internet is Growing
3
• YouTube alone claims 48 72 100 hours video uploads every minute.
• Youku (Chinese YouTube) claims 80k video uploads per day
• Flickr, Instagram, Liveleak, Vimeo...
4
The Opportunity
5
The Opportunity
• Consumer-Produced Multimedia allows empirical studies at never-before-seen scale:
5
The Opportunity
• Consumer-Produced Multimedia allows empirical studies at never-before-seen scale:– sociology,
5
The Opportunity
• Consumer-Produced Multimedia allows empirical studies at never-before-seen scale:– sociology, – medicine,
5
The Opportunity
• Consumer-Produced Multimedia allows empirical studies at never-before-seen scale:– sociology, – medicine,– economics,
5
The Opportunity
• Consumer-Produced Multimedia allows empirical studies at never-before-seen scale:– sociology, – medicine,– economics, – …
5
The Opportunity
• Consumer-Produced Multimedia allows empirical studies at never-before-seen scale:– sociology, – medicine,– economics, – …
• Problem: Videos need to be searchable beyond keywords.
5
The Opportunity
• Consumer-Produced Multimedia allows empirical studies at never-before-seen scale:– sociology, – medicine,– economics, – …
• Problem: Videos need to be searchable beyond keywords.
• 5
Our Approach
6
Ball soundMale voice (near)
Child’s voice (distant)Child’s whoop (distant)
Room tone
Cameron learns to catch (http://www.youtube.com/watch?v=o6QXcP3Xvus)
Our Approach
Multimodal exploitation of video content, including audio and temporal information.
6
Ball soundMale voice (near)
Child’s voice (distant)Child’s whoop (distant)
Room tone
Cameron learns to catch (http://www.youtube.com/watch?v=o6QXcP3Xvus)
Location Estimation
7
J. Choi, G. Friedland, V. Ekambaram, K. Ramchandran: "Multimodal Location Estimation of Consumer Media: Dealing with Sparse Training Data," in Proceedings of IEEE ICME 2012, Melbourne, Australia, July 2012.
Bayesian graphical framework
8
{berkeley, sathergate, campanile}
{berkeley, haas}
{campanile} {campanile, haas}
Node: Geoloca7on of the image
Edge: Correlated loca7ons (e.g. common tag)
Edge Poten,al: Strength of an edge, (e.g. posterior distribu7on of loca7ons given common tags)
p(xi, xj |{tki } � {tkj })
p(xj |{tkj })p(xi|{tki })
