Multimedia Information Retrieval

Prof Stefan Rüger

Multimedia and Information SystemsKnowledge Media Institute

The Open Universityhttp://kmi.open.ac.uk/mmis

kmi.open.ac.uk

kmi.open.ac.uk

kmi.open.ac.uk

Since 1995: 117 projects & 67 technologies

Current year

17 live projects , typically per year£2.5m (¥300m) ext, £1m (¥120m) internal• 10 EU• 3 UK • 1 US• 3 internal (iTunes U, SocialLearn)

Multimedia Information Retrieval

1. What are multimedia queries?

2. Fingerprinting

3. Metadata & piggy-back retrieval

4. Automated image annotation

5 Visual content-based retrieval I

6 Visual content-based retrieval II

7. Evaluation

8. Browsing, search and geography

Multimedia Information Retrieval

1. What are multimedia queries? - What is multimedia? - Query by image - Current best practice for image search - Snaptell/Google goggles - Shazam - Discussion: Challenges and difficulties

2. Fingerprinting

3. Metadata & piggy-back retrieval

4. Automated image annotation

5 Visual content-based retrieval I

6 Visual content-based retrieval II

7. Evaluation8. Browsing, search and geography

What is Multimedia?

Within this lecture:One or more mediaPossibly interlinkedDigitalFor communication (not only entertainment)‏

Sensō-ji ( � � � � � � Kinryū-zan Sensō-ji?) is an ancient Buddhist templelocated in Asakusa, Taitō, Tokyo, Japan. It is Tokyo's oldest temple, and one of its most significant. Formerly associated with the Tendai sect, it became independent after World War II. Adjacent to the temple is a Shinto shrine,the Asakusa Shrine [http://en.wikipedia.org/wiki/Sensō-ji]

Multimedia queries

Web-based image searching

“Tokyo temple”

Google ImagesBing ImagesFlickrYahoo ImagesЯндекс

Web-based image searching

Best current practice is a text search:Find text in filename, anchor text, caption, ...

Text search works by creating a large index:

GoogleTokyo temple

BingTokyo temple

FlickrTokyo temple

YahooTokyo temple

YandexTokyo temple

New search types

query doc

conventional text retrieval

hum a tune and get a music piece

you roar and get a wildlife documentarytype “floods” and get BBC radio news

Example

text

video

images

speech

music

sketches

multimedia

loca

tion

sound

hum

min

g

mot

ion

text

imag

e

spee

ch

Exercise

Organise yourself in groupsDiscuss with neighbours - Two Examples for different query/doc modes? - How hard is this? Which techniques are involved? - One example combining different modes

Exercise

query doc

Discuss

- 2 examples

- How hard is it?

- 1 combination

loca

tion

sound

hum

min

g

mot

ion

text

imag

e

spee

ch

loca

tion

sound

hum

min

g

mot

ion

text

imag

e

spee

ch

text

video

images

speech

music

sketches

multimedia

Near-duplictate detection:Cool access mode!

Snaptell: Book, CD and DVD covers

Snaptell: Book, CD and DVD covers

Snaptell: Book, CD and DVD covers

Snaptell: Book, CD and DVD covers

Snaptell: Book, CD and DVD covers

Link from real world to databases

doi: 10.2200/S00244ED1V01Y200912ICR010

The Open Univerity'sSpot & Search

Scott Forrest: E=MC squared

"Between finished surface texture and raw quarried stone. Between hard materials and soft concepts.

Between text and context."

More information

[with Suzanne Little]

Spot & Search

[with Suzanne Little]

Near duplicate detection

Works well in 2d: CD covers, wine labels, signs, ...Less so in near 2d: buildings, vases, …Not so well in 3d: faces, complex objects, ...

Shazam

Rueger, Multimedia IR, 2010explains it all! Buy it now

Near duplicate detectionExercise

Find applications for near-duplicate detection - be imaginative: the more “outragous” the better - can be other media types (audio, smells, haptic, ...) - can be hard to do

Near-duplicate detectionWhere are the challenges?

[Victoria and Albert museum, London, ceramics collection, 2010]

Leaf detectionWhat are the challenges?

[with Natural History Museum, London, and Goldsmiths]