IEEE 2017 Conference on

Computer Vision and Pattern

Recognition Discover and Learn New Objects from Documentaries

Kai Chen, Hang Song, Chen Change Loy, Dahua Lin

The Chinese University of Hong Kong

• We develop a novel approach to learning object detectors from

documentary videos and subtitles in an weakly supervised way.

• We propose a framework that can effectively integrate visual and

linguistic cues.

Welcome to visit our project homepage or scan the qrcode.

http://www.chenkai.site/projects/documentary-learning/index.html

Introduction

WildLife Documentary(WLD) Dataset

• Video frames + subtitles

• 15 documentary videos

• >700k frames (7.4h)

• >50 categories

• >4000 annotated tracklets

Framework Results

By the time they leave

her, they will weigh eight

times as much.

Most animals fear sloth

bears as but not

apparently wild boar, at

least not in this food

around.

The sambar deers are feeling

the heat too.

417-542

Background

Timeline

Iter 0

Iter 1

Iter 2

4149-4218

Boar6105-6232

Background

6232-6270

Background6268-6291

Background

4149-4218

Bear

417-542

Cub

6105-

6270(merged)

Deer

6268-6291

Background

417-542

Cub

4149-4218

Bear6105-6291

(merged)

Deer

Ground

truthTiger/Cub Bear Deer

The elephant are about to march throughthem. The spiders themselves have a span aswide as a human hand.

Tigers are one of the few cats that actuallyenjoy swimming.

Unlike mechanics, langurs are the friends ofspotted deer.

But the love serenade is over once a dogarrives.

Male koalas play no role in parenting.

There's a turf war going on and the koalasare losing.

Australian camels appear sick and emaciated.

About 50 animals have died in just threemonths, including this adult orangutan onthe day we arrived.

The mayor has declined offers of assistanceand expert advice from animal welfaregroups.

Examples

She needs to be good. Her cubs havealready got huge appetites

Most animals fear sloth bears as butnot apparently wild boar, at least not inthis food around.

Langurs are the friends of spotted deer.

merge tracklets

and categories

c3

for some babies drinking isn‘t themost comfortable oftime

These are among the rarest sloth bearsThe cubs also like a

cooling dip

c1

c1

c2

cn

…

update

keywords tracklets

screen

samples

…

classifier

classify

tracklets and

ground

keywords

1. Bootstrap 2. Joint Analysis

𝑝 𝑧, 𝑎, 𝑟 𝑜; Θ =1

𝑍 Θexp Ψap z o; 𝜃 + Φ𝑘𝑡 𝑧, 𝑎 𝑜; 𝜂 + Φ𝑠𝑡(𝑟, 𝑧|𝑜)

CRF formulation

Appearance

potentialKeyword-tracklet

potentialGeometric

potential

Ψap z o; 𝜃 = 𝑖=1

𝑛

𝜓𝑎𝑝(𝑧𝑖|𝑣𝑖; 𝜃)

Φkt z, a o; 𝜂 = 𝑖,𝑗 ∈𝐺

𝜙𝑘𝑡(𝑧𝑖 , 𝑎𝑖𝑗|𝜂)

Φst r, z o = 𝑖,𝑖′ ∈𝑅

𝜙𝑠𝑡(𝑟𝑖𝑖′ , 𝑧𝑖 , 𝑧𝑖′|𝑢𝑖 , 𝑢𝑖′)

object category𝑧𝑖whether tracklet 𝜏𝑖assiciates withkeyword 𝑤𝑗

𝑎𝑖𝑗

whether two trackletsshould be merged

𝑟𝑖𝑖′