Fcv hum mach_perona

Humans & Machinescollaborating on vision

Pietro PeronaCalifornia Institute of Technology

NSF Workshop - Frontiers in VisionCambridge, 23 Aug 2011

Friday, August 26, 2011

“Collaborative vision’’ ?

Pietro PeronaCalifornia Institute of Technology

NSF Workshop - Frontiers in VisionCambridge, 23 Aug 2011


Objectives

• Sketch new area of research

• Sampler of initial work

• Drawing lessons

• Brainstorm: potential, way forward


Plan

• Define area (10’)

• Presentations (50’): Perona, Geman, Grauman, Berg, Belongie

• Discussion (15’)


Definition


6


?

6


7





9


Lessons:

• Visual queries

• Easy for humans

• Difficult for machines

• Much information is available on line

• Pictures are digital dark matter

• Experts not providing visual knowledge

10


Unsupervised learning?

[Fergus et al., CVPR03] 11


Unsupervised learning?

[Fergus et al., CVPR03] 11


12



Throat


Throat


Visual knowledge

Task-oriented (practitioners)Categorical (experts) 14


Annotators Automata

ExpertsShared

knowledgeUsers

World

Querie

s

Answ

ers

Education

Models

ObservationObserv

ation

Science,expertise

Imageannotations

Machine visionscientists15


Annotators Automata

ExpertsShared

knowledgeUsers

World

Querie

s

Answ

ers

Education

Models

ObservationObserv

ation

Science,expertise

Imageannotations



Annotators Automata

ExpertsShared

knowledgeUsers

World

Querie

s

Answ

ers

Education

Models

ObservationObserv

ation

Science,expertise

Imageannotations



Annotators Automata

ExpertsShared

knowledgeUsers

World

Querie

s

Answ

ers

Education

Models

ObservationObserv

ation

Science,expertise

Imageannotations



Some progress...


Waterbirds

Mallard American Black Duck

Canada Goose Red Necked Grebe Clutter

DUCKS


x1i

x2i

xi = (x1i , x

2i )

p(xi | zi = 1)

p(xi | zi = 0)

Multidimensional signals and annotators


x1i

x2i

xi = (x1i , x

2i )

p(xi | zi = 1)

p(xi | zi = 0)



x1i

x2i

xi = (x1i , x

2i )

p(xi | zi = 1)

p(xi | zi = 0)


wj = (w1j , w

2j )

τj


lijxi

N

M

ij

σj

yij

θz

zi

Ji

βwj τj

γα

images

annotators

labels |Lij |

Full model

[Welinder et al., NIPS2010]Friday, August 26, 2011

Is there a duck in the image?

x1

i

x2

i



x1

i

x2

i



x1

i

x2

i



x1

i

x2

i



x1

i

x2

i



x1

i

x2

i



x1

i

x2

i


Concluding...


Aut

omat

ion

Performance

100%

100%

100%

0%

Collaborative vision


Aut

omat

ion

Performance

100%

100%

100%

0%



Aut

omat

ion

Performance

100%

100%

100%

0%



Aut

omat

ion

Performance

100%

100%

100%

0%



Aut

omat

ion

Performance

100%

100%

100%

0%


+ApplicationsTraining data

-ComplexityCost


Annotators Automata

ExpertsShared

knowledgeUsers

World

Querie

s

Answ

ers

Education

Models

ObservationObserv

ation

Science,expertise

Imageannotations



New research directions• Incremental learning

• Models of human vision, decision, attention

• Systems composed of machines and humans

• Performance bounds (humans, machines)

• Representations (human-machine-friendly)

• Extracting visual knowledge from experts
