Embed Size (px)
Citation preview
Actively Selecting Annotations Among Objects and Attributes
Adriana Kovashka, Sudheendra Vijayanarasimhan, and Kristen Grauman
University of Texas at Austin
Active Learning with Objects and Attributes
Interleave actively selected questions for object and attribute labels
Candidate annotations
o Pick the n most informative <image, label> pairs, where label is an
object label or label for some attribute m
Entropy-based selection function
o Maximizes expected object label entropy reduction
Inferring Missing Attribute Labels
Problem
Labeled data is critical for object category learning, yet expensive.
Active learning can mitigate the cost, but existing methods
o Restrict requests to “What object is it? Where in the image?”
o Consider object models independently
Our Idea
Minimize annotation effort by exploiting shared attributes among
objects and the relationships between them in active learning.
Why would this allow more efficient learning?
o An attribute label can influence many object models at once
o Correlation in attributes’ presence means only partial set needed
Entropy-Based Selection Function
Object class entropy on labeled and unlabeled image sets:
Seek maximal entropy reduction = minimum entropy:
Expected entropy scores for object and attributes:
Best <image, label> choice:
Object-Attribute Model
For classifier, we adopt a discriminative latent SVM model capturing
all object-attribute interactions from Wang & Mori, ECCV 2010.
Training
o Initial training images fully annotated with object+attributes labels
o Train with non-convex cutting plane method
Testing
o Predict the object label as:
o Features depend on latent attributes:
What is this object?
Does this object have spots?
(attribute)
target training data possible object labels
expected entropy after candidate label addition current entropy
object label object label
attribute label attribute label
Labeled
data
Current
classifiers
Unlabeled
data
Bat?
…
Panda?
Zebra?
Black?
White?
Big? …
Furry?
Saddle
Rein
Snout
Wool Furry
Horn
“Horse”
brown = 0
legs = 1
horns = 0
object classifier
attribute classifiers
attribute-attribute relationships
object-attribute relationships
object is?
has stripes?
object is?
is blue?
…
Cu
rren
t m
od
el
Selected questions to human
Update model with new labels
siamese cat blue = 0
antelope horns = 1 white = 1
Initial labeled data
bobcat stripes = 1
whale lean = 0
dalmatian spots = 1
panda stripes = 0
…
…
…
…
…
…
Sorted <image, label request>
pairs from unlabeled data
Results
Animals with Attributes – 1
(1003 unlabeled, 732 test)
Animals with Attributes – 2
(1002 unlabeled, 993 test)
aYahoo
(703 unlabeled, 200 test)
aPascal
(903 unlabeled, 287 test)
What active requests does our method make?
Learning curves: how quickly does the method learn?
Entropy reduction: how good are the selected queries?
hamster hippopotamus horse humpback whale killer whale
tiger walrus weasel wolf zebra
centaur donkey goat monkey wolf zebra
aeroplane bicycle boat bus car motorbike train
Hidden
attribute
labels Image
object
classifier
attribute
classifiers
attribute – attribute
relationships
object – attribute
relationships Object
label
Comparable!
Key point: By predicting entropy change over all data,
selection accounts for impact of all desired interactions.
For partially labeled data, we infer missing attribute labels
Enables faster learning – avoid waiting for full labeling to see effects
Conclusion and Impact Richer annotations “beyond labels” are critical for active learning
Most efficient use of annotator effort to train multi-class object models
Natural means to enhance multi-class object category learning
Proposed method builds accurate models with less total human effort,
outperforming traditional active approach that selects object labels only.
Distribution of all
requests:
Indicates impact of
shared attributes on
reducing uncertainty
Sample
<image, label>
requests selected
by our method
Does it swim? What is this object? Does it live in the ocean?
What is this object? Does it walk? Is it an arctic animal?
