View
222
Download
0
Category
Tags:
Preview:
Citation preview
Unsupervised discovery of visual object class hierarchies
Josef Sivic (INRIA / ENS), Bryan Russell (MIT), Andrew Zisserman (Oxford), Alyosha Efros (CMU) and
Bill Freeman (MIT)
Levels of supervision for training object category models
• None? Images only
[Agarwal & Roth, Leibe & Schiele, Torralba et al., Shotton et al.]
[Barnard et al.][Csurka et al, Dorko & Schmid, Fergus et al., Opelt et al]
• Object label +
segmentation
• Object label only
[Viola & Jones]
Can we learn about objects just by looking at images?
Goal: Given a collection of unlabelled images, discover a hierarchy of visual object categories
Which images contain the same object(s)?
Where is the object in the image?
Organize objects into a visual hierarchy (tree).
I. Represent an image as a bag-of-visual-words
II. Apply topic discovery methods to find objects in the corpus of images
Review: Object discovery in the visual domain
Decompose image collection into objects common to all images and mixture coefficients specific to each image
Hofmann: Probabilistic latent semantic analysisBlei et al.: Latent Dirichlet Allocation
[Sivic, Russell, Efros, Freeman, Zisserman, ICCV’05]
Topic discovery models
‘Flat’ topic structure – all topics are ‘available’ to all documents
d … documents (images)
w … visual words
z … topics (‘objects’)
Probabilistic Latent Semantic Analysis (pLSA) [Hofmann’99]
M documents
N words per document
Hierarchical topic models
• Topics organized in a tree
• Document is a superposition of topics along a single path
• Topics at internal nodes are shared by two or more paths
• The hope is that more specialized topics emerge as we descend the tree
c … paths
z … levels
[Hofmann’99, Blei et al. ’2004, Barnard et al.’01]
Hierarchical topic models
• Topics organized in a tree
• Document is a superposition of topics along a single path
• Topics at internal nodes are shared by two or more paths
• The hope is that more specialized topics emerge as we descend the tree
c … paths
z … levels
[Hofmann’99, Blei et al. ’2004, Barnard et al.’01]
Hierarchical topic models
• Topics organized in a tree
• Document is a superposition of topics along a single path
• Topics at internal nodes are shared by two or more paths
• The hope is that more specialized topics emerge as we descend the tree
c … paths
z … levels
[Hofmann’99, Blei et al. ’2004, Barnard et al.’01]
Hierarchical topic models
• Topics organized in a tree
• Document is a superposition of topics along a single path
• Topics at internal nodes are shared by two or more paths
• The hope is that more specialized topics emerge as we descend the tree
c … paths
z … levels
[Hofmann’99, Blei et al. ’2004, Barnard et al.’01]
Hierarchical topic models
d … documents (images)
w … words
z … levels of the tree
c … paths in the tree
For each document:
Introduce a hidden variable c indicating the path in the tree
c … paths
z … levels
[Hofmann’99, Blei et al. ’2004, Barnard et al.’01]
Hierarchical Latent Dirichlet Allocation (hLDA)
d … documents (images)
w … words
z … levels of the tree
c … paths in the tree
Treat P(z|d) and P(w|z,c) as random variables sampled from Dirichlet prior:
c … paths
z … levels
[Blei et al. ’2004]
Hierarchical Latent Dirichlet Allocation (hLDA)
d … documents (images)
w … words
z … levels of the tree
c … paths in the tree
c … paths
z … levels
[Blei et al. ’2004]
Tree structure is not fixed:
assignments of documents to paths, cj, are sampled from
the nested Chinese restaurant process prior (nCRP)
CRP: customers sit in a restaurant with unlimited number of tables
Nested Chinese restaurant process (nCRP)
1,2,3,4
1,2,3
1,2 3
4
4
[Blei et al.’04]
Nested CRP: extension of CRP to tree structures
• Prior on assignments of documents to paths in the tree (of fixed depth L)
• Each internal node corresponds to a CRP, each table points to a child node
Example:
Example: Tree of depth 3 with 4 documents
Sample path for the 5-th document
5th customer arriving
A
CB
D E F
hLDA model fitting
Use Gibbs sampler to generate samples from P(z,c,T|w)
c … paths
z … levels
For a given document j:
• sample zj while keeping cj fixed (LDA along one path)
• sample cj while keeping zj fixed (can delete/create branches)
Image representation – ‘dense’ visual words
Represent each region by a SIFT descriptor
Extract circular regions on a regular grid, at multiple scales
Cf. [Agarwal and Triggs’05, Bosch and Zisserman’06]
Build a visual vocabulary
Quantize descriptors using k-means
K = 10 + 1 K = 100 + 1
Visualization by ‘average’ words from the training set (single scale)
Vocabulary with varying degree of spatial and appearance granularity
K1 = 11
K2 = 101
K3 = 101
K4 = 101
Granularity
Appearance Spatial
Bag of words
Bag of words
3x3 grid
5x5 grid
Combined vocabulary:
K = 11+101+909+2,525
= 3,546 visual words
V1:
V2:
V3:
V4: Cf. Fergus et al.’ 05 Lazebnik et al.’06
Example I. – cropped LabelMe images
• 125 images, 5 object classes:
cars side, cars rear, switches, traffic lights, computer screens
• Images cropped to contain mostly the object, and normalized for scale
Example I. – cropped LabelMe images
• Learn 4-level tree hierarchy
• Initialization:
• c with a random tree (125 documents) sampled from nCRP (=1)
• z based on vocabulary granularity
c … paths
z … levelsK1 = 11
K2 = 101
K3 = 101
K4 = 101
Bag of words
Bag of words
3x3 grid
5x5 grid
V1:
V2:
V3:
V4:
Example I. – cropped LabelMe images
Learnt object hierarchy
Nodes visualized by average images
Example images assigned to different paths
Quality of the tree?
Intersection Union
ground truth images in class i
For each node t and class i measure the classification score:
Images assigned to a path passing through t
Good score:
- All images of class i assigned to node t (high recall)
- No images of other classes assigned to t (high precision)
Score for class i:
Quality of the tree?
Intersection Union
ground truth images in class i
For each node t and class i measure the classification score:
Images assigned to a path passing through t
Score for class i:
Example: traff. lights, node 2
Quality of the tree?
Intersection Union
ground truth images in class i
For each node t and class i measure the classification score:
Images assigned to a path passing through t
Score for class i:
Example: switches, node 9
Quality of the tree?
Intersection Union
ground truth images in class i
For each node t and class i measure the classification score:
Images assigned to a path passing through t
Overall score:
Example II. – MSRC b1 dataset
240 images, 9 object classes, pixel-wise labelled
Cars
Airplanes
Cows
Buildings
Faces
Grass
Trees
Bicycles
Sky
Example II. – MSRC b1 dataset
Experiment 1: Known object mask (manual), unknown class labels
Experiment 2: Both segmentation and class labels unknown (just images)
- More objects and images (than Ex. I)
- Measure classification performance
- Compare with the standard `flat’ LDA
- ‘Unsupervised discovery’ scenario
- Employ the ‘multiple segmentations’ framework of [Russell et al.,’06]
- Measure segmentation accuracy
MSRC b1 dataset – known object mask
Learnt tree visualized by average images, nodes size indicates # of images
Some nodes visualized by top 3 images (sorted by KL divergence)
MSRC b1 dataset – known object mask
Classification performance: comparison with ‘flat’ LDA
Flat LDA:
Estimate mixing weights for each topic i
Assign each image to a single topic:
MSRC b1 dataset – unknown object mask and image labels
Multiple segmentation approach [Russell et al.’06]
1) Produce multiple segmentations of each image
2) Discover clusters of similar segments
3) Score segments by how well they fit object cluster
Images Multiple segmentations Cars Buildings
(review)
(here use hLDA)
Road/asphalt
Conclusions
• Investigated learning visual object hierarchies using hLDA
• The number of topics/objects and the structure of the tree is estimated automatically from the data
• Topic/object hierarchy may improve classification performance
Recommended