Tutorial

Advanced Information Theory in CVPR “in a Nutshell”

CVPRJune 13-18 2010

San Francisco,CAIntroduction to ITinCVPR:The 4 Axes

Francisco Escolano

Why ITinCVPR?

Information theory (IT) has been growing in interest within theComputer Vision and Pattern Recognition (CVPR) community.In confluence with Bayesian theory, energy minimization, and othermethodologies, the role of IT is key to face the complex task ofdeveloping reliable and efficient CVPR algorithms.However, such confluence must evolve from exploiting IT for solvingspecific tasks towards an identification of common IT elements to alltasks and, mostly important, their interrelation.Here we consider four dimensions (axes) for describing theseinterrelations: measures, principles, theories, and entropy estimators.

2/24

The four Axes

Figure: ITinCVPR algorithms lay in a 4D conceptual manifold

3/24

IT Measures

Definition

IT measures are probability-based criteria designed for quantifying:information content, information gain and loss, information bounds,high-order statistical dependencies,....

Roles

The main roles of IT measures are:

I Definition of error bounds.

I Definition of discrimination criteria.

I Definition of optimization criteria for generative processes.

I Definition of convergence criteria.

4/24

IT Measures (2)

Error-bound Discriminate Generate Converge

Entropy√ √

Kullback-Leibler√ √ √

Jensen-Shannon√ √

Bregman√

Mutual Info√ √

Chernoff Info√ √

Channel Capacity√ √

Fisher-Rao Tensor√

Table: The roles of some IT measures

5/24

IT Measures (3)

MD Feature Selection

Number of Selected Gene

Cla

ss (

dise

ase)

MELANOMAMELANOMAMELANOMAMELANOMAMELANOMAMELANOMA

BREASTBREAST

MELANOMANSCLCNSCLCNSCLC

BREASTMCF7D−repro

BREASTMCF7A−repro

COLONCOLONCOLONCOLONCOLONCOLONCOLON

LEUKEMIALEUKEMIALEUKEMIALEUKEMIALEUKEMIA

K562A−reproK562B−reproLEUKEMIA

NSCLCNSCLCNSCLC

PROSTATEOVARIANOVARIANOVARIANOVARIANOVARIAN

PROSTATEMELANOMA

OVARIANUNKNOWN

RENALNSCLC

BREASTRENALRENALRENALRENALRENALRENALRENALNSCLCNSCLC

BREASTCNSCNS

BREASTRENAL

CNSCNSCNS

19→

135

246

663

766

982

1177

1470

1671

→ 2

080

3227

3400

3964

4057

4063

4110

4289

4357

4441

4663

4813

5226

5481

5494

5495

5508

5790

5892

6013

6019

6032

6045

6087

→ 6

145

6184

6643

mRMR Feature Selection

Number of Selected Gene

MELANOMAMELANOMAMELANOMAMELANOMAMELANOMAMELANOMA

BREASTBREAST

MELANOMANSCLCNSCLCNSCLC

BREASTMCF7D−repro

BREASTMCF7A−repro

COLONCOLONCOLONCOLONCOLONCOLONCOLON

LEUKEMIALEUKEMIALEUKEMIALEUKEMIALEUKEMIA

K562A−reproK562B−reproLEUKEMIA

NSCLCNSCLCNSCLC

PROSTATEOVARIANOVARIANOVARIANOVARIANOVARIAN

PROSTATEMELANOMA

OVARIANUNKNOWN

RENALNSCLC

BREASTRENALRENALRENALRENALRENALRENALRENALNSCLCNSCLC

BREASTCNSCNS

BREASTRENAL

CNSCNSCNS

133

134

→ 1

3523

325

938

156

113

7813

8214

0918

41→

208

020

8120

8320

8632

5333

7133

7243

8344

5945

2754

3555

0455

3856

9658

1258

8759

3460

7261

15→

614

563

0563

9964

2964

3065

66

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Figure: Microarray Gene Selection with Multi-dimensional Mutual Info

6/24

IT Measures (and 4)

Figure: Mutliple Point-Set Registration though Havrda-Chavrat Divergence

7/24

IT Principles

Definition

Formal requirements casting the optimal solutions to the CVPRproblem at hand and thus the design of a proper algorithm.

Roles

The main roles of IT principles are the definition of:

I Model-order selection criteria (] regions/clusters).

I Feature selection and transformation (coding) criteria.

I Optimization criteria for generating models for the data.

I Classification-design criteria.

8/24

IT Principles (2)

Model order Features Generate Classify Coding

MDL√

MML√ √ √

MaxEnt√ √

MiniMax√ √ √

Infomax√ √

Non-Gaussianity√ √

MED√

Min Correlation√

Inf.Bottleneck√

Table: The roles of some IT principles

9/24

IT Principles (and 3)

Figure: MDL Model Order Selection in Gaussian Mixtures

10/24

IT Theories

Definition

Mathematical developments leading to different formulations of thesame problem and, thus contributing with new formal perspectives.

Main Roles

The main roles of IT theories are the definition of:

I Generative/discriminative formulations.

I Matching formulations.

I Clustering formulations.

I Feature and classification design formulations.

11/24

IT Theories (2)

Discr. Generate Segment Cluster Feature/Class.

Method of Types√ √ √

Rate-Distortion√ √ √

Info geometry√ √ √ √

Proj. pursuit√ √ √

Bregman divs.√ √ √

Table: The roles of some IT theories

12/24

IT Theories (3)

Figure: Saliency Filtering through Method of Types

13/24

IT Theories (and 4)

Figure: Minimum Enclosing Bregman Balls

14/24

Entropy Estimation

Definition

Asymptotically unbiased non-parametric estimator of entropy orrelated IT measure (KL-divergence, Mutual Info) considering alsodifferent definitions of entropy and their formal interrelations.

Definitions of entropy

I Shannon: H(p) = −∑

i pi log pi .

I Renyi: Rα(p) = 11−α log (

∑i pα

i ).

I Tsallis: Sα(p) = 1α−1 (1−

∑i pα

i ).

I Burg: B(p) =∑

i log pi .

15/24

Entropy Estimation (2)

Problems in EE

Typically a small number of samples N and high number ofdimensions d : curse of dimensionality.

Types of EEs

I Plug-in: Necessarily estimate the underlying pdf (binning,Parzen Windows, Voronoi-based...). Useful for low d .

I Bypass: Exploit only the available data avoiding pdf estimation.Typically nearest neighbors-based methods (Entropic Graphs,Leonenko) or KD-partitions. Useful for intermediate and evenhigh d .

16/24

Entropy Estimation (3)

−20 −15 −10 −5 0 5 10 15 20−20

−15

−10

−5

0

5

10

15

20

25

0 2 4 6 8 10 12 14 16 18 200

2

4

6

8

10

12

14

16

18

20

Figure: Entropic Graph Entropy Estimation (Gaussian vs Uniform)

17/24

Entropy Estimation (and 4)

Figure: Joint PDFs for Image Alignmnet: without and with interpolation

18/24

The ITinCVPR Tube

Figure: ITinCVPR tube lines, stations and quarters. 19/24

ITinCVPR Tube

Quarters ≡ Tasks

Our tube links IT elements for solving 6 CVPR tasks: featureextraction and grouping, segmentation, registration/matching andrecognition, feature selection and transformation, image and patternclustering and classifier design

Lines and transfer stationsI Measures, principles and theories lines interact through transfer

stations. Transferring can imply carrying on previous ITelements.

I Entropy estimation (fourth dimension) is marked at somepoints where such estimation is either critical or challenging.

20/24

The Measures Lines

Figure: ITinCVPR tube lines, stations and quarters. 21/24

The Principles Lines

Figure: ITinCVPR tube lines, stations and quarters. 22/24

The Theories Lines

Figure: ITinCVPR tube lines, stations and quarters. 23/24

Tutorial Schedule

Hour Topic Instructor

2-2:15pm Introduction: The 4 axes Escolano

2:15-2:45pm Interest Points & Method of Types Escolano

2:45-3:30pm High-Dimensional Feature Selection Escolano

3:30-4pm COFFEE BREAK4-4:30pm Isocontours and Image Registration Rangarajan

4:30-5:15pm Shape Matching with I-Divergences Rangarajan

5:15-6pm Gaussian Mixtures: MDL and Variational Both

Table: Tutorial Schedule

24/24