Canonical correlation analysis (CCA)

Proposed method: SemiCCA

[WeBCT8.14] International Conference on Pattern Recognition (ICPR2010)

SemiCCA: Efficient semi-supervised learning of canonical correlationsAkisato Kimura (1),Hirokazu Kameoka (1),Masashi Sugiyama (2),Takuho Nakano (1,3),

Eisaku Maeda (1),Hitoshi Sakano (1),Katsuhiko Ishiguro (1)

(1) NTT Communication Science Laboratories, NTT Corporation, Japan(2) Tokyo Institute of Technology, Japan (3) the University of Tokyo, Japan

<Abstract> Semi-supervised variant of canonical correlation analysis (CCA)1. Incorporating additional unpaired samples for mitigating overfitting

even when the number of paired samples is quite limited2. Can be computed efficiently (just by solving a single eigenvalue problem)

smoothly bridges the eigenvalue problems of CCA and PCA

Linear projection

Linear projection

Observation 1 Observation 2

Maximizing correlations

Multi-label predictionFeature vectors Sets of class labels

Multi-modal relationshipImage features Audio features

Paired

Unpaired

Application to automatic image annotation

Problem :

The number of paired samplesis often limited overfitting

(1) CCA: a method of finding bases: maximizing the correlationamong projected vectors

(Mean = 0 for bravity)

(Covariance matrices of paired samples)

(2) Taking derivatives of Lagrangean with respect to , we obtain

Matrix form

PCA for paired& unpaired samples in each domain

SemiCCAFormulated as a generalized eigenvalueproblem ----- almost the same as that for CCA

Experiments with artificial data

Projection basis for discrimination

Discriminant boundary

Linear discriminant function

Framework Calculus

Paired

Framework

How to incorporate unpaired samples ?Needs some assumptions for the nature of unpaired samples A global structure in each domain revealed

by unpaired samples would be consistent with co-occurrence information.

Calculus

Subspace estimated by CCA in case all unpaired samples in X and Y are paired

Subspace estimated by semiCCA from both unpaired and paired samples

paired samplesunpaired samples in Xunpaired samples in YSubspace estimated

from paired samples

Smoothlybridging

: CCA for paired samples

: PCA for paired & unpaired samplesin each domain

: inherits both the propertiesglobal structure in each domainco-occurrence information

Relationships among previous researchesMultivariate analysis

CCA

FDAMLR

PCA

MLR := multiple linear regression,FDA := Fisher linear discriminant analysis

Semi-supervised variant

SemiCCA

SELF[Sugiyama 2009]SemiMLR?

SemiPCA?

Semi-supervised variant of CCA• Graph Laplacian regularization

[Blaschko 2008]• Tikhonov regularization

[Hardoon 2004]• SemiCCA = generalization of

Tikhomov regularization(when considering kernelization)

Contact: Akisato Kimura (NTT CS Labs, akisato@ieee.org)

Method [Nakayama 2008]SemiCCA is applied here.

ConditionsDataset: VOC2008/2009 dataset

(removed all the bounding boxes)Labeled images = 500 (from 2008 training)Unlabeled images = 13743

(4096 from 2008 training,9647 from 2009 training/test)

Test images = 500 (from 2008 training)

Result

ConditionsSamples: 10000 samples drawn from a Gaussian topic model

--- Latent variable Z : drawn from the normal Gaussian--- Gaussian parameters for p(X|Z) and p(Y|Z) were drawn

randomly from the normal Gaussian for each trial--- Dimension: X=15, Y=20, Z=10

Generating unpaired samples:--- remove several Y samples with the following linear

discriminant function

Average evaluation score

Average trade-off parameter

Histograms of trade-off parameters

Discriminant boundary

Evaluation measure: Weighted sum of cosine distances--- “True” eigenvectors and eigenvalues: