An Asymptotic Analysis of Generative, Discriminative,
and Pseudolikelihood Estimators by
Percy Liang and Michael Jordan
(ICML 2008 )
Presented by Lihan He
ECE, Duke University
June 27, 2008
Introduction
Exponential family estimators
Generative
Fully discriminative
Pseudolikelihood discriminative
Asymptotic analysis
Experiments
Conclusions
Outline
Introduction
Data points are not considered to be drawn independently.
There are correlations between data points.
Given data , we have to
consider the joint distribution over all the data points.
Correspondingly, the overall likelihood is not the product of the
likelihood for each data point.
1 1 2 2( , ) {( , ), ( , ),..., ( , )}n nz x y x y x y x y
1 1 2 1 1 1( ) ( ,..., ) ( ) ( | )... ( | ,..., )n n np z p z z p z p z z p z z z
Introduction
Generative vs. Discriminative
Generative model: • A model for randomly generating observed data;• Learning a joint probability distribution over both observations and labels
1 1( , ) ( ,..., , ,..., )n np x y p x x y y
Discriminative model: • A model only of the label variables conditional on the observed data;• Learning a conditional distribution over labels given observations
1 1( | ) ( ,..., | ,..., )n np y x p y y x x
Introduction
Full Likelihood vs. Pseudolikelihood
Full likelihood:
Pseudolikelihood: • An approximation of the full likelihood;• Computationally more efficient.
( ) ( | for all { , } )i j i ji
p z p z z z z E
1 1 2 1 1 1( ) ( ,..., ) ( ) ( | )... ( | ,..., )n n np z p z z p z p z z p z z z
• Could be intractable;• Computationally inefficient.
A set of dependencies between data points
Estimators
Exponential Family Estimators
( ) exp{ ( ) ( )}Tp z z A for zZ
( , ) and z x y Z X Yx( ) :z features
: model parameters
( ) :A normalization
Example: conditional random field
Estimators
Composite Likelihood Estimators [Lindsay 1988]
One class of pseudolikelihood estimator;
Consists of a weighted sum of component likelihoods, each of which is
the probability of one subset of data points conditioned on another.
Partitions the output space (denoted by r) according to a fixed distribution
Pr, and obtains the component likelihood.
Defines criterion function
which reflects the quality of the estimator.
The maximum composite likelihood estimator
~( ) log ( | ( ))rr Pm z p z z r z Ee
ˆ [ ( )]z m zEˆ arg max
Estimators
Three estimators to be compared in the paper:
Generative:
one component
Fully discriminative:
one component
Pseudolikelihood discriminative:
for each data point, we have one component
( , )gr x y X Y
( , )dr x y x x Y
( , ) {( ', ') : ' , ' , ' for }i j jr x y x y x x y y y j i Y
Estimators
Risk Decomposition
Bayes risk *
*
( , )~( | ) [ log ( | )]
X Y pR H Y X E p Y X
have only finite data intrinsic suboptimality of the estimator
*~arg max ( )o
Z pm Z
EDefine unrelated to data samples z
Asymptotic Analysis
before
Well-specified model: , achieves O(n-1) convergence rate.Misspecified model: only fully discriminative estimator achieves O(n-1) rate.
Asymptotic Analysis
Experiments
Toy example: four-node binary-valued graphical model 1 2 1 2( , , , )z x x y y
True model:* * *
1 2 1 1 2 2 1 2 2 1( ) ( ) [ ( ) ( )] [ ( ) ( )]Tz y y x y x y x y x y 1 1 1 1 1
1 2 1 1 2 2( ) ( ) [ ( ) ( )]Tz y y x y x y 1 1 1
Learned model:
When , the learned model is well-specified;
When , the learned model is misspecified.
* 0 * 0
Experiments
* 0 well-specified
* 0.5 misspecified
20000n* *( ) 1g
* *( ) 1, 0h
Experiments
Part-of-speech (POS) Tagging:
Input: a sequence of words 1( ,..., )lx x x
Output: a sequence of POS tags , i.e. noun, verb,etc. (45 tags total)1( ,..., )ly y y
Specified model:
Node features : indicator functions of the form( , )node i iy x ( , )i iy a x b 1
Edge features : indicator functions of the form1( , )edge i iy y 1( , )i iy a y b 1
Training: Wall Street Journal, 38K sentences.
Testing: Wall Street Journal, 5.5K sentences, different sections from training.
Experiments
Use the learned generative model to sample 1000 training samples and 1000 test samples, as synthetic data.
Conclusions
When model is well-specified: Three estimators all achieve O(n-1) convergence rate;
There are no approximation error;
The asymptotic estimation error
generative < fully discriminative < pseudolikelihood
discriminative When model is misspecified:
Fully discriminative estimator still achieves O(n-1) convergence
rate, but the other two estimators achieve O(n-1/2) convergence rate ;
The approximation error and asymptotic estimation error for
fully discriminative estimator is lower than the generative estimator
and
the pseudolikelihood discriminative estimator.