Bayesian inference and latent variable models in machine learning (by Dmitry Vetrov)

Bayesian Inference and Latent Variable Models in

Machine LearningDmitry P. Vetrov

Head of Bayesian methods research group

http://bayesgroup.ru,

Faculty of Computer Science, HSE

Skoltech

Outline

Today

• Probabilistic modeling in Machine Learning

• Exponential class of distributions

• Learning with latent variables

• EM-algorithm

Next time

• Examples of models with discrete and continuous latent variables

• Extensions of EM-algorithm

• Stochastic optimization in EM framework

What is machine learning?

Simple example

Conditional and marginal distributions

Bayesian Framework

Frequentist vs. Bayesian frameworks

Bayesian Learning and Inference

Combining models

Maximal a posteriori (MAP) learning

Exponential class of distributions

Log-concavity of exponential class

Log-concavity of exponential class

Example: Gaussian distribution

Incomplete likelihood

Variational lower bound

EM-algorithm

EM-algorithm

EM-algorithm

EM-algorithm

EM-algorithm

EM-algorithm

EM-algorithm

Discrete T

Mixture of gaussians

Mixture of gaussians

Mixture of gaussians

Mixture of gaussians

Mixture of gaussians

Mixture of gaussians

Mixture of gaussians: formal description

EM-algorithm for mixture of gaussians

Continuous T

Example: PCA model

Advantages of EM PCA

Mixture of PCA

Example: Latent Dirichlet Allocation

LDA: formal description

General nature of EM-framework

Extending E-step

Examples of conjugate distributions

Crisp E-step

Variational E-step

Stochastic optimization

Stochastic EM

Summary: extensions of basic EM

Conclusion

Challenge

For those who’s interested

• Help Nick Carter to find the criminal who kidnapped lady Thun’s dog http://cmp.felk.cvut.cz/cmp/courses/recognition/Labs/em/index_en.html