The Multivariate Gaussian Jian Zhao. Outline What is multivariate Gaussian? Parameterizations...

The Multivariate Gaussian

Jian Zhao

Outline

What is multivariate Gaussian? Parameterizations Mathematical Preparation Joint distributions, Marginalization and

conditioning Maximum likelihood estimation

What is multivariate Gaussian?

Where x is a n*1 vector, ∑ is an n*n, symmetric matrix

11/ 2/ 2

1 1( | , ) exp{ ( ) ( )}

2(2 )T

np x x x

2

1 1 2

211 2 2

,

,

x x x

x x x

Geometrical interpret

Exclude the normalization factor and exponential function, we look the quadratic form

1( ) ( )Tx x

This is a quadratic form. If we set it to a constant value c, considering the case that n=2, we obtained

22

2 211 1 12 1 2 22a x a x x a x

Geometrical interpret (continued)

This is a ellipse with the coordinate x1 and x2.

Thus we can easily image that when n increases the ellipse became higher dimension ellipsoids

Geometrical interpret (continued)

Thus we get a Gaussian bump in n+1 dimension

Where the level surfaces of the Gaussian bump are ellipsoids oriented along the eigenvectors of ∑

Parameterization

1

1

( )

( )( )TE x

E x x

Another type of parameterization, putting it into the form of exponential family

1( log(2 ) log | | )

2Ta n

1( | , ) exp{ }

2T Tp x a x x x

Mathematical preparation

In order to get the marginalization and conditioning of the partitioned multivariate Gaussian distribution, we need the theory of block diagonalizing a partitioned matrix

In order to maximum the likelihood estimation, we need the knowledge of traces of the matrix.

Partitioned matrices

Consider a general patitioned matrix

E FM

G H

To zero out the upper-right-hand and lower-left-hand corner of M, we can premutiply and postmultiply matrices in the following form

1

1

0 0

0 0

I FH E F I E FH G

I G H H G I H

Partitioned matrices (continued)

Define Schur complement of Matrix M with respect to H , denote M/H as the term 1E FH G

Since1 1 1 1 1

1 1

( )XYZ Z Y X W

Y ZW X

1 1 1

1 1

1 1 1

1 1 1 1 1 1

0 ( / ) 0

0 0

( / ) ( / )

( / ) ( / )

E F I M H I FH

G H H G I H I

M H M H FH

H G M H H H G M H FH

So

Partitioned matrices (continued)

Note that we could alternatively have decomposed the matrix m in terms of E and M/E, yielding the following expression for the inverse.

1 1

1 1 1

0 0

0 00 0

I E F E F EI E F I E F

GE I G H GE F H GE F HI I

1 1 1

11

1 1 1

11

1 1 1 1 1 1

1 1 1

00

0 0 ( / )

0( / )

0 ( / )

( / ) ( / )

( / ) ( / )

E F II E F E

G H GE II M E

IE E F M E

GE IM E

E E F M E GE E F M E

M E GE M E

Partitioned matrices (continued)

Thus we get

1 1 1 1 1 1 1( ) ( )E FH G E E F H GE F GE 1 1 1 1 1 1( ) ( )E FH G FH E F H GE F

At the same time we get the conclusion

|M| = |M/H||H|

Theory of traces

[ ]iii

i i

tr A a

[ ] [ ]T T Tx Ax tr x Ax tr xx A

Define

It has the following properties:

tr[ABC] = tr[CAB] = tr[BCA]

Theory of traces (continued)

[ ] kl lk jik lij ij

tr AB a b ba a

[ ] Ttr BA BA

so

[ ] [ ]T T T T Tx Ax tr xx A xx xxA A

Theory of traces (continued)

We want to show that

Since

Recall

This is equivalent to prove

Noting that

log | | TA AA

1

log | | | |ij ij

A Aa A a

1 1

| |A A

A

| |ij

A Aa

| | ( 1)i j ij ijj

A a M

Joint distributions, Marginalization and conditioning

1

2

11 12

21 22

1

1 1 11 12 1 1

1/ 2( ) / 22 2 21 22 2 2

1 1( | , ) exp{ }

2(2 )

T

p q

x xp x

x x

We partition the n by 1 vector x into p by 1 and q by 1, which n = p + q

Marginalization and conditioning

1

1 1 11 12 1 1

2 2 21 22 2 2

11 1 22

12 2 22 21 22

11 112 22

2 2

1 11 1 12 22 2 2 22

1exp{ }

2

0 ( / ) 01exp{

2 0

}0

1exp ( ( )) ( / ) (

2

T

T

T

x x

x x

x I

x I

xI

xI

x x x

11 1 12 22 2 2

12 2 22 2 2

( ))

1exp ( ) ( )

2T

x

x x

Normalization factor

1/ 2 ( ) / 2 1/ 2( ) / 222 22

/ 2 1/ 2 / 2 1/ 222 22

1 1

(2 ) (| / || |)(2 )

1 1

(2 ) (| / |) (2 ) (| |)

p qp q

p q

Marginalization and conditioning

Thus

12 2 2 22 2 2/ 2 1/ 2

22

1 1( ) exp ( ) ( )

(2 ) (| |) 2T

qp x x x

1 2 / 2 1/ 222

1 1 11 1 12 22 2 2 22 1 1 12 22 2 2

1( | )

(2 ) (| / |)

1exp ( ( )) ( / ) ( ( ))

2

p

T

p x x

x x x x

Marginalization & Conditioning

Marginalization

Conditioning

2 2

2 22

m

m

11|2 1 12 22 2 2

11|2 11 12 22 21

( )c

c

x

In another form

1|2 1 12 2

1|2 11

c

c

x

Marginalization

Conditioning

12 2 21 11 1

12 22 21 11 12

m

m

Maximum likelihood estimation

1

1

1( , | ) log | | ( ) ( )

2 2

NT

i ii

Nl D x x

1

1

( )N

Ti

i

lx

Calculating µ

1

1ˆ

N

ii

xN

Taking derivative with respect to µ

Setting to zero

Estimate ∑

We need to take the derivative with respect to ∑

according to the property of traces

1

1 1

1 1

1( | ) log | | ( ) ( )

2 2

1log | | [( ) ( )]

2 2

1log | | [( )( ) ]

2 2

T

n

T

n

T

n

Nl D x x

Ntr x x

Ntr x x

1

1( )( )

2 2T

n nn

l Nx x

Estimate ∑

Thus the maximum likelihood estimator is

The maximum likelihood estimator of canonical parameters are

1ˆ ( )( )TML n nn

x xN

1

1

ˆˆ

ˆˆ ˆ

ML

ML ML