# Mmse Estimation

• View
219

0

Embed Size (px)

### Text of Mmse Estimation

• 7/24/2019 Mmse Estimation

1/12

10 - 1 MMSE Estimation S. Lall, Stanford 2011.02.02.01

10 - MMSE Estimation

Estimation given a pdf

Minimizing the mean square error

The minimum mean square error (MMSE) estimator

The MMSE and the mean-variance decomposition

Example: uniform pdf on the triangle

Example: uniform pdf on an L-shaped region

Example: Gaussian

Posterior covariance

Bias

Estimating a linear function of the unknown MMSE and MAP estimation

10 - 2 MMSE Estimation S. Lall, Stanford 2011.02.02.01

Estimation given a PDF

Suppose x: Rn is a random variable with pdfpx.

One can predictorestimatethe outcome as follows

Givencost function c: Rn Rn R

pickestimatexto minimize E c(x,x)

We will look at the cost function

c(x,x) = kx xk2

Then themean square error(MSE) is

Ekx xk2

=

Z kx xk2px(x) dx

• 7/24/2019 Mmse Estimation

2/12

10 - 3 MMSE Estimation S. Lall, Stanford 2011.02.02.01

Minimizing the MSE

Lets find the minimum mean-square error (MMSE) estimate ofx; we need to solve

minx

Ekx xk2

We have

Ekx xk2

= E

(x x)T(x x)

= E

xTx 2xTx + xTx

= Ekxk2 2xTE x+ xTx

Differentiating with respect to xgives the optimal estimate

xmmse= Ex

10 - 4 MMSE Estimation S. Lall, Stanford 2011.02.02.01

The MMSE estimate

The minimum mean-square error estimate ofxis

xmmse= Ex

Its mean square error is

E

kx xmmsek

2

= trace cov(x)

since Ekx xmmsek

2

= Ekx Exk2

• 7/24/2019 Mmse Estimation

3/12

10 - 5 MMSE Estimation S. Lall, Stanford 2011.02.02.01

The mean-variance decomposition

We can interpret this via the MVD. For any random variable z, we have

Ekzk2

= E

kz E zk2

+ kE zk2

Applying this to z=x xgives

Ekx xk2

= E

kx E xk2

+ kx Exk2

The first term is thevarianceofx; it is the error wecannot remove

The second term is the biasofx; the best we can do is make this zero.

!2 !1 0 1 2!3

!2

!1

0

1

2

3

10 - 6 MMSE Estimation S. Lall, Stanford 2011.02.02.01

The estimation problem

Suppose x, y are random variables, with joint pdfp(x, y).

We measure y=ymeas.

We would like to find the MMSE estimate ofxgiveny = ymeas.

Theestimatoris a function : Rm Rn.

We measure y = ymeas, and estimate xby xest= (ymeas)

We would like to find the function which minimizes the cost function

J= Ek(y) xk2

• 7/24/2019 Mmse Estimation

4/12

10 - 7 MMSE Estimation S. Lall, Stanford 2011.02.02.01

Notation

Well use the following notation.

py is themarginalor inducedpdf ofy

py(y) = Z p(x, y) dx

p|y is the pdfconditionedon y

p|y(x, y) =p(x, y)

py(y)

10 - 8 MMSE Estimation S. Lall, Stanford 2011.02.02.01

The MMSE estimator

Themean-square-error conditioned on y is econd(y), given by

econd(y) =

Z k(y) xk2p|y(x, y) dx

Then the mean square error J is given by

J= E

econd(y)

because

J=

Z Z k(y) xk2p(x, y) dxdy

=

Z py(y) econd(y) dy

• 7/24/2019 Mmse Estimation

5/12

10 - 9 MMSE Estimation S. Lall, Stanford 2011.02.02.01

The MMSE estimator

We can write the MSE conditioned on y as

econd(ymeas) = Ek(y) xk2 | y=ymeas

For eachymeas, we can pick a value for (ymeas)

So we have an MMSE prediction problem for each ymeas

10 - 10 MMSE Estimation S. Lall, Stanford 2011.02.02.01

The MMSE estimator

Apply the MVD to z=(y) xconditioned on y=w to give

econd(w) = Ek(y) xk2 | y=w

= E

kx h(w)k2 | y= w

+ k(w) h(w)k2

whereh(w) is the mean ofxconditioned ony = w

h(w) = E(x | y=w)

To minimizeecond(w)we therefore pick

(w) =h(w)

• 7/24/2019 Mmse Estimation

6/12

10 - 11 MMSE Estimation S. Lall, Stanford 2011.02.02.01

The error of the MMSE estimator

With this choice of estimator

econd(w) = Ekx h(w)k2 | y = w

= trace cov(x | y = w)

10 - 12 MMSE Estimation S. Lall, Stanford 2011.02.02.01

Summary: the MMSE estimator

The MMSE estimator is

mmse(ymeas) = E(x | y = ymeas)

econd(ymeas) = trace cov(x | y = ymeas)

We often write

xmmse= mmse(ymeas) econd(ymeas) = Ekx xmmsek

2 y=ymeas

The estimate only depends on the conditional pdfofx | y = ymeas

The means and covariance are those of theconditional pdf

The above formulae give the MMSE estimate for any pdfonxand y

• 7/24/2019 Mmse Estimation

7/12

0 0.5 1

0

0.5

1

10 - 13 MMSE Estimation S. Lall, Stanford 2011.02.02.01

Example: MMSE estimation

(x, y)are uniformly distributed on the triangle A.i.e., the pdf is

p(x, y) =

(2 if(x, y) A

0 otherwise

the conditional distribution ofx | y = ymeasis uniform on[ymeas, 1]

the MMSE estimator ofx given y = ymeas is

xmmse= E(x | y = ymeas) =1 + ymeas

2

the conditional mean square error of this estimate is

Ekx xmmsek

2 y = ymeas= 1

12(ymeas 1)

2

0 0.5 1

0

0.5

1

10 - 14 MMSE Estimation S. Lall, Stanford 2011.02.02.01

Example: MMSE estimation

The mean square error is

Ekmmse(y) xk

2

= E

econd(y)

=Z 1

x=0

Z xy=0p(x, y) econd(y) dy dx

=

Z 1

x=0

Z xy=0

1

6(y 1)2 dy dx

= 1

24

• 7/24/2019 Mmse Estimation

8/12

0 0.5 1

0

0.5

1

10 - 15 MMSE Estimation S. Lall, Stanford 2011.02.02.01

Example: MMSE estimation

(x, y)are uniformly distributed on the L-shapedregionA, i.e., the pdf is

p(x, y) =

(4

3 if(x, y) A

0 otherwise

the MMSE estimator ofx given y = ymeas is

xmmse= E(x | y = ymeas) =

(1

2 if0 ymeas

1

2

3

4 if 1

2< ymeas 1

the mean square error of this estimate is

Ekx xmmsek

2 y=ymeas=

( 1

12 if0 ymeas

1

2

1

48 if 1

2< ymeas 1

0 0.5 1

0

0.5

1

10 - 16 MMSE Estimation S. Lall, Stanford 2011.02.02.01

Example: MMSE estimation

The mean square error is

Ekmmse(y) xk

2

= E

econd(y)

=Z

Ap(x, y) econd(y) dy dx

=

Z 1

x=0

Z 12

y=0

1

12p(x, y) dy dx+

Z 1

x=12

Z 12

y=12

1

48p(x, y) dy dx

= 1

16

• 7/24/2019 Mmse Estimation

9/12

!3 !2 !1 0 1 2 30

0.05

0.1

0.15

0.2

0.25

0.3

0.35

!3 !2 !1 0 1 2 3!3

!2

!1

0

1

2

3

10 - 17 MMSE Estimation S. Lall, Stanford 2011.02.02.01

MMSE estimation for Gaussians

Suppose

xy

N(,)where

= xy = x xy

Txy y

We know that the conditional density ofx | y = ymeas isN(1, 1)where

1= x+ xy1y (ymeas y)

1= x xy1y

Txy

10 - 18 MMSE Estimation S. Lall, Stanford 2011.02.02.01

MMSE estimation for Gaussians

The MMSE estimator when x, y are jointly Gaussian is

mmse(ymeas) =x+ xy1y (ymeas y)

econd(ymeas) = tracex xy

1y

Txy

The conditional MSE econd(y)is independent ofy; a special property of Gaussians

Hence the optimum MSE achieved is

Ekmmse(y) xk

2

= tracex xy

1y

Txy

The estimate xmmse is an affine function ofymeas

• 7/24/2019 Mmse Estimation

10/12

10 - 19 MMSE Estimation S. Lall, Stanford 2011.02.02.01

Posterior covariance

Lets look at the error z= (y) x. We have

cov(z | y = ymeas) = cov

x| y= ymeas

We use this to find a confidence region for the estimate

cov(x| y=ymeas

is called theposterior covariance

!3 !2 !1 0 1 2 3

!3

!2

!1

0

1

2

3

10 - 20 MMSE Estimation S. Lall, Stanford 2011.02.02.01

Example: MMSE for Gaussians

Here

xy

N(0,)with =

2.8 2.42.4 3

We measureymeas= 2

and the MMSE estimator is

mmse(ymeas) = 12122

ymeas

= 0.8ymeas= 1.6

The posterior covariance is Q = x xy1y yx = 0.88

Let 2 =QF12

(0.9), then 1.54and the confidence interval is

Prob

|x 1.6|

y = ymeas

= 0.9

• 7/24/2019 Mmse Estimation

11/12

10 - 21 MMSE Estimation S. Lall, Stanford 2011.02.02.01

Bias

The MMSE estimator is unbiased; that is the mean erroris zero.

Emmse(y) x

= 0

10 - 22 MMSE Estimation S. Lall, Stanford 2011.02.02.01

Estimating a linear function of the unknown

Suppose

p(x, y) is a pdf on x, y

q= C xis a random variable

we measure y and would like to estimate q

The optimal estimator is

qmmse= CE

x| y=ymeas

Because E

q| y = ymeas

= CE

x| y = ymeas

The optimal estimate ofCx is Cmultiplied by the optimal estimate ofx

Only works forlinear functionsofx

• 7/24/2019 Mmse Estimation

12/12

!3 !2 !1 0 1 2 3!3

!2

!1

0

1

2

3

10 - 23 MMSE Estimation S. Lall, Stan

Documents
Documents
Science
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents