Mmse Estimation

  • View
    219

  • Download
    0

Embed Size (px)

Text of Mmse Estimation

  • 7/24/2019 Mmse Estimation

    1/12

    10 - 1 MMSE Estimation S. Lall, Stanford 2011.02.02.01

    10 - MMSE Estimation

    Estimation given a pdf

    Minimizing the mean square error

    The minimum mean square error (MMSE) estimator

    The MMSE and the mean-variance decomposition

    Example: uniform pdf on the triangle

    Example: uniform pdf on an L-shaped region

    Example: Gaussian

    Posterior covariance

    Bias

    Estimating a linear function of the unknown MMSE and MAP estimation

    10 - 2 MMSE Estimation S. Lall, Stanford 2011.02.02.01

    Estimation given a PDF

    Suppose x: Rn is a random variable with pdfpx.

    One can predictorestimatethe outcome as follows

    Givencost function c: Rn Rn R

    pickestimatexto minimize E c(x,x)

    We will look at the cost function

    c(x,x) = kx xk2

    Then themean square error(MSE) is

    Ekx xk2

    =

    Z kx xk2px(x) dx

  • 7/24/2019 Mmse Estimation

    2/12

    10 - 3 MMSE Estimation S. Lall, Stanford 2011.02.02.01

    Minimizing the MSE

    Lets find the minimum mean-square error (MMSE) estimate ofx; we need to solve

    minx

    Ekx xk2

    We have

    Ekx xk2

    = E

    (x x)T(x x)

    = E

    xTx 2xTx + xTx

    = Ekxk2 2xTE x+ xTx

    Differentiating with respect to xgives the optimal estimate

    xmmse= Ex

    10 - 4 MMSE Estimation S. Lall, Stanford 2011.02.02.01

    The MMSE estimate

    The minimum mean-square error estimate ofxis

    xmmse= Ex

    Its mean square error is

    E

    kx xmmsek

    2

    = trace cov(x)

    since Ekx xmmsek

    2

    = Ekx Exk2

  • 7/24/2019 Mmse Estimation

    3/12

    10 - 5 MMSE Estimation S. Lall, Stanford 2011.02.02.01

    The mean-variance decomposition

    We can interpret this via the MVD. For any random variable z, we have

    Ekzk2

    = E

    kz E zk2

    + kE zk2

    Applying this to z=x xgives

    Ekx xk2

    = E

    kx E xk2

    + kx Exk2

    The first term is thevarianceofx; it is the error wecannot remove

    The second term is the biasofx; the best we can do is make this zero.

    !2 !1 0 1 2!3

    !2

    !1

    0

    1

    2

    3

    10 - 6 MMSE Estimation S. Lall, Stanford 2011.02.02.01

    The estimation problem

    Suppose x, y are random variables, with joint pdfp(x, y).

    We measure y=ymeas.

    We would like to find the MMSE estimate ofxgiveny = ymeas.

    Theestimatoris a function : Rm Rn.

    We measure y = ymeas, and estimate xby xest= (ymeas)

    We would like to find the function which minimizes the cost function

    J= Ek(y) xk2

  • 7/24/2019 Mmse Estimation

    4/12

    10 - 7 MMSE Estimation S. Lall, Stanford 2011.02.02.01

    Notation

    Well use the following notation.

    py is themarginalor inducedpdf ofy

    py(y) = Z p(x, y) dx

    p|y is the pdfconditionedon y

    p|y(x, y) =p(x, y)

    py(y)

    10 - 8 MMSE Estimation S. Lall, Stanford 2011.02.02.01

    The MMSE estimator

    Themean-square-error conditioned on y is econd(y), given by

    econd(y) =

    Z k(y) xk2p|y(x, y) dx

    Then the mean square error J is given by

    J= E

    econd(y)

    because

    J=

    Z Z k(y) xk2p(x, y) dxdy

    =

    Z py(y) econd(y) dy

  • 7/24/2019 Mmse Estimation

    5/12

    10 - 9 MMSE Estimation S. Lall, Stanford 2011.02.02.01

    The MMSE estimator

    We can write the MSE conditioned on y as

    econd(ymeas) = Ek(y) xk2 | y=ymeas

    For eachymeas, we can pick a value for (ymeas)

    So we have an MMSE prediction problem for each ymeas

    10 - 10 MMSE Estimation S. Lall, Stanford 2011.02.02.01

    The MMSE estimator

    Apply the MVD to z=(y) xconditioned on y=w to give

    econd(w) = Ek(y) xk2 | y=w

    = E

    kx h(w)k2 | y= w

    + k(w) h(w)k2

    whereh(w) is the mean ofxconditioned ony = w

    h(w) = E(x | y=w)

    To minimizeecond(w)we therefore pick

    (w) =h(w)

  • 7/24/2019 Mmse Estimation

    6/12

    10 - 11 MMSE Estimation S. Lall, Stanford 2011.02.02.01

    The error of the MMSE estimator

    With this choice of estimator

    econd(w) = Ekx h(w)k2 | y = w

    = trace cov(x | y = w)

    10 - 12 MMSE Estimation S. Lall, Stanford 2011.02.02.01

    Summary: the MMSE estimator

    The MMSE estimator is

    mmse(ymeas) = E(x | y = ymeas)

    econd(ymeas) = trace cov(x | y = ymeas)

    We often write

    xmmse= mmse(ymeas) econd(ymeas) = Ekx xmmsek

    2 y=ymeas

    The estimate only depends on the conditional pdfofx | y = ymeas

    The means and covariance are those of theconditional pdf

    The above formulae give the MMSE estimate for any pdfonxand y

  • 7/24/2019 Mmse Estimation

    7/12

    0 0.5 1

    0

    0.5

    1

    10 - 13 MMSE Estimation S. Lall, Stanford 2011.02.02.01

    Example: MMSE estimation

    (x, y)are uniformly distributed on the triangle A.i.e., the pdf is

    p(x, y) =

    (2 if(x, y) A

    0 otherwise

    the conditional distribution ofx | y = ymeasis uniform on[ymeas, 1]

    the MMSE estimator ofx given y = ymeas is

    xmmse= E(x | y = ymeas) =1 + ymeas

    2

    the conditional mean square error of this estimate is

    Ekx xmmsek

    2 y = ymeas= 1

    12(ymeas 1)

    2

    0 0.5 1

    0

    0.5

    1

    10 - 14 MMSE Estimation S. Lall, Stanford 2011.02.02.01

    Example: MMSE estimation

    The mean square error is

    Ekmmse(y) xk

    2

    = E

    econd(y)

    =Z 1

    x=0

    Z xy=0p(x, y) econd(y) dy dx

    =

    Z 1

    x=0

    Z xy=0

    1

    6(y 1)2 dy dx

    = 1

    24

  • 7/24/2019 Mmse Estimation

    8/12

    0 0.5 1

    0

    0.5

    1

    10 - 15 MMSE Estimation S. Lall, Stanford 2011.02.02.01

    Example: MMSE estimation

    (x, y)are uniformly distributed on the L-shapedregionA, i.e., the pdf is

    p(x, y) =

    (4

    3 if(x, y) A

    0 otherwise

    the MMSE estimator ofx given y = ymeas is

    xmmse= E(x | y = ymeas) =

    (1

    2 if0 ymeas

    1

    2

    3

    4 if 1

    2< ymeas 1

    the mean square error of this estimate is

    Ekx xmmsek

    2 y=ymeas=

    ( 1

    12 if0 ymeas

    1

    2

    1

    48 if 1

    2< ymeas 1

    0 0.5 1

    0

    0.5

    1

    10 - 16 MMSE Estimation S. Lall, Stanford 2011.02.02.01

    Example: MMSE estimation

    The mean square error is

    Ekmmse(y) xk

    2

    = E

    econd(y)

    =Z

    Ap(x, y) econd(y) dy dx

    =

    Z 1

    x=0

    Z 12

    y=0

    1

    12p(x, y) dy dx+

    Z 1

    x=12

    Z 12

    y=12

    1

    48p(x, y) dy dx

    = 1

    16

  • 7/24/2019 Mmse Estimation

    9/12

    !3 !2 !1 0 1 2 30

    0.05

    0.1

    0.15

    0.2

    0.25

    0.3

    0.35

    !3 !2 !1 0 1 2 3!3

    !2

    !1

    0

    1

    2

    3

    10 - 17 MMSE Estimation S. Lall, Stanford 2011.02.02.01

    MMSE estimation for Gaussians

    Suppose

    xy

    N(,)where

    = xy = x xy

    Txy y

    We know that the conditional density ofx | y = ymeas isN(1, 1)where

    1= x+ xy1y (ymeas y)

    1= x xy1y

    Txy

    10 - 18 MMSE Estimation S. Lall, Stanford 2011.02.02.01

    MMSE estimation for Gaussians

    The MMSE estimator when x, y are jointly Gaussian is

    mmse(ymeas) =x+ xy1y (ymeas y)

    econd(ymeas) = tracex xy

    1y

    Txy

    The conditional MSE econd(y)is independent ofy; a special property of Gaussians

    Hence the optimum MSE achieved is

    Ekmmse(y) xk

    2

    = tracex xy

    1y

    Txy

    The estimate xmmse is an affine function ofymeas

  • 7/24/2019 Mmse Estimation

    10/12

    10 - 19 MMSE Estimation S. Lall, Stanford 2011.02.02.01

    Posterior covariance

    Lets look at the error z= (y) x. We have

    cov(z | y = ymeas) = cov

    x| y= ymeas

    We use this to find a confidence region for the estimate

    cov(x| y=ymeas

    is called theposterior covariance

    !3 !2 !1 0 1 2 3

    !3

    !2

    !1

    0

    1

    2

    3

    10 - 20 MMSE Estimation S. Lall, Stanford 2011.02.02.01

    Example: MMSE for Gaussians

    Here

    xy

    N(0,)with =

    2.8 2.42.4 3

    We measureymeas= 2

    and the MMSE estimator is

    mmse(ymeas) = 12122

    ymeas

    = 0.8ymeas= 1.6

    The posterior covariance is Q = x xy1y yx = 0.88

    Let 2 =QF12

    (0.9), then 1.54and the confidence interval is

    Prob

    |x 1.6|

    y = ymeas

    = 0.9

  • 7/24/2019 Mmse Estimation

    11/12

    10 - 21 MMSE Estimation S. Lall, Stanford 2011.02.02.01

    Bias

    The MMSE estimator is unbiased; that is the mean erroris zero.

    Emmse(y) x

    = 0

    10 - 22 MMSE Estimation S. Lall, Stanford 2011.02.02.01

    Estimating a linear function of the unknown

    Suppose

    p(x, y) is a pdf on x, y

    q= C xis a random variable

    we measure y and would like to estimate q

    The optimal estimator is

    qmmse= CE

    x| y=ymeas

    Because E

    q| y = ymeas

    = CE

    x| y = ymeas

    The optimal estimate ofCx is Cmultiplied by the optimal estimate ofx

    Only works forlinear functionsofx

  • 7/24/2019 Mmse Estimation

    12/12

    !3 !2 !1 0 1 2 3!3

    !2

    !1

    0

    1

    2

    3

    10 - 23 MMSE Estimation S. Lall, Stan