-
LRA
DSP
LRA
DSP Estimation
Nonlinear Estimation
Linear & Polynomial Estimation
Minimum Mean Square Error Estimation:Wiener Filtering
Professor L R Arnaut © 1
-
LRA
DSP
LRA
DSP
Given:
Source generating nonobservable stochastic signals Xe.g., signal
+ noise, bits, characters, …
Stochastic LTI channel (extendable to LTV)e.g., 1/f noise from
resistor, ionospheric propagation, …
Observable outputs of channel, Yconstitutes a new random
variable, because stochastic channel
Find:“best” estimate of X, based on Y
Professor L R Arnaut © 2
Estimation Problem
X̂
-
LRA
DSP
LRA
DSP
Assume:If we know the distribution of X, , then we do not know Y
exactly, but we know its conditional distribution (=forward
probability),
NB: (=backward probability) is unknown!
Then:We know the joint distribution of X and Y:
Now: design optimal estimator for Xi.e., design system that out
of all possible transfer functions g(.) it selects the one gopt(.)
for which
is “closest” to xProfessor L R Arnaut © 3
Optimal Estimation
( )xXyf XY =||
)()|(),( |, xfxXyfyxf XXYYX ⋅==
)(xfX
)|(| yYxf YX =
)(ˆ ygx opt=
-
LRA
DSP
LRA
DSP
Criterion: minimise the mean squared deviation
Since , we should minimise it:
Thus,
Professor L R Arnaut © 4
Optimal MMSE Estimator
[ ]
∫=∫ ∫ =−=
∫∫ −=−
dyyKyf
dxyYxfygxyfdy
dydxyxfygxXX
Y
YXY
YX
)()(
)|()]([)(
),()(])ˆ[(E
|2
,22
&
0)( ≥yK
)()|(E)(2)|(E
)|()]([)(22
|2
ygyYXygyYX
dxyYxfygxyK YX+=−==
∫ =−=
⇔ )|(E)( yYXyg ==yields best (MMSE) estimate
0)( =dg
ydK
-
LRA
DSP
LRA
DSP
Associated mean squared deviation (error) for this choice of
g(.):
Using Schwartz inequality ⇒Optimal estimator is in general
nonlinear
Professor L R Arnaut © 5
Optimal MMSE Estimation Error
∫ =−= dyyfyYXEXE Y )()|()(222
optε
22opt0 Xσε ≤≤
-
LRA
DSP
LRA
DSP Optimal MMSE Estimation Why MMSE?
With this choice: estimation error is orthogonal to g(y)Proof:
projection of MMSE error onto g(y):
[ ]dydxyxfygyX
dydxyxfygxdydxyxfygyXx
YX
YXYX
),()()|(E),()(),()()|(E
,
,,
∫∫−∫∫=∫∫ −
dyyfygyXdydxyxfygx YYX ∫−∫∫= )()()|(E),()(
,dyyfygdxyxfxdydxyxfygx YYXYX ∫ ∫−∫∫= )()()|(),()( |,
dydxyxfygxdydxyxfygx YXYX ∫∫−∫∫= ),()(),()( ,,0=
Professor L R Arnaut © 6
-
LRA
DSP
LRA
DSP Optimal Estimation Special cases:
(i) X and Y are statistically independenti.e., observation of Y
does not teach us anything about X:
Corollary: if X andY are dependent, then optimal estimate of X
after observation of Y yields MSE that is never larger than if we
choose without observing Y
(ii) Deterministic lossless channel:
Professor L R Arnaut © 7
)(),( 1 YXXY −== φφ
),(E)|(E)( XyYXyg === 22opt Xσε =
)(Eˆ xx =
0,)(ˆ 2opt1 === − εφ XYX
-
LRA
DSP
LRA
DSP
Optimal estimator often difficult to realiseSuboptimal:
approximate by best linearestimator:
Then: yields
Using Schwartz inequality (do!) ⇒
Professor L R Arnaut © 8
Linear Estimator: Wiener Filter
βα += yx̂
⎪⎩
⎪⎨⎧
−=
=
)(E)(E
,
YX
rY
X
αβσσα
⎪⎪⎩
⎪⎪⎨
⎧
=∂
−−∂
=∂
−−∂
0])[(E
0])[(E
2
2
ββα
αβα
YX
YX
)1( 222min rX −=σε
22min
2opt Xσεε ≤≤
YX
XYrσσ
σ=
)|(E yYX =
-
LRA
DSP
LRA
DSP
For Gaussian X and Y (do!):
i.e., linear!
Thus, linear estimate for Gaussian random variable is its
optimal estimate
Professor L R Arnaut © 9
Optimal Linear Estimation
[ ] )(E)(E)|(E XYyryYXY
X +−==σσ
-
LRA
DSP
LRA
DSP
Extension of suboptimal linear estimator to quadratic:
i.e., a linear problem of solving for α, β, γ
Now, 3rd- and 4th-order moments are neededFor Gaussian X, Y:
3rd-order moments =0; 4th-order moments calculable from 2nd-order
moments via Isserlis’s theorem
Professor L R Arnaut © 10
Polynomial Estimation
γβα ++= yyx 2ˆ
⎪⎪⎩
⎪⎪⎨
⎧
=++
=++
=++
XYY
YXYYY
YXYYY
γβα
γβα
γβα
2
33
2234
-
LRA
DSP
LRA
DSP Optimum LSE Filters
Minimum Mean Square Error Estimation:IIR Wiener Filtering
Professor L R Arnaut © 1
-
LRA
DSP
LRA
DSP Optimum LSE Filters
Innovations Process & Whitening Filter
Professor L R Arnaut © 2
-
LRA
DSP
LRA
DSP Innovations ProcessProblem statement:
Find:(I) linear causal filter such that a given WSS random
process x(t) as input yields a white noise output signal: whitening
filter
(II) (inverse problem) linear causal filter such that white
noise input process produces a given WSS random process x(t) as its
output (“correlator”)
We shall show: transfer functions of both filters are each
other’s inverse
Practical significance: non-ideal WSS process (non-vanishing
correlation) can be represented as a linearly filtered ideal
(white) WSS process
Professor L R Arnaut © 3
-
LRA
DSP
LRA
DSP Innovations ProcessDefine:
: wide-sense stationary (WSS) random process
: autocorrelation sequence of : power spectrum of
Assume is analytic (holomorph) in an annular region in the
z-plane comprising the unit circle
Professor L R Arnaut © 4
)}({ nx
)}({ mxxγ)( fxΓ )}({ nx
)}({ nx
)(ln zxΓ
21 1 rr
-
LRA
DSP
LRA
DSP Innovations ProcessParenthetically, Wiener-Khinchine theorem
(for z-transforms):
Expansion of in Laurent series w.r.t. z=0 (generalization of
Taylor series in z-plane):
Alternative interpretation of : the function is the z-transform
of
∑=Γ∞+
−∞=
−
m
mxxx zmz )()( γ
)(ln zxΓ
∑=Γ∞+
−∞=
−
m
mx zmvz )()(ln
)}({ mv)(ln zxΓ∑=Γ∞+
−∞=
−
m
mx zmvz )()(ln
Professor L R Arnaut © 5
-
LRA
DSP
LRA
DSP Innovations Process
Professor L R Arnaut © 6
For real & even: Fourier coefficients are even:
Factorisation:
Thus,
where
{v(m)} = cepstrum of {γxx(m)} and Γx(f )
∑=Γ∞+
−∞=
−
m
mx zmvz )()(ln
( ) ∑+∑ −+= ∞+=
−∞+
=
−−
11'
'1 )()'()0(m
m
m
mzmvzmvv
( ) ∑+∑+= ∞+=
−∞+
=
−−
11'
'12 )()'(lnm
m
m
mw zmvzmvσ
)( fxΓ
( )( ) ( ) )(2expln)( 5.0 5.0 mvdffmjfmv x −=∫ Γ= − π
)()()(exp)( 12 zHzHzmvz wm
mx
−∞+
−∞=
− =⎟⎠⎞
⎜⎝⎛ ∑=Γ σ
⎟⎠⎞
⎜⎝⎛ ∑=
∞+
=
−
1)(exp)(
m
mzmvzH
-
LRA
DSP
LRA
DSP Innovations ProcessInterpretation:
for : causal response
for : non-causal
In , is analytic (no poles) with Laurent series reducing to a
Taylor series:
If is analytic in , then is analytic in
⎟⎠⎞
⎜⎝⎛ ∑=
∞+
=
−
1)(exp)(
m
mzmvzH 1||1
-
LRA
DSP
LRA
DSP Innovations ProcessFurther interpretation:
From , i.e., this represents a linear filter having input with
power spectral density , output with power spectral density Since
is causal, represents a linear filter with input and as output the
innovation process : whitening filter
Professor L R Arnaut © 8
)(zH
)()()( 12 −=Γ zHzHz wx σ22*2 )()()()( fHfHfHf wwx σσ ==Γ
)}({ nw)}({ nx )( fxΓ
2)( ww f σ=Γ
H(z)w(n) x(n)
)(zH
)}({ nw
)}({ nx
22 )(/)( fHfxw Γ=σ
)(/1 zH
1/H(z) w(n)x(n))(/1 zH
-
LRA
DSP
LRA
DSP Optimum LSE Filters
IIR Wiener Filter
Professor L R Arnaut © 9
-
LRA
DSP
LRA
DSP IIR Wiener Filter
( ) ⎟⎟⎠
⎞⎜⎜⎝
⎛∑ −−=∞+
=
2
0
2 )()()()(k
knxkhndEneE
MMSE problem:
Filter cffs. to be chosen as solutions of infinite linear system
(Wiener-Hopf equation)
Problem: system cannot be solved with z-transform for only
(Wiener-Khinchine N/A)
Solution: use auxiliary equivalent representation of input ,
which transforms it into sequence defined over to which z-transform
can be applied: innovations representation
∑ ≥=−∞+
=00),()()(
kdxxx mmkmkh γγ
)(kh
)(nx
0≥m
+∞
-
LRA
DSP
LRA
DSP IIR Wiener FilterInnovations representation of x(n):
Idea: Cascading whitening filter 1/G(z) with second filter Q(z)
such that cascade is optimum Wiener filter H(z) for x(n):
(Wiener-Hopf)Since , filter cffs. are
)()()( 12
Professor L R Arnaut © 11
−=Γ zGzGz wx σ1/G(z)x(n)
w(n)Q(z)
y(n)
∑ ≥=−∞+
=00),()()(
kdwww mmkmkq γγ
H(z)
,)()()()()(00
∑ −=∑ −=∞+
=
∞+
= kkknxkhknwkqny )(/)()( zGzQzH =
( )kmkm www −=− δσγ 2)(
0,)()( 2 ≥= mmmq
w
dw
σγ
-
LRA
DSP
LRA
DSP IIR Wiener FilterRepresentation of in terms of :
hence
Professor L R Arnaut © 12
)(mdwγ (.)dxγ
∑ ==∞+
=
−
0 )(1)()(
k
k
zGzkvzV,)()()(
0∑ −=∞+
=kknxkvnw
V(z)=1/G(z) w(n)x(n)
)()(
)]()([)(
)]()([)(
0
*
0
*
mkmv
mknxndEmv
knwndEk
dxm
m
dw
+∑=
−−∑=
−=
∞+
=
∞+
=
γ
γ
-
LRA
DSP
LRA
DSP IIR Wiener Filterz-transformation:
Extracting causal part:
⇒ Optimum IIR Wiener filter:(causal)
Professor L R Arnaut © 13
)(
10
)(
)()()()(
+
−
∞+
=
−+⎥⎦
⎤⎢⎣
⎡ Γ≡∑=Γ
zGzzkz dx
k
kdwdw γ
)(
12 )()(
)(1)(
+
− ⎥⎦
⎤⎢⎣
⎡ Γ=
zGz
zGzH dx
wσ
( ))()()(
)()(
)()(
)()()()(
11
0
)(
0
0
−−
∞+
=
∞+
−∞=+
+−+
∞+
=
∞+
−∞=
−
∞+
−∞=
−∞+
=
∞+
−∞=
−
Γ≡Γ=
∑ ∑ +=
∑ ∑ +=
∑ ⎥⎦⎤
⎢⎣⎡ ∑ +=∑=Γ
zGzzzV
zmkzmv
zmkmv
zmkmvzkz
dxdx
m mk
mkdx
m
m k
kdx
k
k
mdx
k
kdwdw
γ
γ
γγ
Wiener_v2.pdfIIRWiener_v2
