faculty.nps.edufaculty.nps.edu/fargues/teaching/ec4440/SpringFY09/ec4440-VI-SpFY... · 05/07/09 Ec44440.SpFY09/MPF - Section VI 2 References: [Therrien], [Manolakis], [Hayes]: M

05/07/09 Ec44440.SpFY09/MPF - Section VI 1

VI. Linear Models

[p. 3] Definition of AR, MA & ARMA models[p. 5] Relationship between AR, MA, & ARMA models[p. 10] How to find ARMA model parameters: Yule-Walker equation solutions[p. 29] AR modeling

- [p. 32] Autocorrelation method- [p. 41] covariance method- [p. 46] SVD-based autocorrelation/covariance method- [p. 55] Applications to signal frequency estimation techniques - [p. 59] Burg method- [p. 62] Filter order selection- [p. 65] Model validation- [p.66] Appendices


References:[Therrien], [Manolakis], [Hayes]: M. Hayes, Statistical Digital Signal Processing and Modeling, Wiley & Sons, 1996.[Haykin]: S. Haykin, Modern Filters, MacMillan, 1989.[Kumaresan82a]: R. Kumaresan & D. Tufts, “Estimating the parameters of exponentially damped sinusoids and pole-zero modeling in noise,” IEEE Trans. on ASSP, Vol. ASSP-30, no. 6, Dec. 1982, pp. 833 - 840.[Kumaresan82b]:D. Tufts & R. Kumaresan, “Estimation of frequencies of multiple sinusoids: Making linear prediction perform like maximum likelihood,” IEEE Proceedings, Vol. 70, Issue 9, Sept. 1982 pp. 975 – 989. [MathWorks]: www.mathworks.com


• To model a stationary RPuse -- correlation function

-- power spectral density -- parametric model

• Why are parametric models important?

(1) Useful in spectral estimation

-- if we have some knowledge about the process generating the RP, we can use it to select a model which is an approximation of the process

-- next, we can get an estimated PSD by using the model

(2) Useful for time series extrapolation and and interpolation.

(3) Useful for bandwidth compression. (speech processing applications)


( )( ) ( )( ) ( ) ( )

( ) ( )( )

( )

* 1

* 1

2

2

n w

jj j

x wj

B z B zS z S z

A z A z

B eS e S e

A e

ωω ω

ω

−

−=

=

• Types of models considered

( )zAb

w(n) x(n)

( )( )zAzB

w(n) x(n)

( )zBw(n) ( ) ( )inwbnxP

ii −= ∑

0=

(1) AR

(2) Moving average (MA)

(3) AR-MA (ARMA)

(IIR)

(FIR)

white noise

noise white:)(nwez j

+= ω ARMA representation:

- more flexible (has both poles and zeroes

- more difficult to study


• How to relate AR, MA, ARMA modelsWold decomposition theorem [1954]: Any wss RP can be decomposed into:

-- a random component

-- a “deterministic” random component (perfectly predictable based on the infinite past)

( ) ( ) ( )nxnxnx dr +=

( ) ( )inxanx di

id −−= ∑∞

1=

*where

{ai } are chosen so that one can perfectly match xd (n)


• How to relate AR, MA, ARMA models

• Consequences:

• if PSD is continuous; any AR or ARMA process can be represented by a MA model of infinite order.

• Kolmogorov’s Theorem [1941]: an ARMA or MA process can be represented by an AR model of infinite order.

• Practical Results: if we choose the wrong model, we can still get a good approximation by choosing a high enough order!!


( )( )

( ) 1−

1−

−∞

1

+1+1

=≡+1

1=

∑ azbzzH

zkczH ARMA

kAR ?

( ) 1−

1−−

∞

0= +1+1

≡∑ bzazzkc k

k

( ) 1−

1−

+1+1

=azbzzHEx: ARMA (1, 1)

numerator order denominator order

• How to represent ARMA (1, 1) with AR (∞)

⇓

compute inverse z-transform to get c(k)


( )⎩⎨⎧

≥=

=1 0

kk

kc

1

1Recall ,1

( )n

z aaz

a u n

− >−↓

( ) 1−

1−−

∞

0= +1+1

≡∑ bzazzkc k

k


To approximate ARMA (1, 1) with AR(P), then we need P so that:

c(P) =

needed order-P depends on _________

Trick: Choose P so that:


• How to find the model parameters: via solving “Yule-Walker” (Y-W) equations for (AR)

models

• via modified Y-W equations for (ARMA) models

w(n) x(n)ARMA

ai : AR parameters bi : MA parameters

00 0

( )( )( )

( ) , 1, ( )P P

i ii i

i i

B zH zA z

A z a z a B z b z− −

= =

=

= = =∑ ∑


( ) ( ) ( )

( ) ( )[ ] ( ) ( )[ ]

( ) ( )ikRbikRa

knxEknxE

inwbinxanx

wx

Q

iix

P

ii

Q

ii

P

ii

−=−

−=−

−=−+

∑∑

∑∑

0=0=

0=1=

• How to find the model parameters:

ai : AR parameters bi : MA parameters

00 0

( )( )( )

( ) , 1, ( )P P

i ii i

i i

B zH zA z

A z a z a B z b z− −

= =

=

= = =∑ ∑


( ) ( ) ( )

( ) ( )[ ] ( ) ( )[ ]

( ) ( )ikRbikRa

knxEknxE

inwbinxanx

wx

Q

iix

P

ii

Q

ii

P

ii

−=−

−=−

−=−+

∑∑

∑∑

0=0=

0=1=

( ) ( )( )

1 ncewith varia

noise whitewhen w(n) __________________________

*

==

−=

xw

xwwx

RRR

( )0 0

QP

i xi i

aR k i= =

− =∑ ∑Y-W equations

Use:


( ) ( )*

0___________ˆ

__________

Q

ii

c k bh i k+∞

= −∞

= − =

=

∑ ∑

If h(n) is causal

• Solution of modified Yule-Walker equations:

( )⎩⎨⎧ ≤≤

=ow

Qibib i

00

( ) ( )

( ) ( )

*

00 for

Q

ii k

k

c k bh i k k Q

k Qc k

b k Q

=

= − ≤

>=

= >

∑

with


( ) ( )kcikRa x

P

ii =−∑

0=

⇒

( )( )

0 _____________________ 0

_____________________1 _____________________ ____

_____________________ ____

k c

k Q c Qk Q

k Q P

= → =

= → =

= + → == + → =

matrix form


⇒

[ ] [ ][ ]

[ ]

⎥⎥⎥⎥⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢⎢⎢⎢⎢

⎣

⎡

=

⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢

⎣

⎡

⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢

⎣

⎡

0

0

10

1

_______

_______

_______

______________

____

____

________

____

____

____

____0

1 Qc

cc

a

a

R

p

x

P+Q unknown

Modified Y-W equations


( )( )

( )

( ) ( ) =++

==

==

≠=

−+∞→+∞→ 11

0

1limlim0

______0________0

0____

zab

zHh

cc

kkc

zz

[ ] [ ] [ ][ ] [ ] [ ]

[ ] [ ] [ ] ⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

=

⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢

⎣

⎡

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

−

+−−−

0

0_____1

01

10110

1

pxxx

xxx

xxx

a

a

RPRPR

PRRRPRRR

Solve for {ai , b0 } (Levinson algorithm)

Case I: AR model Q = 0


( )( )

( )

( )

( )⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡ 0

=

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡10

Qc

c

QR

RR

x

x

x

Case II: MA model P = 0

(1) identify c(k)(2) identify b(k) 2 methods available

option (a) to identify b(k):

Recall MA model is a FIR filter ⇒ h(n) = b(n)

( ) ( ) ( ) ( ) ( )kibibkbkbkc −=−= ∑ ***

And extract b(k) from c(k)

….. Which may be hard….


( )( )( )

( )( )( )

( ) ( )

( ) ( )( ) ( )

22

0* *

1 0 2 1*

2 0

0 00 01 1 1 12 2 2 2

i xx

x x

x x

c b RR cR c c bb b b RR c c b b R

⎧= =⎪⎡ ⎤ ⎡ ⎤ ⎪⎪⎢ ⎥ ⎢ ⎥= ⇒ = + =⎨⎢ ⎥ ⎢ ⎥

⎪⎢ ⎥ ⎢ ⎥ = =⎣ ⎦ ⎣ ⎦ ⎪⎪⎩

∑

and solve for b0 , b1 , b2 (may not be easy)

Ex: Q = 2, P = 0


Option (b) to identify b(k): use spectral factorization

Use the fact that

• Compute Sx (z) = Z [Rx (k)]

• Extract B(z); so that zeros are inside u.c. (may be hard)

( ) ( ) ( ) ( )1=

1= zSzBzBzS wx**


[ ] [ ] [ ][ ] [ ] [ ]

[ ] [ ] [ ]⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

−1−

−101−1−0

=

PQRQRQR

PRRRPRRR

R

xxx

xxx

xxx

B

Case III: ARMA model

Rewrite Y-W equations as:0

B

A

R ca

R⎡ ⎤ ⎡ ⎤

=⎢ ⎥ ⎢ ⎥⎣ ⎦⎣ ⎦

[ ]

[ ]

[ ] [ ]

[ ] ⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡ 1+−

+

1+=

QR

PQRQR

PQR

QRR

x

xx

x

x

A

[ ] [ ] [ ]0 , 1 ,...,T

c c c c Q⎡ ⎤= ⎣ ⎦

where RB is the (Q+1) ×

(P+1) matrix

RA is the P ×

(P+1) matrix

c is the vector


Case III: ARMA model

• Solve for a with RA a = 0

• Solve for c with RB a = c

• How to find b from c non-linear job!

Use spectral factorization (as for MA model)

0B

A

R ca

R⎡ ⎤ ⎡ ⎤

=⎢ ⎥ ⎢ ⎥⎣ ⎦⎣ ⎦


( ) ( )zSzS wx .

=

( )zA1( )zB

( )nx′( )nw ( )nx

residual process

Decompose ARMA into MA + AR portions

( ) ( )( ) ( )

( ) ( )

________*

_____*

_____*

x w

x x

x x

S z S z

S z S z

S z S z

′

′

′

=

=

→ =

Note:


Procedure: 1) find a from Y-W equations

2) recall ( ) ( ) ( ) ⎟⎠⎞

⎜⎝⎛ 1=′ *

*

zAzSzAzS xx

( ) ( ) ( ) ( )kRkaaikRkc xi

P

ix *=−= ∑

0=

( ) Qkkc ≤0≠

( )( ) _____________

_____________ ==

′ zSzC

x

Note:

( )for , 0, ,ka k a k P= =


( ) ( ) ⎟⎠⎞

⎜⎝⎛ 1

=′ **

zBzBzS x

• Find the spectral factorization of ( )zSx′

pick B(z): minimum phase

(to insure causality and invertibility)

• Note: Extension to the Levinson algorithm available to solve for (ai , bi ) for an ARMA (p, q) system is

possible (see appendix)


( ) 1−1

1−10

+1+

=zazbbzHEx: to be fit to Rx (k)

Rx (k)

k

3 21

4321





• AR ModelingAssume Rx (k) not known but estimated

Autocorrelation method Covariance method

SVD-based covariance /correlation method

BurgNotes:

(1)

• Performances are the same for large data records

• Strong differences for short data records.

(2) Easy to implement ⇒ “fast” algorithms implemented

(3) well suited for data with PSD behavior large at certain frequencies, i.e., with “peaky behavior”. Well suited for speech data.

I

IIIII


AR Methods lead to characterizing the data with a pole-only model, (can be done simply)

A(z)x(n) e(n)

White if model is accurateStochastic data

1/A(z)

Analysis (Modeling) Phase

e(n) x(n)

Synthesis Phase

Need stable filter only if x(n) is to be regenerated

If A(z) is good then: 2( ) should be "close" to |1/A( ) |j j

xS e eω ω Why?


Schemes Correlation CovarianceSVD-based correlation/covariance

Burg

Characteris tics

• Applies data windowing • Minimizesforward prediction errors via Levinson Algorithm

• No data windowing• Minimizes forward prediction errors

•Same windowing characteristics as basic correlation & covariance schemes• Minimizes forward prediction errors

• No data windowing• Minimizesforward & backward prediction errors

Advantages • Always gives a stable model• Fast Levinson implementations

• Better resolution for short data length than correlation

• Cleans up noise impacts• Easier to estimate frequency locations

• Good resolution for short data length • Always gives a stable model

Drawbacks • Poor performances for short data length• Frequency bias

• May produce unstable model• May have false peaks• Frequency bias

• Require good estimation of the matrix rank• Computationally expensive

• May suffer from spectral line splitting in high SNRs• Frequency bias• Lower accuracy for high-order models, or long data length

Overall AR Schemes Comparisons


Problem? What happens when is estimated from short data records?

Poor resolution!

( ) ( )

( ) ( ) ⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢

⎣

⎡

0

0=

⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢

⎣

⎡ 1

⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢

⎣

⎡

0

−0 2

1

εσ

pxx

xx

a

a

RPR

PRR

ˆˆ

ˆˆ

Autocorrelation Method

1) Replace true Rx (k) with estimated in Weiner Hopf eqs and solve using Levinson algorithm.

Solve:

( )kR xˆ

( )kRxˆ

A(z) is insured to be minimum phase

H(z) = 1/A(z) insured to be stable


2) Using the data directly. ⎥⎦

⎤⎢⎣

⎡

0=

2εσ

aR• Recall a is obtained by solving

• Correlation matrix can be estimated as: given we have {x(0), …, x(n−1)}

XHX =

x(0), x(1), …, x(p), …, x(n−1), 0, …

x(0), …, x(p−1), … x(n−1)

x(0), …, x(n−p−1) x(n−1)

x(0)

x(1) x(0)

x(p) x(0)

x(n−1) x(n−p−1)

x(n−p)

x(n−1)

*

Note: implicitly assume data is zero outside {x(0), …, x(n-1)}

Data windowing


XHX (1, 1) =

XHX (2, 2) =

XHX (2, 1) =

XHX =

x(0), x(1), …, x(p), …, x(n−1), 0, …

x(0), …, x(p−1), … x(n−1)

x(0), …, x(n−p−1) x(n−1)

x(0)

x(1) x(0)

x(p) x(0)

x(n−1) x(n−p−1)

x(n−p)

x(n−1)

*


⎥⎥

⎦

⎤

⎢⎢

⎣

⎡=optaXX H

• Error variance can be found as:

=2εσ


⎥⎦

⎤⎢⎣

⎡′1

=opt

opt aa

⎥⎦

⎤⎢⎣

⎡0

=S

aXXHopt

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡= 10 XXX

• Goal: Compute

by solving

(1)

• To solve (1): we need to split the matrix equation to

• get first

• compute S after (and )2εσ

opta′


⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎦

⎤

⎢⎢⎣

⎡= 10

1

0 XXXX

XXH

HH

then:


• Goal: need to solve:

( ) 0111 −=′ XXaXX HHopt

0 0 0 1

01 1 1

H H

H

H H

X X X XX X

X X X X

⎡ ⎤⎢ ⎥

= ⎢ ⎥⎢ ⎥⎣ ⎦

⎥⎦

⎤⎢⎣

⎡0

=S

aXX Hopt ⎥

⎦

⎤⎢⎣

⎡′1

=opt

opt aawhere


• Different procedures to estimate the correlation matrix exist, resulting in different X1 and X0

Define:

X =x(0) x(1)

x(p) x(0)

x(n−1) x(n−p−1) x(n−p)

x(n−1)

B4

B3

B1 B2


• Correlation method (B1 range of equations)

X =x(0) x(1)

x(p) x(0)

x(n−1) x(n−p−1) x(n−p)

x(n−1)

B4

B3

B1B2

• Resulting XHX is Toeplitz

• Data window is “assumed” implicitly to solve this case

-- reduces resolution of AR spectrum

-- always stable


• Covariance method (B2 range of equations)

B4

X =x(0) x(1)

x(p) x(0)

x(n−1) x(n−p−1) x(n−p)

x(n−1)

B3

B1B2

• Resulting XHX not Toeplitz (but fast solver 0(P2) exist)

• A(z) not minimum phase (may be unstable, happens rarely)

• Spectrum may have false peaks


• Pre-windowed method (B3 range of equations)

• Post-windowed method (B4 range of equations)

B4

X =

x(0) x(1)

x(p) x(0)

x(n−1) x(n−p−1) x(n−p)

x(n−1)

B3

B1B2


(1) using generic matrix approach

(2) using SVD-based approach

• 2 possible methods to solve system of equations:

(1) Generic approach

( ) 0111 −=′ XXaXX HHopt

(fast solvers exist in certain cases)

(2) SVD-based approach

generalized inverse

( ) 01

1−

11

0+1

−=′ XXXXaXX

HHopt


Example: Compare FFT and Correlation models, using generic approach

Speech signal1024 fft length14th order model correlation (Y-W) method

[Mathworks]


Example: Compare FFT and Correlation models, using generic approach

To generate AR parameters from data x, for filter of order 7[d1,p1] = aryule(x,7);

To visualize how good the model is: overlay the periodogram to 1/A(z) s = spectrum.periodogram; Hpsd = psd(s,x); plot(Hpsd); hold on; hp = plot(w1/pi,20*log10(2*abs(H1)/(2*pi)),'r');

% Scale to make one-sided P SD set(hp,'LineWidth',2); xlabel('Normalized frequency (\times \pi rad/sample)') ylabel('One-sided PSD (dB/rad/sample)') legend('PSD estimate of x','PSD of model output')

[www.mathworks.com]


∑=1HVUX

• Generalized inverse can be computed with the SVD

when is singular or ill-conditioned.( )11 XX H

• SVD computes:

PK× KK×PP×

PK×

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

00

000

=

1

∑pσ

σ

U V unitary UH U = U UH = I VH V = I

P columns

K rows

• SVD-based Autocorrelation/Covariance Method

Assume K>P


⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

00

000

=

1

∑pσ

σ

Assume K > P

• if rank (X1 ) = r < P ⇒ only r σi are nonzero

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

00

0=

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

00

0=

11

∑S

rσσ

P columns

K rows

• SVD-based Autocorrelation/Covariance Method


[ ]⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎦

⎤⎢⎣

⎡000

==

2

11

211 ∑H

H

H

V

VS

UUVUX

• Properties of the SVD.

HVSUX 1111 =⇒

Comment: Only the portions of U and V which are associated to the non-zero singular values are taken into account.

Contains the non-zero singular values


(1) Using the SVD for the generalized inverse: +1X

( ) ( )1 1

HH H H

H H H

IH H

X X U V U V

V U U V

V V

=

=

=

∑ ∑∑ ∑

∑ ∑

( ) ( ) ( )( )( ) PKUV

uVVV

VUVVXXXX

H HH

H H

I

HH

HHH HHH

×==

=

==

∑ ∑∑ ∑

∑∑ ∑

∑∑ ∑

−

−

−−+

;1

1

1

11

11

05/07/09 Ec44440.SpFY09/MPF - Section VI 50P

11

22

21

22

2

00 0

0 00 0 0

00

00 0 00

0

0 0

rr

r

σσ

σσ

σσ

σσ

σ

⎡ ⎤⎡ ⎤ ⎢ ⎥⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎢ ⎥⎣ ⎦⎡ ⎤⎢ ⎥⎢ ⎥=⎢ ⎥⎢ ⎥⎣ ⎦

P

( )

( )1

1

1 1

1 ;

H H

H H H

X X X X

V U K P

−+

−

=

= = ×∑ ∑ ∑ ∑


( )

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

010

101

=

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

00

0

⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢

⎣

⎡

010

101

=

2

1

1

2

22

21

1−

∑∑ ∑

r

rr

HH

σσ

σ

σ

σ

σσ

σ⇒

K

P

P

PK

P


( )1

12

1 01

0 10

H H

r

σσ

σ

−

⎡ ⎤⎢ ⎥⎢ ⎥=⎢ ⎥⎢ ⎥⎣ ⎦

∑ ∑ ∑⇒

( )

[ ]

H

H

H

HHH

USVX

U

US

VV

UVXXX

11

111

2

111

21

11

11

000

−+

−

+−

=

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎦

⎤⎢⎣

⎡=

= ∑

…

⇒

Comment: Only the portions of U and V which are associated with the non-zero values of X1 are taken into account.


0opt 1

1

01 1

1 0

0 1

H

r

a X X

V U Xσ

σ

+′ = −

⎡ ⎤⎢ ⎥= − ⎢ ⎥⎢ ⎥⎣ ⎦

• Generalized inverse can be used to stabilize the solution when data is noisy

Recall:

What if we have?σi

i


⎪⎩

⎪⎨⎧

==

∑++

+

HUVXdXh

• Assume K < P -- less equations than unknowns -- no unique solution

pick the one with minimum-norm

-- leads to same solution

Practical problems using the SVD

correct determination of the rank r of the SVD

( )( )

( )

( )

( )

( )( )

( )

( )

( ) ( )Nkpk

kp

kMDL

kpp

kp

kAIC

Nkp

P

kii

kpi

P

kk

Nkp

P

kii

kpi

P

kk

log221

1log

221

log

1

1

110

1

1

110

−+

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

−

Π−=

−+

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

−

Π−=

−

+=

−

+=

−

+=

−

+=

∑

∑

λ

λ

λ

λ


( ) ( )( )( )

exp 0.1 2 0.52

exp 0.2 2 0.42

y n j n

j n

π

π

⎡ ⎤= − +⎣ ⎦+ − +⎡ ⎤⎣ ⎦

Figure 1. Zeroes of B(z); 40 independent trials. Intersecting radial lines and arcs show the true locations of the two zeros. If the intersecting lines are inside the unit circle they correspond to exp(s1 ) and exp(s2 ) and if they are outside they correspond to exp(-s1

*) and exp(-s2* )

(a) Prony method, L = M = 2.

(b) Covariance method, L = 4.

(c) Covariance method, L = 8.

(d) Autocorrelation method, L = 4.

(e) Autocorrelation method, L = 8.

(f) Backward covariance method, L = 8. [Kumaresan82a]

b)

c) d)

e)f)

a)


Figure 1 (continued).

(g) Backward covariance method using SVD, L = 8.

(h) Backward covariance method using SVD, L = 18.

(i) Covariance method using SVD, L = 8. [Kumaresan82a]


( ) ( )[ ]

ex p 2 0 .5

ex p 2 0 .5 2

y n j n

j n

π

π

= ⎡ ⎤⎣ ⎦+

Figure 4. (a)The zeros of the prediction-error filter for 50 independent trials. True frequency locations are shown by arrows. Two unit amplitude-sinusoid as f1 = 0.52 Hz and f2 = 0.5 Hz. SNR −10 dB, L = 4, 25 data samples are used.

(b) L = 8.

(c) L = 12.

(d) L = 16.

(e) L = 18.

(f) L = 20.

[Kumaresan82b]


( ) ( )[ ]

ex p 2 0 .5

ex p 2 0 .5 2

y n j n

j n

π

π

= ⎡ ⎤⎣ ⎦+

Figure 4. (a)The zeros of the prediction- error filter for 50 independent trials. True frequency locations are shown by arrows. Two unit amplitude-sinusoiad as f1 = 0.52 Hz and f2 = 0.5 Hz. SNR −10 dB, L = 4, 25 data samples are used.

(a) L=4.

(b) L = 8.

(c) L = 12.

(d) L = 16.

(e) L = 18.

(f) L = 20. (g) L = 22.

[Kumaresan82b]

• SVD-based techniques


• Burg’s AlgorithmMethod

• Estimates the reflection coefficients which minimize the sum of forward and backward prediction errors.

• Guaranteed to generate a stable filter. Solved from reflection coefficients. Does not need correlation estimates

• AR parameters obtained from reflection coefficients with Levinson algorithm. Fats implementation exists

• Problems may occur for high SNRs or long data length(see equation details in appendix)


Example: Compare Correlation and Burg methods

Speech signal1024 fft length14th order correlation method correlation (Y-W) methodGeneric equation solver

[Mathworks]


Example: Compare FFT and Burg methods

Noisy sine signal1024 fft lengthBurg method

[Mathworks]


Note:

• Optimal model order obtained at minimum of above functions.

• Work well on data derived from AR signals, but not so much on MA, ARMA signals.

( )

( )

( )

( ) sps

s

sp

ps

sP

p ps

s

s

ps

NPNPMDL

PNPNPFPE

NPN

NpN

NPCAT

PNPAIC

lnlnMDL

FPE

CAT

lnAIC

+=

⎟⎟⎠

⎞⎜⎜⎝

⎛1−−1++

=

−−⎟

⎟⎠

⎞⎜⎜⎝

⎛ −1=

2+=

2

2

21=

2

2

∑

ε

ε

εε

ε

σ

σ

σσ

σ

• Filter Order Selectionchoose for order p of AR(p) system the one at which seems to stabilize

In practice this doesn’t work too well

Use model order selection criteria

2εσ

Akaike’s information-theoretic criteria

Parzen’s criterion autoregressive transfer

Akaike’s final prediction error

Schwartz & Rissanen’s minimum description length

Ns number of points


[Haykin]


[Haykin]


• How to evaluate the quality of the model

(i.e., Model validation)Goal: model validation process checks whether:

- model agrees sufficiently with data- model fits sufficiently well the data

Validation technique: checks whether the residual sequence e(n) is white, or white enough

How can we check a sequence is white?autocorrelation test

if distribution of estimated autocorrelation coefficients

is white ~N(0,1/N) , coefficients should fall within 95% CI bounds

( ) ( )/ (0)x xx k R k Rρ =

1.96 / , where is the data lengthN N±

Note: other checks are possible, see [Manolakis, p. 447 - 448] for examples


Appendix


[ ] [ ][ ]

[ ] [ ][ ][ ] [ ]( ) [ ]qrbaqr

qrb

qrqra

xx

xx

xx

xx

xx

11−1=

1−−=1

1+−=1

111

1

1

ρ

• Extension to Levinson algorithm is available to solve for (ai , bi ) for an ARMA (p, q) system:

[ ][ ] [ ] [ ]

[ ] [ ] [ ] [ ] 1−21=−+=

−+++−=

1−1−

1−

1−

1=1−∑

kiikbkaiaia

lkqrlakqrka

kkkk

k

xx

k

lkxx

k

,,,ρ

[ ][ ] [ ] [ ]

[ ] [ ] [ ] [ ][ ] [ ]( ) 1−

1−1−

1−

1−

1=1−

−1=1−21=−+=

−−+−−=

∑

kkkk

kkkk

k

xx

k

lkxx

k

kbkakiikakbibib

lkqrlbkqrkb

ρρ

ρ,,,

with the recursion for k = 2, 3, …, p given by

If k = p, exit; if not, continue.

(Modern Spectral Estimation, S. Kay)


( ) ( ) ( )( ) ( ) ( )

*1 1

*1 1

1 F P E

1 B P E

f f bp p p p

b b fp p p p

n n n

n n n

ε ε γ ε

ε ε γ ε− −

− −

⎧ = − − ←⎪⎨

′= − − ←⎪⎩

• Burg’s Algorithm DetailsMethod• Estimates the reflection coefficients which

minimize the sum of forward and backward prediction errors.

• Guaranteed to generate a stable filter.

• AR parameters obtained from reflection coefficients with Levinson algorithm.

pp γγ =′ *

Recall(1) (*) (2)

• It turns out:

• Compute:( ) ( ) ( )

( ) ( ) ( )

, 1 , , 1

, , 2 , 1

fp

bp

Te

fp

T

b b b bp p p p

e

e e p e p e n

e e p e n e n

⎡ ⎤⎢ ⎥= + −⎢ ⎥⎢ ⎥⎣ ⎦

⎡ ⎤⎢ ⎥= − −⎢ ⎥⎢ ⎥⎣ ⎦


• Using (*) from n = p, …, n−1 in (1) from n = p, …, n−1 in (2)

we get a matrix form of (*)

*1 1

1 1

f f bp p pp

b b fp p pp

e e

e e

ε γ

ε γ− −

− −

⎧ = −⎪⎨

= −⎪⎩


2

2

2

2

bp

fp

fbpS εε +=

• Burg’s method minimizes

( ) ( )1<+

2= 2

21−

2

21−

1−1−pb

pfp

bp

Hfp

pee

eeγγ

( ) ( )

( )

( )c

b

a

*

*

fpp

bp

bp

bpp

fp

fp

bp

fp

bp

Tfp

p

eee

eee

ee

ee

1−1−

1−1−

2

1−

2

1−

1−1−

−=⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

×

−=⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡ ×

+

2=

γ

γ

γ

0

0

0

xe

e

b

f

=⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

×=

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡ ×

which leads to

Algorithm:

which is iterated for p = 1, 2, …, P starting from the initial conditions

Note: |γp | < 1 ∀

p ⇒

filter A(z) is minimum phase ⇒

filter A(z) is insured to be stable

[Therrien, p.545]


Note: was used for AR(p) estimation

instead of

Adapt (*)

• Prediction variance:

• AR parameters computed from Levinson algorithm

Recall

(*)

( ) 2221−

−1=ppp εε σγσ

( ) ( ) ( )

( ) ( )⎪⎩

⎪⎨⎧

=−=

−=−= −−

−

1;

1,,1

0

1*1

nn

nn

ninn

ni

ni

aa

niaaa

γ

γ

( ) ( )inxanxP

ii −−= ∑

0=

ˆ

( ) ( )inxanxP

ii −−= ∑

=0

*ˆ

*ii aa →

( ) ( ) ( )

( ) ( )

1 1* *

*0

1, , 1

; 1

n n ni i n n i

n nn n

a a a i n

a a

γ

γ

− −−

⎧ = − = −⎪⎨

= − =⎪⎩


( ) ( ) ( )

( ) ( )

1 1* *

*0

1, , 1

; 1

n n ni i n n i

n nn n

a a a i n

a a

γ

γ

− −−

⎧ = − = −⎪⎨

=− =⎪⎩

Ef = x(2:N); eb = x(1:N-1); L = length (ef); for p = 1:P;

gamm(p) = (2*ef’*eb)/(ef’*ef + eb’*eb)

if p < P tmp1 = ef - conj(gamma(p))*eb; tmp2 = eb - gamma(p)*ef; ef = tmp1(2:L); eb = tmp2(1:L-1);

end; L = L-1;

end;

Algorithm: Main part of function (Burg algorithm)

Documents

faculty.nps.edufaculty.nps.edu/fargues/teaching/ec4440/SpringFY09/ec4440-VI-SpFY... · 05/07/09 Ec44440.SpFY09/MPF - Section VI 2 References: [Therrien], [Manolakis], [Hayes]: M