The process has correlation sequence

)()(0, kijnikkj jinin WWEbbXXEc

dBdecdZn nnX

221

21 |)(|)()(

22110 nnnn WbWbWbXCorrelation and Spectral Measure

nnjkj jkijnikkj j bbbbbb )( *

0,0,

where )(2* Zb , the adjoint of ),(2 Zb is defined by .,* Zkbb kk

The process has spectral measure

)()())(()( 221101 ebebbbB -Fwhere

dBdecdZn nnX

221

21 |)(|)()(

The process

is a moving average (MA) if

22110 nnnn WbWbWbX

MA and AR Processes

b has finite support.

The process is autoregressive (AR) if there exists

).(/1)( iePB

a polynomial dd zpzpzppzP 2

210)(no zeros in the closed unit disc }1||:{ zCzD

such that

with

Then

kmn

d

m kk mmn

d

m m WbpXp

0 00

njj jnjj

d

m jmnm WXWbp

0

0

0 0

so .000

2

0

1 121 npdnp

pnp

pnp

pn WXXXX d

The MA process has spectral measureMA and AR Processes

The AR process has spectral measure

where is a trigonometric

roots have modulus > 1.

dB 221 |)(| )( Bpolynomial so

2|)(| B is a nonnegative trig. poly.

221 |)(|/1

ieP where P is a polynomial whose

Lemma (L. Fejer & F. Riesz) Every nonnegativetrigonometric polynomial has the form

2|)(| iePwhere P is a polynomial whose root moduli < 1.Proof pages 117-118.Corollary Every nonneg. trig. poly. can be the spec. meas. of a MA process. Every reciprocalpos. trig. poly. can be the spec. meas. of a AR pr.

Assume that we are given correlation coefficientsPrediction

and wish to determineddd cccccc ,,,,,,, 1011

Caaa d ,,, 21 so as to minimize

A standard fact about least squares estimation isthat this quantity is minimized iff

.),||( 22211 ZnXaXaXaXE dndnnn

,,...,1,0))(( 2211 dnnkXXaXaXaXE kdndnnn ,,...,1,0

2211 dnnkcacacac kdndknkn kn

,,...,1,0

2211 djcacacac djdjj j

.,...,1,

2211 djccacaca jdjdj j

These equations, in matrix form, are the

Normal Equations

ddddd

d

d

d

c

c

c

c

a

a

a

a

cccc

cccc

cccc

cccc

3

2

1

3

2

1

0321

3012

2101

1210

Notice that the matrix is Hermitian, Toeplitz andpositive semi-definite. If it is positive definite thenthe solution is unique. If it is singular then Theorem 3.1.2 implies that it has a solution with

.,2211 ZnXaXaXaX dndnnn

For the AR process

Prediction for AR Process

if we choose then,,...,2,1,0

dja p

p

jj

,000

2

0

1 121 npdnp

pnp

pnp

pn WXXXX d

))(( 2211 kdndnnn XXaXaXaXE

,,...,1,0)( dnnkXWE kn Therefore the prediction coefficients are uniquely determined and yield a residual error that equalsthe innovation process .nW In fact since

1,0)( nkXWE kn the best predictor for nXas a linear combination of deXX enn ,,...,1

.2211 dndXn XaXaXa

n

is

Empirical PredictionIn many (perhaps most) prediction applicationswe do not have a statistical model for a random

process but instead we have sample valuesnX

of a ‘time series’ and for

some positive integer

LnFCxn ,

FLd we want tocompute daaa ~,...,~,~ 21 that minimizes

L

dFn dndnnn xaxaxax 22211 ||

or equivalently (prove this) solves the system

.,...1,)( 2211 djxxaxaxaxL

dFn jndndnnn

These equations can be expressed in the form

Empirical Normal Equations

ddddddd

d

d

d

c

c

c

c

a

a

a

a

cccc

cccc

cccc

cccc

,0

3,0

2,0

1,0

3

2

1

.3,2,1,

,33,32,31,3

,23,22,21,2

,13,12,11,1

~

~

~

~

~

~

~

~

~~~~

~~~~

~~~~

~~~~

,,...1,,)1(~ 1, djixxFLc

L

dFn jninji

.,...1,)1(~ 1,0 djxxFLc

L

dFn jnnj

where

Remark It is only approximately ToeplitzQuestion Show this matrix is positive semidefinite

Improved Emp. Normal Equations

11,~ dddc jwhere

hence

in the trig. poly.

ddddd

d

d

d

c

c

c

c

a

a

a

a

cccc

cccc

cccc

cccc

~

~

~

~

~

~

~

~

~~~~

~~~~

~~~~

~~~~

3

2

1

3

2

1

0321

3012

2101

1210

is the coefficient of je2

22111 ||)1( FLLFFF exexexxFL

.)1(~ },min{

},max{

1n

jLL

jFFn jnj xxFLc

Show this Toeplitz matrix is pos. semidefinite.

Law of Large Numbers

numbers that

This argument can be made precise using

Give an informal argument using the law of large

,,...,1,,~lim~lim , djiccc jijiFL

jiFL

.,...,1,~lim~lim ,0 djccc jjFL

jFL

ergodic theory which we will not pursue inthis course.

Homework Project

in MATLAB generate a sequence one million

of an white noise process that has a real

1. Using the normal random number generator

10000001,...,ww

gaussian distribution.2. Generate samples of an AR process using:

samples

.10000001,36.05.0,0 2101 nwxxxxx nnnn

3. Compute the spectral measure for the ARprocess and use to compute a closed form for the

4. Compute the empirical correlation sequencecorrelation sequence of the AR process.

coefficients 22,~ jc jand use them to estimate the AR coefficients.