ARMA modelsARMA models

Gloria González-RiveraUniversity of California, RiversideandJesús Gonzalo U. Carlos III de Madrid

White NoiseWhite Noise

A sequence of uncorrelated random variables is called a white noise process.

0for 0),(

)(

)0 (normally )(:2

kaaCov

aVar

aEa

ktt

at

aatt

00

01

00

01

0 0

0

ationautocorrel and anceAutocovari2

k

k

k

k

k

k

kk

k

ak

. . . .1 2 3 4 k

k

The Wold DecompositionThe Wold Decomposition

If {Zt} is a nondeterministic stationary time series, then

tic.determinis is }tV{ 5.

and ,ts,sZ of nscombinatiolinear oflimit theis ta .4

,t and s allfor 0)tV ,sa(Cov .3

,02 with ),2,0(WN is }ta{ .2

,

0j

2j and 10 .1

where

,tVta)L(tVjta

0j

jtZ

Some Remarks on the Wold DecompositionSome Remarks on the Wold Decomposition

n as 02]jta

n

0j

jtZ[E

???

0j

by mean wedo What )(

,...]2tZ,1tZ|tZ[PtZta )(

What the Wold theorem does not sayWhat the Wold theorem does not say

• The at need not be normally distributed, and hence need not be iid

• Though P[at|Zt-j]=0, it need not be true that E[at|Zt-j]=0 (think on the possible consequences???)

• The shocks a need not be the “true” shocks to the system. When will this happen???

• The uniqueness result only states that the Wold representation is the unique linear representation where the shocks are linear forecast errors. Non-linear representations, or representations in terms of non-forecast error shocks are perfectly possible.

Birth of the ARMA modelsBirth of the ARMA models

)L(p

)L(q)L(

Under general conditions the infinite lag polynomial of the WoldDecomposition can be approximated by the ratio of two finite lag polynomials:

Therefore

qtaq...1ta1taptZp...1tZ1tZ

ta)qLq...L11(tZ)pLp...L11(

ta)L(qtZ)L(p

, ta)L(p

)L(qta)L(tZ

AR(p) MA(q)

MA(1) processesMA(1) processes

Let ta a zero-mean white noise process )2a,0(ta

)1(MA1tatatZ Expectation

Variance

Autocovariance

)()()( 1ttt aEaEZE

)1()2(

)()()(22

12

122

21

2

atttt

tttt

aaaaE

aaEZEZVar

221

22

211

2111

)(

))(())(

order 1st.

attttttt

tttttt

aaaaaaaE

aaaaEZE(Z

MA(1) processes (cont)MA(1) processes (cont)Autocovariance of higher order

Autocorrelation

1 0 ) (

) )( ( ) )( (

1 12

1 1

1 1

j a a a a a a a a E

a a a a E Z Z E

j t t j t t j t t j t t

j t j t t t j t t

10

1)1( 222

2

0

11

jj

MA(1) process is covariance-stationary because22 )1()()( tt ZVarZE

MA(1) process is ergodic because

0

222 )1(j

j

If were Gaussian, then would be ergodic for all momentsta tZ

Plot the function 21 1

1-1

0.5

-0.5

1

2for 4.0

5.0for 4.0

1for 5.0 )max(

1

1

1

22121 1)/1(1

/1,

1 substitute we

1 in If

1

1

)1

( ttt

ttt

aaZ

aaZ

Both processes share the same autocorrelation function

MA(1) is not uniquely identifiable, except for 1

InvertibilityInvertibility

Definition: A MA(q) process defined by the equation

is said to be invertible if there exists a sequence of constants

and

ta)L(qtZ

|

0jj|such that }j{ ,...1,0t ,jtZ

0j

jta

Theorem: Let {Zt} be a MA(q). Then {Zt} is invertible if and only if

The coefficients {j} are determined by

the relation

1.|x|such that C xallfor 0)x(

1.|x| ,)x(

1jx

0j

j)x(

Identification of the MA(1)Identification of the MA(1)

• If we identify the MA(1) through the autocorrelation structure, we need to decide with value of to choose, the one greater than one or othe one less than one. Requiring the condition of invertibility (think why????) we will choose the value .

• Another reason to choose the value less than one can be found by paying attention to the error variance of the two “equivalent” representations:

)ta(V)ta(V

invertible-non ,)2

11(

0)ta(V , ta)L1

11(tZ

invertible ,)2

11(

0)ta(V , ta)L11(tZ

2

1

MA(q)MA(q)

qtqtttt aaaaZ 2211

Moments

011

MA(2)Example

for 0

for )(

))((

)1()var(

)(

4322

21

222

22

1

2111

1

2

2211

0

22211

1111

2222

210

k

q

ii

jqqjjjjj

jqqjjjj

qjtqjtjtqtqttj

aqt

t

qj

qj

aaaaaaE

Z

ZE

MA(q) is covariance-stationary

and ergodic for the same reasonsas in a MA(1)

MA(infinite)MA(infinite)

0j

10jtajtZ

Is it covariance-stationary?

0i

2i

0i

jii

j

0i

jii2)jtZ)(tZ(Ej

0i

2i

2a)tZ(Var,)tZ(E

The process iscovariance-stationaryprovided that

0

2

ii

(square summable sequence)

Some interesting resultsSome interesting results

Proposition 1.

Proposition 2.

0

2

0 ii

ii

(absolutelysummable)

00 ii

ii

(squaresummable)

Ergodic for the mean

Proof 1.

0

2

0 ii

ii

Nii

N

ii

Nii

N

ii

ii

Nii

Niiii

ii

i

Ni

NiN

||

Now,

1 that such If

1

0

221

0

2

0

2

22

0

(1) (2)

(1) It is finite because N is finite(2) It is finite because is absolutely summable

0

2

iithen

Proof 2.

00 ii

ii

M

MM

jji

i ii

jjii

jji

ii

j j ijiij

ijii

ijiij

ijiij

0

22

0 0

2

0

2

0 00 0

2

0

2

0

2

0

2

0

2

assumptionby because

AR(1)AR(1)

ttt aZcZ 1Using backward substitution

22

12

122

)1( ttt

tttt

aaac

aaZccZ

geometric progression )( MA

1 if 1

1)2(

sequence bounded 1

11)1(

1 if

00

2

j

j

jj

Remember:

0jj is the condition for stationarity and ergodicity

AR(1) (cont)AR(1) (cont)

Hence, this AR(1) process has a stationary solution if 1

Alternatively, consider the solution of the characteristic equation:

11

01

xx

i.e. the roots of the characteristic equation lie outside of the unit circle

Mean of a stationary AR(1)

1)(

1 22

1

cZE

aaac

Z

t

tttt

Variance of a stationary AR(1)

2

2242

0 1

11 a

Autocovariance of a stationary AR(1)

Rewrite the process as ttt aZZ )()( 1 11

1

jjttjtt

jtttjttj

ZaZZE

ZaZEZZE

11 jjj Autocorrelation of a stationary AR(1)

jjjjj

jj

o

jj j

033

22

10

1 1

ACF

PACF: from Yule-Walker equations

20

011

1

1

1

21

22

21

212

1

1

21

1

22

1̀11

kkk

Causality and StationarityCausality and Stationarity

Definition: An AR(p) process defined by the equation

is said to be causal, or a causal function of {at}, if there exists a sequence of constants

and

Causality is equivalent to the condition

tatZ)L(p

|

0jj|such that }j{ ,...1,0t ,jta

0j

jtZ

1.|x|such that C xallfor 0)x(

Definition: A stationary solution {Zt} of the equation exists (and is also the unique stationary solution) if and only if

tatZ)L(p

1.|x|such that C xallfor 0)x(

From now on we will be dealing only with causal AR models

AR(2)AR(2)

t t t ta Z Z c Z 2 2 1 1

Stationarity Study of the roots of the characteristic equation

01 221 xx

2

22

112

2

22

111

12

2

2

4

2

4

01

x

x

xx

(a) Multiply by -1 (b) Divide by 2x

2

41

2

41

0/1)/1(

22

11

2

22

11

1

212

x

x

xx

For a stationary causal solution is required that

21111

111

1

2,111

1

211

21

2221

xxxx

xx

ix

xi

i

Necessary conditions for a stationary causal solution

22

11

1

2

Roots can be real or complex.(1) Real roots

1 (2) From

1 (1) From

11

2

41

2

41

04

12

21

)1()2(

1

22

11

2

22

11

22

1

xx

(2) Complex roots

04

042

12

22

1

1

-1

1 2-1-2

12 1 12 1

4

21

2

1

2

real

complex

Mean of AR(2)

t t t ta Z Z c Z 2 2 1 1

211

c

Variance and Autocorrelations of AR(2)

2211

2

0

20220110

222110

22112

0

1

)()())(()(

a

a

a

ttttttt aZEZZEZZEZE

1))(( 2211 jZZE jjjttj

12211 jjjj

12213

22

21

2

2

11

02112

12011

3

1

1

2

1

j

j

j

Difference equation

different shapes according to the roots, real or complex

Partial autocorrelations: from Yule-Walker equations

0;1

;1 332

1

212

222

1111

AR(p)AR(p)

tptpttt aZZZcZ .......2211

stationarity All p roots of the characteristic equation outside of the unit circle

ACF

02211

2021112

112011

2211

......

......

......

......

pppp

pp

pp

pkpkkk

System to solve for the first pautocorrelations:p unknowns and p equations

ACF decays as mixture of exponentials and/or damped sine waves, Depending on real/complex roots

PACFpkkk for 0

Relationship between AR(p) and MA(q)Relationship between AR(p) and MA(q)Stationary AR(p)

)()(

1

....)1()()()(

1

)....1()()(

221

221

LL

LLLaLaL

Z

LLLLaZL

p

ttp

t

pppttp

1)()( LLp ? from obtain How to

Example

1222

113

22

12

11

12213

2112

11

312

22

321

2111

33

221

221

221

)(0

0

0

:spolynomial both from tscoefficien equating

1.............

......

......1

1.....)1)(1()2(

LL

LLL

LLL

LLLLAR

22211 jjjj

Invertible MA(q)

)()(

1

....)1()()(

1)(

)....1()()(

221

221

LL

LLLaZL

ZL

LLLLaLZ

q

ttq

t

qqqtqt

1)()( LLq

? from obtain How to

Write an example, i.e. MA(2), and proceed as in the previous example

ARMA (p,q)ARMA (p,q)

ttp

q

ttq

pt

q

tqtp

aLaL

L

aZL

LZL

xx

xx

aLZL

)()(

)(ZtionrepresentaMA Pure

)(

)()( tion representa AR Pure

10)( of roots Causal

10)( of roots ity Invertibil

)()(

t

p

Autocorrelations of ARMA(p,q)Autocorrelations of ARMA(p,q)

ktqtqkttkttktptpkttktt

qtqttptptt

ZaZaZaZZZZZZ

aaaZZZ

..........

:zero toequal mean assume ,generality of loss ofwithout

..........

1111

1111

taking expectations:

1 and on depend will

1......

ii

2211

qk

qk

k

pkpkkk

ikaZE itkt 0)(

that Note

PACFARMAMA

ARMA(1,1)ARMA(1,1)

1)((L)formMA pure

1)((L)Zform AR pure

1ityinvertibil

1 causal

)1()1(

1

1t

jaZ

ja

aLZL

jjtt

jjt

tt

ACF of ARMA(1,1)

kttkttkttktt ZaZaZZZZ 11

taking expectations

)()( 11 kttkttkk ZaEZaE

1

201

2210

21

2

2

1

)(

)()()(0

kk

a

aa

attatt

k

k

ZaEZaEk

10 and for solve

unknowns 2 and equations 2 of system

2

121

1)(

0 1

1

2

k

k

k

k

k

PACF

decays lexponentia

)1,1()1( ARMAMA

ACF

ACF and PACF of an ARMA(1,1)ACF and PACF of an ARMA(1,1)

ACF and PACF of an MA(2)ACF and PACF of an MA(2)

ACF and PACF of an AR(2)ACF and PACF of an AR(2)

ProblemsProblems

P1: Determine which of the following ARMA processes are casual and which of them are invertible (in each case at denotes a white noise):

P2: Show that the two MA(1) processes

have the same autocovariances functions.

ta2tZ81.01tZ8.1tZ .d1ta2.1ta1tZ6.0t Z.c

2tZ7.01tZ2.0ta2tZ88.01tZ9.1tZ .bta2tZ48.01tZ2.0tZ .a

1||0 where

)22 WN(0,is }t{a 1ta1

tatZ

)2 WN(0,is }t{a 1tatatZ

Problems (cont)Problems (cont)

P.3: Let {Zt} denote the unique stationary solution of the autoregressive equations

Where . Then is given by the expression

Define the new sequence

These calculations show that {Zt} is the (unique stationary) solution of the causal

AR equations

1,.... 0, t,ta1tZtZ

1|| and )2,0(WN is }ta{ jta

1j

jtZ

. and of in terms express and ) WN(0,is }W{ that show

,1tZ1

tZtW

22

W

2

Wt

1,... 0, t,tW1tZ1

tZ

Problems (cont)Problems (cont)

P4: Let Yt be the AR(1) plus noise time series defined by Yt =Zt + Wt, where

for all s and t.

• Show that {Yt} is stationary and find its autocovariance functions.

• Show that the time series is an MA(1).

• Conclude from the previous point that {Yt} is an ARMA(1,1) and express the three

parameters of this model in terms of

0)tasE(W and )2a WN(0,a ta with ta1tZt Zand ) 2

W WN(0,is }tW{

1tYtYtU

2a ,2

W ,

Appendix: Lag Operator LAppendix: Lag Operator L

Definition 1 tt ZLZ

Properties

11

1

)(.3

)(.2

.1

tttttt

ttt

kttk

YZLYLZYZL

ZLZZL

ZZL

Examples

tttttt aZLLaZZZ )1(.1 2212211

tt

tttt

tt

aZL

aLaaZ

ZLLLZLL

)1(.4

)1(.3

)1()1)(1(.2

1

2212121

Appendix: Inverse OperatorAppendix: Inverse Operator

Definition

operator)(identity )1()1( that such

).......1(lim)1(01

33221

LLL

LLLLL jjj

Note that :

1 if this definition does not hold because the limit does not exist

Example:

......

)1()1()1(

)1()1(

22

1

11

tttt

tt

tt

aaaZ

aLZLL

aZLAR

Appendix: Inverse Operator (cont)Appendix: Inverse Operator (cont)

Suppose you have the ARMA model and want to findthe MA representation . You could try to crank out directly, but that’s not much fun. Instead you could find

and matching terms in Lj to make sure this works.

ta)L(tZ)L(

ta)L(tZ

)L()L(1

)L()L()L( hence , ta)L()L(tZ)L(ta)L(

Example: Suppose .

Multiplying both polynomials and matching powers of L,

)2L2L10()L( and )L10()L(

3.j ; j01j10

...............2

10011

000

which you can easily solver recursively for the TRY IT!!!j

Appendix: Factoring Lag PolynomialsAppendix: Factoring Lag Polynomials

Suppose we need to invert the polynomial

We can do that by factoring it:

Now we need to invert each factor and multiply:

)2L2L11()L(

121

221

with)L21)(L11()2L2L11(

jL)

0j

j

0k

kj2

k1(

...L)21(1()

0j

jL

0j

j2)(jLj

1(1)L21(1)L11(

Check the last expression!!!!

Appendix: Partial Fraction TricksAppendix: Partial Fraction Tricks

There is a prettier way to express the last inversion by using the partial fraction tricks. Find the constants a and b such that

)L21)(L11(

)L21(b)L11(a

)L21(

b

)L11(

a

)L21)(L11(

1

The numerator on the right hand side must be 1, so

)j2)21(

2

0j

j1)21(

1(

)L21(

1

)21(2

)L11(

1

)21(1

)L21)(L11(

1

so

,21

1a ,12

2b

Solving,

0b1a2

1ba

Appendix: More on InvertibilityAppendix: More on Invertibility

Consider a MA(1)

t

AR

t

ttt

t

aZLLL

aaLLZ

L

aL

)(

3322

11-

1

t

).......)(1(

)1()1()(L)(1

defined is )1(,1 If

1Z

Definition

A MA process is said to be invertible if it can be written as an AR( )

• For a MA(1) to be invertible we require ]11

01[ 1

xx

• For a MA(q) to be invertible, all roots of the characteristic equation should lie outside of the unit circle• MA processes have an invertible and a non-invertible representations• Invertible representation optimal forecast depends on past information• Non-invertible representation forecast depends on the future!!!