DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

Dynamic Linear Models And KalmanFiltering

Dr. Holger Zien

12th December 2014

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

IntroductionOne of The Most Common Problems Is To Forecast Time Series

?600

800

1000

1200

1400

1875 1900 1925 1950 1975 2000Year

Nile

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

IntroductionOne of The Most Common Problems Is To Forecast Time Series

600

800

1000

1200

1400

1875 1900 1925 1950 1975 2000Year

Nile

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

IntroductionOne of The Most Common Problems Is To Forecast Time Series

600

800

1000

1200

1400

1875 1900 1925 1950 1975 2000Year

Nile

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

IntroductionOne of The Most Common Problems Is To Forecast Time Series

600

800

1000

1200

1400

1875 1900 1925 1950 1975 2000Year

Nile

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

IntroductionTypical Time Series In Classical Textbooks

I many samplesI more or less stationaryI typical for physical measurementsI the method of choice is ARMA

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

IntroductionIn Practice We Are Facing A Different Kind of Time Series

I ridiculous shortI often non–stationaryI ARMA models do not workI Sometimes Bayesian modeling of

Dynamic Linear Models/KalmanFiltering will help

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

IntroductionAutoregressive Moving Average Model (ARMA)

yt = εt +p∑

i=1φiyt−i︸ ︷︷ ︸

autoregressive part

+q∑

j=1ψjεt−j︸ ︷︷ ︸

moving average part

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

Dynamic Linear ModelSimplest Version

yt = Fθt + νtobservation eq.θt = Gθt−1 + ωtsystem eq.

νt , ωt : mutually independent random variables

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

Dynamic Linear ModelVector–Valued State/Observation, Time–Dependent Coefficents

y t = FTt · θt + νtobservation eq.

θt = Gt · θt−1 + ωtsystem eq.

θ, ω: n–dimensional random vectorsy , ν: r–dimensional random vectors

G: n × n dimensional state evolution matrixF : n × r dimensional dynamic regression matrix

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

Dynamic Linear ModelKalman Filtering

y t = FTt · θt + νt

θt = Gt · θt−1 + ωt

ωt ∼ N(0, W t)νt ∼ N(0, V t)

θ0|D0 ∼ N(m0, C0)

θt |Dt ∼ N(mt , C t) mt = at + At · (y t − f t)post. distr. θt :

C t = R t − At ·Qt · ATt

At = R t · F t ·Q−1t

y t |Dt−1 ∼ N(f t , Qt) f t = F Tt · atforecast:

Qt = F Tt · R t · F t + V t

θt |Dt−1 ∼ N(at , R t) at = G t ·mt−1prior distr. θt :

R t = G t · C t−1 · GTt + W t

Bayes’ Law

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

Dynamic Linear ModelKalman Filtering

y t = FTt · θt + νt

θt = Gt · θt−1 + ωt

ωt ∼ N(0, W t)νt ∼ N(0, V t)

θ0|D0 ∼ N(m0, C0)

θt |Dt ∼ N(mt , C t) mt = at + At · (y t − f t)post. distr. θt :

C t = R t − At ·Qt · ATt

At = R t · F t ·Q−1t

y t |Dt−1 ∼ N(f t , Qt) f t = F Tt · atforecast:

Qt = F Tt · R t · F t + V t

θt |Dt−1 ∼ N(at , R t) at = G t ·mt−1prior distr. θt :

R t = G t · C t−1 · GTt + W t

Bayes’ Law

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

Dynamic Linear ModelKalman Filtering

y t = FTt · θt + νt

θt = Gt · θt−1 + ωt

ωt ∼ N(0, W t)νt ∼ N(0, V t)

θ0|D0 ∼ N(m0, C0)

θt |Dt ∼ N(mt , C t) mt = at + At · (y t − f t)post. distr. θt :

C t = R t − At ·Qt · ATt

At = R t · F t ·Q−1t

y t |Dt−1 ∼ N(f t , Qt) f t = F Tt · atforecast:

Qt = F Tt · R t · F t + V t

θt |Dt−1 ∼ N(at , R t) at = G t ·mt−1prior distr. θt :

R t = G t · C t−1 · GTt + W t

Bayes’ Law

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

Dynamic Linear ModelKalman Filtering

y t = FTt · θt + νt

θt = Gt · θt−1 + ωt

ωt ∼ N(0, W t)νt ∼ N(0, V t)

θ0|D0 ∼ N(m0, C0)

θt |Dt ∼ N(mt , C t) mt = at + At · (y t − f t)post. distr. θt :

C t = R t − At ·Qt · ATt

At = R t · F t ·Q−1t

y t |Dt−1 ∼ N(f t , Qt) f t = F Tt · atforecast:

Qt = F Tt · R t · F t + V t

θt |Dt−1 ∼ N(at , R t) at = G t ·mt−1prior distr. θt :

R t = G t · C t−1 · GTt + W t

Bayes’ Law

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

Dynamic Linear ModelKalman Filtering

y t = FTt · θt + νt

θt = Gt · θt−1 + ωt

ωt ∼ N(0, W t)νt ∼ N(0, V t)

θ0|D0 ∼ N(m0, C0)

θt |Dt ∼ N(mt , C t) mt = at + At · (y t − f t)post. distr. θt :

C t = R t − At ·Qt · ATt

At = R t · F t ·Q−1t

y t |Dt−1 ∼ N(f t , Qt) f t = F Tt · atforecast:

Qt = F Tt · R t · F t + V t

θt |Dt−1 ∼ N(at , R t) at = G t ·mt−1prior distr. θt :

R t = G t · C t−1 · GTt + W t

Bayes’ Law

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

Glossary

Filtering: Estimate of the current value of thestate/system variable.

Smoothing: Estimate of past values of the state/systemvariable, i.e., estimating at time t givenmeasurements up to time t ′ > t.

Forecasting: Forecasting future observations or values of thestate/system variable.

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

Some Applications

I Modeling of short time series (Bayes model).I Models composed of different components (trends,

seasonality, ARMA)I Non–stationary modelsI Models with interventionsI Regression model with time–dependent coefficients

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

Some ApplicationsRegression Model

Regression model: yt = αt + β1,tx1,t + β2,tx2,t + εt

yt = FTt · θt + εtDLM:

θt = θt−1 + ωt

F t = (1, x1,t , x2,t)Twith:θt = (αt , β1,t , β2,t)T

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

Some ApplicationsRegression Model

Regression model: yt = αt + β1,tx1,t + β2,tx2,t + εt

yt = FTt · θt + εtDLM:

θt = θt−1 + ωt

F t = (1, x1,t , x2,t)Twith:θt = (αt , β1,t , β2,t)T

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

Some ApplicationsRegression Model

Regression model: yt = αt + β1,tx1,t + β2,tx2,t + εt

yt = FTt · θt + εtDLM:

θt = θt−1 + ωt

F t = (1, x1,t , x2,t)Twith:θt = (αt , β1,t , β2,t)T

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

Some Applications

I Modeling of short time series (Bayes model).I Models composed of different components (trends,

seasonality, ARMA)I Non–stationary modelsI Models with interventionsI Regression model with time–dependent coefficientsI ARMA models with known or unknown coefficients

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

Some ApplicationsARMA Model With Known Coefficients

ARMA(2,2) model: yt = εt +2∑

i=1φiyt−i +

2∑j=1

ψjεt−j

yt = F Tt · θtDLM:

θt = G · θt−1 + ωt

F t = (1, 0, 0)Twith:

G =

φ1 1 0φ2 0 10 0 0

ωt = (1, ψ1, ψ2)T

εt

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

Some ApplicationsARMA Model With Known Coefficients

ARMA(2,2) model: yt = εt +2∑

i=1φiyt−i +

2∑j=1

ψjεt−j

yt = F Tt · θtDLM:

θt = G · θt−1 + ωt

F t = (1, 0, 0)Twith:

G =

φ1 1 0φ2 0 10 0 0

ωt = (1, ψ1, ψ2)T

εt

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

Some ApplicationsARMA Model With Known Coefficients

ARMA(2,2) model: yt = εt +2∑

i=1φiyt−i +

2∑j=1

ψjεt−j

yt = F Tt · θtDLM:

θt = G · θt−1 + ωt

F t = (1, 0, 0)Twith:

G =

φ1 1 0φ2 0 10 0 0

ωt = (1, ψ1, ψ2)T

εt

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

Some ApplicationsARMA Model With Unknown Coefficients

ARMA(2,0) model: yt = εt +2∑

i=1φiyt−i

yt = F Tt · θt + εtDLM:

θt = θt−1 + ωt

F t = (yt−1, yt−2)Twith:

θt = (φ1, φ2)T

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

Some ApplicationsARMA Model With Unknown Coefficients

ARMA(2,0) model: yt = εt +2∑

i=1φiyt−i

yt = F Tt · θt + εtDLM:

θt = θt−1 + ωt

F t = (yt−1, yt−2)Twith:

θt = (φ1, φ2)T

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

Some ApplicationsARMA Model With Unknown Coefficients

ARMA(2,0) model: yt = εt +2∑

i=1φiyt−i

yt = F Tt · θt + εtDLM:

θt = θt−1 + ωt

F t = (yt−1, yt−2)Twith:

θt = (φ1, φ2)T

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

Experience

Pros: I Applicable to short time seriesI Not restricted to stationary dataI Large DLM models can be build through

composing small DLM modelsI May be extended to non-normal distributionsI Results are easy to interpret

Cons: I Finding size of noise terms νt , ωt difficultMLE yields unreasonable results frequently.A solution might be Bayesian estimation ofνt , ωt which is not part of the R–libraries

I Incomplete R libraries, no “standard” libraryI Difficult numerical implementation

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

R–Libraries

dlm Personally preferred library, see Petris (2010)FKF Performance–optimized Kalman filtering, no

functions for model buildingsspir Library used by Cowpertwait u. Metcalfe

(2009), no longer maintaineddlmodeler Common interface to libraries dlm, FKF, KFAS

. . . Several other libraries, see CRAN Task View:“Time Series Analysis”, for comparison seeTusell (2011) and Commandeur u. a. (2011)..

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

Some References I[Commandeur u. a. 2011] Commandeur, Jacques J. F. ; Koopman,

Siem J. ; Ooms, Marius: Statistical Software for State SpaceMethods. In: Journal of Statistical Software 41 (2011), 5, Nr. 1,1–18. http://www.jstatsoft.org/v41/i01. – ISSN 1548–7660

[Cowpertwait u. Metcalfe 2009] Cowpertwait, Paul S. P. ;Metcalfe, Andrew V.: Introductory Time Series with R. Springer,2009 (Use R!). – 256 S.http://dx.doi.org/10.1007/978-0-387-88698-5.http://dx.doi.org/10.1007/978-0-387-88698-5. – ISBN978–0–387–88697–8

[Dethlefsen u. Lundbye-Christensen 2006] Dethlefsen, Claus ;Lundbye-Christensen, Søren: Formulating State Space Models inR with Focus on Longitudinal Regression Models. In: Journal ofStatistical Software 16 (2006), Mai, Nr. 1, 1–15.http://www.jstatsoft.org/v16/i01/paper, Abruf: 2011-07-25

[Durbin u. Koopman 2002] Durbin, J. ; Koopman, S.J.: Time SeriesAnalysis by State Space Methods: Second Edition. OUP Oxford,2002 (Oxford Statistical Science Series).http://books.google.de/books?id=fOq39Zh0olQC. – ISBN9780199641178

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

Some References II[Honerkamp 1990] Honerkamp, J.: Stochastische Dynamische

Systeme. Weinheim : vch, 1990[Hyndman 2009] Hyndman, Rob J.: forecast: Forecasting functions for

time series. 1.25, 2009.http://CRAN.R-project.org/package=forecast

[Petris 2010] Petris, Giovanni: An R Package for Dynamic LinearModels. In: Journal of Statistical Software 36 (2010), Nr. 12, 1–16.http://www.jstatsoft.org/v36/i12/paper, Abruf: 2010-10-15

[Petris u. Petrone 2011] Petris, Giovanni ; Petrone, Sonia: StateSpace Models in R. In: Journal of Statistical Software 41 (2011),Mai, Nr. 4, 1–25. http://www.jstatsoft.org/v41/i04/paper,Abruf: 2011-05-20

[Petris u. a. 2007] Petris, Giovanni ; Petrone, Sonia ; Campagnoli,Patrizia: Dynamic Linear Models with R. Version: aug 2007.http://www.math.u-bordeaux1.fr/~pdelmora/dynamics-linear-models.petris_et_al.pdf, Abruf: 2013-07-10.– Preprint

[Tusell 2011] Tusell, Fernando: Kalman Filtering in R. In: Journal ofStatistical Software 39 (2011), 3, Nr. 2, 1–27.http://www.jstatsoft.org/v39/i02. – ISSN 1548–7660

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

Some References III

[West u. Harrison 1997] West, Mike ; Harrison, Jeff: BayesianForecasting and Dynamic Models. 2. Springer-Verlag, 1997 (SpringerSeries in Statistics). – 680 S. – ISBN 0–387–94725–6

DLM

Dr. Holger Zien

IntroductionARMA

DLMKalman Filtering

Glossary

ApplicationsRegression

ARMA

Experience

R-Libraries

References

Finally

Questions?

Thank You! ,

Ü LUX