State Space Modeling of Sequence Data...State Space Model and Notation Y t = Z t t + X t + t Observation equation t+1 = T t t + W t+1 + t+1 State transition equation 1 = A 1 + W 1

Outline Intro State Space Model KFS Illustrations Software References

State Space Modeling of Sequence Data

Rajesh Selukar

SAS Institute Inc., Cary, [email protected]

June 19, 2015


Outline

• Three sessions of about 50 minutes each

• Session 1• Introductory remarks• (Linear) state space model (SSM)• Kalman filter/smoother (KFS) output

• Session 2 and most of session 3• Illustrative examples

• Last few minutes• Brief overview of available software for state space modeling

• State space modeling in these slides is done by using PROCSSM, a SAS/ETS procedure


Initial Comments

• SSM encompasses many important model classes:• Univariate and multivariate ARIMAX• (Linear) exponential smoothing models (ESMs)• Unobserved components models (UCMs)• Dynamic factor models• Functional mixed-effects models• ...

• For some of these classes, SSM is a very natural framework

• SSMs are useful for modeling a wide variety of sequence data:• Univariate and multivariate time series• Panels of univariate and multivariate time series• Univariate and multivariate longitudinal data• Multivariate, multifrequency• Sequential data collected according to complex survey designs


Why Use the SSM Formulation?

• SSM analysis provides very rich output that enables you to:• Interpolate and extrapolate (forecast) response variables• Interpolate and extrapolate latent components such as trends,

seasonal effects• Detect structural breaks in the evolution of the latent

components and detect outliers in the response values• ...

• Even for models such as ARIMAX, the analysis based on SSMformulation provides added benefits.

• For some models, such as functional mixed-effects models,SSM formulation leads to highly efficient algorithms.

• SSM allows for data irregularities:• Permits missing values in the response variables• ...


Why Isn’t State Space Modeling More Widespread?

• Availability of good software is relatively recent

• Coursework on the topic in colleges is limited

• Learning resources are limited: few good books fornon-specialists

• In addition• if modeling goals are limited (such as ARIMAX

fitting/forecasting), the state space framework is not neededand can be computationally expensive

• formulating a model as an SSM is not always easy


State Space Model and Notation

Yt = Ztαααt + Xtβββ + εεεt Observation equation

αααt+1 = Ttαααt + Wt+1γγγ + ηηηt+1 State transition equation

ααα1 = A1δδδ + W1γγγ + ηηη1 Initial condition

• SSM form is not unique; many equivalent alternate forms arepossible. This one is quite general.

• Response values y and predictor vectors x = (x1, x2, . . . , xk)are recorded at τ1 < τ2 < · · · < τn.

• Time points τi need not be equally spaced, and there can bemultiple measurements at the same time point. Number ofmeasurements at τt = pt , t = 1, 2, · · · , n.

• Yt and Xt denote the vector and matrix formed by verticallystacking y values and x vectors at τt . Dim(Yt) = pt , andDim(Xt) = pt × k .

• For univariate time series, τt are equispaced and pt = 1∀t.


Data Used for Modeling




Matrix Dim Description

Yt pt × 1 Response values at τtXt pt × k Design matrix for the observation regression vector βββWt m × g Design matrix for the state regression vector γγγ

• Missing values are allowed in Yt . In fact, they indicate placesof interpolation and extrapolation.

• Missing/unknown values are not allowed in Xt ,Wt .


Latent Quantities in the Model




Vector Dim Description

αααt m State vectors (time-varying regression-like vectors)βββ k Regression vector in the observation equationγγγ g Regression vector in the state equationδδδ d Diffuse part of ααα1 (regression-like vector)εεεt pt Observation noise (zero-mean, Gaussian)ηηηt m State noise (zero-mean, Gaussian)

• Noise/shock/disturbance variables εεεt and ηηηt are mutuallyindependent white noise sequences (frequently withtime-varying covariances).


Model System Matrices




Matrix Dim Description

Zt pt ×m Design matrix for αααt

Tt m ×m State transition matrixA1 m × d Diffuse condition specifier made up of 0’s and 1’sCov(εεεt) pt × pt Often diagonalCov(ηηηt) m ×m Often nondiagonal

• Missing elements are not allowed in any system matrix.However, the system matrices can depend on some unknownparameter vector θθθ (which must be estimated first for themodel to be practically useful).


Comments




• SSM is structurally rich enough so that appropriate choices ofsystem matrices lead to a wide variety of data generationprocesses.

• For equispaced data, the system matrices (such as Tt andCov(ηηηt)) often do not depend on t. Otherwise, they usuallydepend on the distance between successive τt : (τt+1 − τt).

• The transition matrix Tt controls the stability and stationarityproperties of the model. Many important models arenonstationary (and sometimes even unstable).


Composition of SSMs

Yt = Xtβββ + Z1tααα

1t + Z2

tααα2t + · · ·+ εεεt

αααt+1 = Ttαααt + Wt+1γγγ + ηηηt+1

• Curves (or sets of curves) formed by sequential responsevalues can often be decomposed into interpretablecomponents such as trend, cycle, ..

• Useful SSMs are often formed by combining smaller SSMsthat capture different aspects of the data generation process.

• The overall state vector is formed by joining the state vectorsof the submodels: αααt = (ααα1

t ααα2t · · · ).

• The other latent vectors and system matrices are also formedby appropriately joining the corresponding parts.

• Any linear combination of state (and regression vector)elements is called a component.


Weighted Least Squares (A Short Detour)

Y = Xβββ + εεε, εεε ∼ N(0,Σ), Dim(Y) = N,Dim(βββ) = k

• βββ = (X′X)−1X

′Y, COV(βββ) = σ2(X

′X)−1 if Σ = σ2I

• βββ = (X′Σ−1X)−1X

′Σ−1Y, COV(βββ) = (X

′Σ−1X)−1

• εεε = Y − Xβββ is an estimate of the latent noise vector• Matrix inversion cost: OLS ∼ k3, WLS ∼ N3 (worst case)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -• Suppose Σ (and occasionally X) are not fully known and can

depend on some unknown parameter vector θθθ, which can beestimated by maximum likelihood.

• Profile and restricted are frequently used likelihood variantsthat yield the ML and REML estimate of θθθ, respectively:

• −2logLp(Y, θθθ) = log(|Σ|) + (Y − Xβββ)′Σ−1(Y − Xβββ) + const

• −2logLr (Y, θθθ) = −2logLp(Y, θθθ) + log(|X′Σ−1X|)

• Finally, βββ and εεε calculated at the ML/REML estimate of θθθ areused for statistical decision-making.


Is SSM Just a Giant WLS Problem?




• Starting at t = 1, expressions for Yt can be written as

Y1 = Z1A1δδδ + Z1W1γγγ + X1βββ + Z1ηηη1 + εεε1

Y2 = Z2T1A1δδδ + (Z2T1W1 + Z2W2)γγγ + X2βββ + Z2T1ηηη1 + Z2ηηη2 + εεε2

Y3 = · · ·

• SSM is a WLS problem with regression vector (δδδ γγγ βββ) and thenoise vector a linear transform of ηηηt , εεεt , t = 1, 2, · · · , n.

• The model parameter vector θθθ, regression vector (δδδ γγγ βββ), andαααt , ηηηt , εεεt (for all t) can be estimated by ML/WLS.

• Kalman filter/smoother (KFS) is a way to do this efficiently.


Informal Description of the Kalman Filter (KF)

• Assume that the model parameter vector θθθ is known.

• KF recursively computes the one-step-ahead predictions of theresponse values and the latent quantities.

• Let DATAt denote all the data up to time τt .

• KF recursively computes:Yt = E(Yt |DATAt−1) Ft = COV(E(Yt |DATAt−1))αααt = E(αααt |DATAt−1) Pt = COV(E(αααt |DATAt−1))

βββt = E(βββ|DATAt−1) Gt = COV(E(βββ|DATAt−1))· · · · · ·

• For latent noise vectors, the one-step-ahead predictions aretrivial:

• E(εεεt |DATAt−1) = 0 and COV(E(εεεt |DATAt−1)) = Cov(εεεt)• E(ηηηt |DATAt−1) = 0 and COV(E(ηηηt |DATAt−1)) = Cov(ηηηt)


Kalman Smoother (KS)

• KS computes the smoothed (full-sample) predictions of themissing response values and the latent quantities.

• It is a backward recursive algorithm that uses theone-step-ahead forecasts generated during the KF phase.

• KS computes:Yt = E(Yt |DATAn) Ft = COV(E(Yt |DATAn)), for missing Yt

αααt = E(αααt |DATAn) Pt = COV(E(αααt |DATAn))

βββ = E(βββ|DATAn) G = COV(E(βββ|DATAn))· · · · · · · · · · · ·ηηηt = E(ηηηt |DATAn) Ht = COV(E(ηηηt |DATAn))

• KS also yields other useful quantities, such as delete-one crossvalidation measures and structural break statistics.


State Space Modeling: General Steps

Phase 1: Choose a good SSM for the observed data.

1. Propose a tentative SSM.

2. If the specified SSM has unknown parameters, estimate them.

3. Check the model adequacy and complexity (residual analysis,other diagnostics, ...).

4. If the model is inadequate or overly complex, modify it (backto the beginning).

Phase 2: Deploy the chosen SSM

• Use the estimated regression vectors for decision making

• Interpolate/extrapolate response values, latent components, ...

• Obtain a seasonal decomposition of the data sequence

• ...

KFS is the main computational tool for both the phases


KFS for Model Fitting and Diagnostics (Phase 1)

• Start with a proposed SSM, possibly with unknown parametervector θθθ.

• KF yields one-step-ahead residuals and the likelihood of thedata (at a specific trial value of θθθ):

• Rt = (Yt − Yt) ∼ N(0,Ft) is an uncorrelated sequence.• −2logL(θθθ,DATAn) =

∑nt=1 log(Det(Ft)) + R

′

tF−1t Rt + · · ·

• Obtain the ML estimate of θθθ by maximizing logL(θθθ,DATAn)with respect to θθθ.

• Check the fitted model for adequacy and compare with otherfitted models:

• Residual analysis, structural break analysis, ...• Compare models by using information criteria.• KS yields delete-one cross validation measures, which can also

be used for model comparison.


State Space Modeling: Computational Cost

• n = number of distinct time points, m = Dim(αtαtαt)

• Cost of single KFS run:• Number of multiplications ∼ nm3

• Memory requirement of a KF run ∼ m2

• Memory requirement of a KS run ∼ nm2 (output of a full KFrun must be stored)

• ML estimation of parameter vector (θθθ) involves several runs ofKFS (KF is used for likelihood computation, and KS is usefulfor the likelihood gradient computation).

• Computational/memory costs increase rapidly with m (onlylinearly with n).

• In some situations, the computational efficiency can beimproved by exploiting the sparsity of the system matrices.


Data and PROC SSM Code for the Illustrations?

• Most illustrations are examples in the PROC SSMdocumentation. More details can be found there.

• Documentation:http://support.sas.com/documentation/cdl/en/etsug/

67525/HTML/default/viewer.htm#etsug_ssm_toc.htm.Contains a good list of references for state space modeling.

• Data and code for the examples: http://support.sas.com/

documentation/onlinedoc/ets/ex_code/132/index.html

(you must scroll down to the SSM section).

• The functional mixed-effects modeling examples (PDG andtheophylline patterns) are explained in a white paper:http://support.sas.com/resources/papers/

proceedings15/SAS1580-2015.pdf

• The data and code for the ”Panel Study of Income Dynamics”example are provided separately.

http://support.sas.com/documentation/cdl/en/etsug/67525/HTML/default/viewer.htm#etsug_ssm_toc.htm


http://support.sas.com/documentation/onlinedoc/ets/ex_code/132/index.html

http://support.sas.com/documentation/onlinedoc/ets/ex_code/132/index.html

http://support.sas.com/resources/papers/proceedings15/SAS1580-2015.pdf



A Transfer Function Model for the Gas Furnace Data

• A well-known example from a book by Box and Jenkins

• Data collected during an experiment at a chemical factory

• Goal: Study the relationship between Output CO2 (yt) andInput Gas Rate (xt)

• (yt , xt) are measured at equally spaced time intervals

• Suggested model: yt = µ+ ft + ξt• Intercept: µ

• Transfer function: ft = γ1B3+γ2B

4+γ3B5

1−ωB xt• ft = ωft−1 + γ1xt−3 + γ2xt−4 + γ3xt−5

• Zero-mean AR(2) noise: ξt• ξt = φ1ξt−1 + φ2ξt−2 + νt ,• νt ∼ N(0, σ2

ν)


With the Usual ARIMAX Modeling

• You can:• Fit/diagnose the model: Estimate model parameters, residual

diagnostics, detect outliers in yt , ...• Forecast the response variable (yt)• Estimate/forecast the latent components: TF input ft and the

AR(2) noise ξt . Usual analyses do not emphasize studyingthese estimates.

• Difficult to do:• Provide standard errors for the estimate of ft• Detect structural breaks in the evolution of ft and ξt


Benefits of the SSM Framework (for This Example)

• SSM can do everything that the usual ARIMAX techniquedoes.

• Additionally, it provides:• Theoretically more satisfying estimates of µ, γ1, γ2, γ3, and ft• Standard errors of the estimate of ft• All of this, in turn, means that the predictions of yt and the

associated confidence intervals are more justifiable.• Detection of structural breaks in the evolution of ft and ξt• ...

• For this particular data set and this model, the (traditional)ARIMAX and SSM parameter estimates turn out to be close.


SSM Formulation• Model: yt = µ+ ft + ξt

• ft = ωft−1 + γ1xt−3 + γ2xt−4 + γ3xt−5

• ξt = φ1ξt−1 + φ2ξt−2 + νt• Suppose

• βββ = µ, γγγ = (γ1 γ2 γ3), δδδ = f1• αααt = ([ft ] [ξt ξt−1]), ηηηt = ([0] [νt 0])• Tt = Block-Diag([ω] [φ1 φ2; 1 0]). A1 = ([1] [0 0])

′.

• Wt = (xt−3 xt−4 xt−5; 0 0 0; 0 0 0).

• Thenyt = µ+ (1 1 0)αααt

αααt+1 = Ttαααt + Wt+1γγγ + ηηηt+1

ααα1 = A1f1 + (0 ξ1 ξ0)′

• Note: The observation equation has no noise term (εt)

• Parameter vector θθθ = (ω φ1 φ2 σ2ν)

• Phase 1: ML estimation of θθθ and model diagnostics

• Phase 2: Model deployment


PROC SSM Code

x3 = lag3(x); x4=lag4(x); x5= lag5(x);

proc ssm data=Seriesj(firstobs=6);

id obsIndex;

/*--Observation equation--*/

model y = intercept tfinput ar2;

/*-- Intercept --*/

intercept = 1;

/*-- TF input specification --*/

parms omega /lower=-1.0 upper=1.0;

state tfstate(1) T(g)=(omega)

W(g)=(x3 x4 x5) a1(1) checkbreak;

comp tfinput = tfstate[1];

/*-- AR(2) Noise --*/

trend ar2(arma(p=2)) checkbreak;

/*--Output a component--*/

eval modelCurve = intercept + tfinput;

output out=co2For;

run;


Transfer Function Model: Structural Break AnalysisBreak analysis of the transfer function ft :

Elementwise Break Summary for tfstate

Element

ID Index Z Value Pr > |z|

264 1 -5.03 <.0001

199 1 4.62 <.0001

198 1 -3.15 0.0016

Break analysis of the AR(2) term ξt :

Elementwise Break Summary for ar2

Element

ID Index Z Value Pr > |z|

264 2 6.09 <.0001

264 1 -5.82 <.0001

199 1 4.16 <.0001


Transfer Function Model: Estimates of ft and ξt• Left panel shows the plot of (µ+ ft) and yt . The break point at 264 is

indicated by the vertical red line.• Right panel shows the estimate of ξt .

Estimate of (µ+ ft) with 95% Confidence Band Estimate of the AR(2) term ξt

• This type of monitoring is very useful for industrial process control.• I have not seen such break analysis in the previous analyses of these data.


Daily Progesterone (PDG) Patterns in Two Groups

• 51 healthy women registered in a fertility clinic• 69 non-conceptive and 22 conceptive daily PDG curves• Ovulation: day 0, conception (if it happens) around day 8.• Hypothesis: The mean PDG curve for the conceiving group is lower

than the non-conceiving group between the days 0 and 8.


Daily PDG Pattern Analysis: Background

• Brumback and Rice (1998)—considered an important paper inthe functional modeling literature—analyzed these data intheir JASA paper. They suggest a functional mixed-effectsmodel based on smoothing splines.

• Smoothing-spline-based functional mixed-effects models canoften be formulated as SSMs.

• SSM formulation is generally far more efficient than otherapproaches for smoothing-spline-based functionalmixed-effects modeling. See http://support.sas.com/

resources/papers/proceedings15/SAS1580-2015.pdf.




A Functional Mixed-Effects Model for the PDG Curves

• A PDG curve = Mean Curve + Deviation Curve + Error

y cit = µct + ξcit + εcit

yncit = µnct + ξncit + εncit

• y cit :ith conceptive PDG curve (i = 1 · · · 22).

• yncit : ith non-conceptive PDG curve (i = 1 · · · 69).

• Mean curves (µct , µnct ): Integrated random walk trends

• functional fixed-effects

• Deviations curves (ξcit , ξncit ): First-order autoregressive

• functional random-effects

• Hypothesis: µct < µnct for 0 ≤ t ≤ 8


Functional Mixed-Effects Model: SSM Formulation

y cit = µct + ξcit + εcit

yncit = µnct + ξncit + εncit

• An integrated random walk (IRW), such as µct , satisfies

µct = µct−1 + λct−1 Level equation

λct = λct−1 + ηct Slope equation

• AR(1) deviation curve, such as ξcit , satisfies

ξcit = φcξcit−1 + νcit

αααt = (µct λct ξc1t ξc2t · · · ) combined state

Dim(αααt) = (2 + 22 + 2 + 69) = 95

An IRW is an example of cubic-smoothing-spline, and AR(1) is anexample of exponential-smoothing-spline


Estimated Mean Curves and Their Difference

µct and µnct (Cubic Smoothing-Splines) Contrast (µct − µnct )

• Hypothesis: (µct − µnct ) < 0 for 0 ≤ t ≤ 8

• Conclusion: Not sufficient evidence


Decomposition of a Non-conceptive CurveCurve1 and the Group Mean µnct Deviation Curve ξnc1t (an AR(1) process)

Curve1 and (µnct + ξnc1t ) Observation Errors εnc1t


Modeling Long-Term Temperature Trends• Monthly temperatures at three locations in Northern

Hemisphere (UAH, CRU, and GISS) are jointly modeled.• CRU: start January 1850• GISS: start January 1880• UAH: start December 1978

• The analysis uses data up to 2012


A Model Proposed by C. Ansley and P. de Jong

GISSt = µt + a ζt + a r1 ε1t

CRUt = βcru + µt + a ζt + a ε2t

UAHt = βuah + µt + a ζt + a r3 ε3t

• βcru and βuah are intercepts that are associated with CRU andUAH, respectively.

• µt is an integrated random walk trend.

• ζt is a zero-mean, autoregressive (AR(1)) term (which isscaled by a scaling factor a).

• εit (i = 1, 2, 3) are independent observation errors withdifferent variances that are also scaled suitably.

• Note that the trend µt and the autoregressive term ζt areshared by the models of the three series.


Long-Term Temperature Trends

Adjusted CRUt = CRUt − βcru, andadjusted UAHt = UAHt − βuah.


Long-Term Temperature Trends

• The left panel shows the long-term forecasts of µt .

• The right panel shows the long-term forecasts of the slope ofµt .

• It appears that there has been statistically significant warmingover the past few decades. However, for the past decade thewarming does not appear to be accelerating.


Panel Study of Income Dynamics• Yearly wages of 595 subjects are monitored over 7 years.• Several more variables that could affect the wages are also

recorded.• Goal is to explain the effect of these variables on the wages:

• How does an extra year of education (ED) improve the wages?• What is the impact of being black (BLK), female (FEM),

member of a union (UNION), married (MS)?• Does living at a location matter: Southern state (SOUTH),

metropolitan area (SMSA)?• Does type of work matter: manufacturing (IND), blue-collar

(OCC)?

• Variables such as gender and race are fixed during the study.• Variables such as union membership and marital status can

change during the study.• This study is discussed in Cornwell and Ruppert (1988) and

Baltagi (2008). They analyze these data usinginstrument-variable methods (which are not discussed in thisillustration).


Income Dynamics: A State Space Model

• lwageit denotes the log-wage for the ith subject in year t

• lwageit = Xitβββ + µt + ξit + εit .• Xitβββ denotes the contribution of regression variables• µt denotes the overall wage trend (modeled as an IRW)• ξit denotes the subject-specific deviation from the overall trend

(modeled as a random walk). For identifiability, itsinitialization is nondiffuse.

• εit are independent noise values

• (µt + ξit) represents the subject-specific wage curve withoutthe regression contribution. It accounts for the within-subjectcorrelation.

• Because of the large state size (about 600), PROC SSM takesabout an hour to fit this model! Prime candidate for efficiencyimprovement in a future version.


Income Dynamics: Regression Estimates

Regression

Variable Estimate StdErr tValue Probt

wks 0.000467 0.0005598 0.83 0.4040

south -0.064715 0.0211267 -3.06 0.0022

smsa 0.053692 0.0160965 3.34 0.0009

ms -0.024144 0.0187401 -1.29 0.1976

exp 0.029991 0.0027906 10.75 <.0001

exp2 -0.000500 0.0000645 -7.75 <.0001

occ -0.040904 0.0125423 -3.26 0.0011

ind 0.018552 0.0132022 1.41 0.1599

union 0.039126 0.0129118 3.03 0.0024

fem -0.408879 0.0398332 -10.26 <.0001

blk -0.128915 0.0445815 -2.89 0.0038

ed 0.059470 0.0044599 13.33 <.0001


A Subject Curve with and without Regressors


A Subject Curve with and without Regressors


Nowcasting the US Economy: The ADS Index

• The Aruoba-Diebold-Scotti (ADS) business conditions index isdesigned to track real business conditions at high frequency(for more information about this index, seehttp://www.philadelphiafed.org/research-and-data/

real-time-center/business-conditions-index/).

• The ADS index:• Is based on an SSM formulation of an elaborate dynamic

factor model (DFM)• Is a highly sensitive index of the US business conditions that

pools together information from six business indicatorsmeasured at different frequencies

• This illustration shows how to create a similar index by usinga slightly simpler DFM.

http://www.philadelphiafed.org/research-and-data/real-time-center/business-conditions-index/

http://www.philadelphiafed.org/research-and-data/real-time-center/business-conditions-index/


Six Business Indicators and Their FRED IDs• The model uses logged and differenced versions of these indicators. The jobless claims are

not differenced.• These variables are freely available from many data sources. Federal reserve’s FRED is one

such data repository.

Name FRED ID Frequency Description

ld payemp PAYEMS Monthly Payroll employmentld pinc W875RX1 Monthly Real personal income excluding current transfer receiptsld mnfctr CMRMTSPL Monthly Real manufacturing and trade industries salesld indpro INDPRO Monthly Industrial production indexld gdp GDPC1 Quarterly Real GDPl icsa ICSA Weekly Initial jobless claims

• All this information is collected in a single table.• The table has seven columns: date, and the six indicators: Y1 — Y6• The data are treated as daily with variables nonmissing only on the dates they are

published.


Nowcasting the US Economy: Dynamic Factor Model

yit = intercepti + βi ∗ irwt + εit 1 ≤ i ≤ 5

y6t = µt + β6 ∗ irwt + ε6t

• y1t to y6t are proxies for the national economic activity.

• Plots of y1t to y5t (which are differenced) show them to behovering around their mean levels, with periods of deviationfrom this level.

• The plot of y6t (which is not differenced) shows a pronouncednonstationary pattern. To account for this, its model has atime-varying intercept µt (an IRW).

• Appropriately scaled common term, irwt , is present in all themodels (also an IRW). It is called a factor, and the scalarsβ1, . . . , β6 are called the associated factor loadings. Foridentifiability, the initial condition of irwt is taken to be 0.

• irwt can be treated as a pooled measure of economic activity.


Estimated Factor Loadings

• y1t to y5t are positively correlated with the economy, whereasy6t (initial jobless claims) is negatively correlated.

• For identifiability purposes, β1 is taken to be 1.

• Therefore, the estimates of β2, . . . , β5 are expected to bepositive and the estimate of β6 is expected to be negative.

Parameter Estimate StdErr tValue

beta2 1.15 0.1276 8.98

beta3 1.96 0.2390 8.20

beta4 2.48 0.1646 15.08

beta5 3.27 0.2653 12.33

beta6 -96.48 9.6151 -10.03


Estimated Economic Activity Index


Estimated Slope of the Economic Activity Index


Yield Surface Modeling: Dynamic Nelson-Siegel Model• Monthly data on the yields of 17 bonds of different maturities

(3 months to 10 years)• Sample has yields from Jan 1970 to Dec 2000.• For λ > 0 and βββ = (β1 β2 β3), the static Nelson-Siegel model

postulates the yield for a bond with maturity τ as

θ(τ ;λ,βββ) = β1+β2

(1− exp(−λτ)

λτ

)+β3

(1− exp(−λτ)

λτ− exp(−λτ)

)

• β1 is the overall yield level

• β2(1−exp(−λτ)

λτ

)is the correction for short-term bonds

• β3(1−exp(−λτ)

λτ − exp(−λτ))

is the correction for

medium-term bonds

• Let yt(τ) denote the actual bond yield at time t for a bond ofmaturity τ . A dynamic Nelson-Siegel model:

yt(τ) = θ(τ ;λt ,βββt) + εt,τ εt,τ ∼ white noise

• βββt = (β1t β2t β3t) ∼ VAR(1)• λt modeled parametrically


Estimated Yield Surface


Estimated β1t : Long-Term Bond Yields


Estimated β2t

(1−exp(−λtτ)

λtτ

)


Yield Curve for Bond with Maturity = 42 Months• A bond with maturity of 42 months does not exist.• An artificial bond that matures at 42 months is added to the data set with yields missing at all time points.• The estimated yield according to the fitted model is calculated.


Yields of Bonds with Maturities of 36, 42, 48 Months


Analyzing Dose Response with One-Compartment Model• Let Ct denote the serum concentration of theophylline (used

to treat lung disease) at time t after an oral dose of d0.• One-compartment model for Ct with absorption rate ra,

elimination rate re , and clearance rate c is:

Ct = d0 c [exp(−ret)− exp(−rat)], c > 0, ra > re > 0

= d0 λt

• Serum concentrations y jti are available on 12 subjects,measured at irregular time points during the day.

• The trajectory of the jth subject is modeled as

y jti = d0j λt + ζ jt + εjti , j = 1, 2, . . . , 12, i = 1, 2, . . . , pjt

• λt is taken to be a biexponential smoothing-spline,subject-specific deviations ζ jt are modeled by continuous-timeRW (Brownian motion), and εjti are independent noisevariables.

• This model has a state space form.


Estimate of Biexponential Smoothing Spline

• The left panel shows the estimate of λt , and the right panelshows the estimate of its derivative.

• The estimate of λt clearly shows many characteristics of thebiexponential form; however, it does not need to be a memberof that family. It is a smoothing spline—that is, it balancestwo competing considerations: closeness to the observed dataand closeness to its favored biexponential form.


Model Fit for Subjects 1 and 2


Estimated Deviation Curves for Subjects 1 and 2


Statistical Software for State Space Modeling

• Journal of Statistical Software Special Volume: StatisticalSoftware for State Space Methods (2011)

• http://www.jstatsoft.org/v41• Features commercial and open-source vendors• Some vendors offer nonlinear and/or Bayesian state space

modeling• Number of downloads of articles from this issue:

• R packages for state space modeling: 35,000+• Open-source MATLAB package (named SSM): 30,000+• State space modeling with SAS (written by me!): 9000+

• PROC SSM also released in 2011• Not featured in the JSS special volume• Three more updates since the first release• Vigorous development will continue for several more years

• R packages and other vendors probably have updates too

• Would be nice to have another special volume to assess thecurrent state!

http://www.jstatsoft.org/v41


Software for Linear State Space Modeling: Desirables• Must handle sufficiently general models

• Time-varying system matrices• Diffuse initial condition• Regression terms in observation and state equations• ...

• Must handle variety of sequential data types• Univariate/multivariate time series and longitudinal• Multifrequency• Complex survey designs• ...

• Ease of model specification• Flexible language for complex model specification• Easy ways to specify commonly needed models• ...

• Rich output• Tabular and graphical diagnostics, structural break analysis• Rich language for component specification (for output)• ...

• Scalable and efficient


PROC SSM: Self Assessment (Honest!)

• Must handle sufficiently general model: A-• Time-varying system matrices• Diffuse initial condition• Regression terms in observation and state equations

• Must handle variety of sequential data types: A-• Univariate/multivariate time series and longitudinal• Multifrequency• Complex survey designs

• Ease of model specification: B-• Flexible language for complex model specification: A-• Easy ways to specify commonly needed models: C+ (a lot

more possible)

• Rich output: B-• Tabular and graphical diagnostics, structural break analysis• Rich language for component specification (for output)

• Scalable and efficient: B- (a lot more possible)


References

• Books:• Durbin, J., and Koopman, S. J. (2012). Time Series Analysis

by State Space Methods. 2nd Ed. Oxford: Oxford UniversityPress.

• Harvey, A. C. (1989). Forecasting, Structural Time SeriesModels, and the Kalman Filter. Cambridge: CambridgeUniversity Press.

• SAS/ETS Procedure Documentation:• PROC SSM (general state space modeling):http://support.sas.com/documentation/cdl/en/etsug/

67525/HTML/default/viewer.htm#etsug_ssm_toc.htm• PROC UCM (for univariate state space modeling):http://support.sas.com/documentation/cdl/en/etsug/

67525/HTML/default/viewer.htm#etsug_ucm_toc.htm

• White paper on state space models for longitudinal data:http://support.sas.com/resources/papers/

proceedings15/SAS1580-2015.pdf



http://support.sas.com/documentation/cdl/en/etsug/67525/HTML/default/viewer.htm#etsug_ucm_toc.htm

http://support.sas.com/documentation/cdl/en/etsug/67525/HTML/default/viewer.htm#etsug_ucm_toc.htm



Documents

State Space Modeling of Sequence Data...State Space Model and Notation Y t = Z t t + X t + t Observation equation t+1 = T t t + W t+1 + t+1 State transition equation 1 = A 1 + W 1