Forecasting with DSGE models

Forecasting with DSGE models

Michał Rubaszek

SGH Warsaw School of Economics

1

Theme A: Prerequisites

Michał Rubaszek

SGH Warsaw School of Economics

2

Michał Rubaszek, DSGE Forecasting

1. Moving average representation of a model

2. Calculating impulse-response functions

Backward looking models (AR/VAR models)

Forward looking models

DSGE models

3. From moving average representation to applications

VAR models

DSGE models

3

Plan for today

Moving Average representation

4


Definition of IRF

Impulse response function – IRF:

describe how variable �� reacts over time to exogenous impulse ��.

Moving Average representation (for a linear model):

�� + �� …

Formula for IRF:

� ��

How to calculate IRF for a model?

Transform a VAR/DSGE model to MA representation

5


Definition of IRF

Moving Average representation: �� + �� …

Formula for IRF: � ��

��

Example: A model for GDP growth rate at home and abroad

��∗�� 0.250.50 � 0.50 0.000.25 0.25��∗�� 0.25 0.000.10 0.20 ��∗

�� 0.125 0.000.005 0.10 ��∗�� ⋯

what is the interpretation of � and ��?

6


Plot of IRF

7

Source: Rubaszek & Uddin (2020)


Calculating IRF: AR(1) model

AR model without a constant:

�� !�� 1 " !# ��

�� 1 " !# �� !�� !�� ⋯ $� lim(→*!(+ ��( ,�� !�

Exercise: Calculate MA representation for �� 0.8��

8



AR model with a constant:

�� . � !�� Substitute:

/� � �� " � , where � � 01

and think in terms of:

/� � !/� � ��

Exercise: Calculate MA representation for �� 2 � 0.5��

9



AR(2) model:

�� !�� !�� 1 " 2# 1 " 2�# ��

Hence AR(2) as a multiplication of two AR(1) processes

�� 1 " 2# /� �� 2�� /�/� � 1 " 2�# �� /� � 2�/� � ��

Exercise: Find roots of characteristic equation for �� 1.3�� " 0.4��

10


Calculating IRF: VAR(1) model

VAR(1) model:

�� 5�� 1 " 5# �� 5�� 5�� ⋯ �� 5�

Spectral decomposition:

5 � 7Λ7 [eigenvalues and eigenvectors matrices]

�� 7Λ7�� 9� � Λ�9� � �� where �9� � 7��

Now you can see that VAR is a linear combination of AR(1) processes.

11


Calculating IRF: VAR(P) model

VAR(P) model:

�� 5�� 5�� ⋯ � 5;��; � ��

Canonical form:

��∗ � 5∗��∗ � ��∗

5∗ �5 5� … 5;� 0 … …0 � … 00 0 … 0

, ��∗ �� …��=�

, ��∗ ��0…0

Hence is linear combination of > ? @ independent AR(1) processes

Exercise:

Write �� 1.3�� " 0.4�� in the canonical form.

Calculate Λ and 7 matrices

12


Calculating IRF: forward looking vars.

Univariate forward looking model:

�� A+B��C � �� 1 " A# �� A+B��C � A�+B��C � ⋯ $� lim(→*A(+ ��( ,

To calculate IRF we need to know how expectations +B��C are formulated

If �� ∼ EF they equal to + �� 0 for G H 0, hence (for |A| J 1):

KL � MLis also white notice!!!

For A H 1 the system is unstable due to the last component (�� ≡ 0)

13


Calculating IRF: forward looking vars.

Forward looking model:

�� A+ �� /� ↔ �� /� � A+B/��C � A�+B/��C � ⋯

Transform into MA representation, assuming that ML ∼ P@:

a. �� 0.8+ �� 0.5��

b. QR� � +BR��C � 0.1�� 0.8�� c. �� 0.4+ �� 0.4��

14


Blanchard-Khan method to calculate IRF

Let us write down a system for S-variate vector T� � ��/� as:

5�+� T�� 5T� � U��where:

�� -- a vector of G forward looking variables

/� -- a vector of S " G backward looking variables

�� -- a vector of IID innovations

here +� T�� $+� �� /��,

Blanhard-Khan method is a smart way to present T� in MA form:

T� � ��+ �� …

15



5�+� T�� 5T� � U��+� T�� 5T� � U�� 5 � 5�, U � 5�U�Spectral decomposition:

5 � 7Λ7

+� TV�� ΛTV� � �� TV� � 7T� , � 7ULet us introduce the following notation:

TV� � �9�/� Λ � Λ 00 Λ� V � X X�X� X�� YY�so that:

Q+� �9�� Λ�9� � Y�� " unstable equation /�� Λ�/� � Y�� " stable equation 16



Q+� �9�� Λ�9� � Y�� " unstable equation /�� Λ�/� � Y�� " stable equation From unstable equation we know that:

�9� � 0 for all cIf we use:

�9�/� � X X�X� X��/� Q�9� � X�� X�/�/� � X�� X��/�

we get:

0 � X�� X�/� �� "XX�/� � de�/�/� � Bde� � X��C/� /� � de� � X�� /� � dee/�

17



Q+� �9�� Λ�9� � Y�� " unstable equation /�� Λ�/� � Y�� " stable equation

From stable equation we can calculate:

/� � ��e9��+ �e9�� e9�� … , where ��e9 � Λ��Y�

If we take into account that:

�� de�/�/� � dee/�

We can easily calculate that:

/� � ��e��+ �e�� e�� … , where ��e � dee��e9�� + �� … , where �� de��e

18


Blanhard-Khan: exercise

Calculate IRF from monetary policy shock (��C to output (��), inflation (R�)

and interest rate (f�) in a 3-equation New-Keynesian model:

R� � A+ R�� g�� + �� " hBf� " + R�� Cf� � iR� � z�/� � !/� � ��

Apply the following parametrization:

A � 0.99, g � 0.1, h � 5, i � 1.5, ! � 0.8

19

From MA representation to empirics

20


VAR vs DSGE

21

Source: Pagan (2003) report on modelling and forecasting at the Bank of England


VAR model

22

Source: Luetkepohl (2011)


Short history of estimated DSGE model

23

Cowles Comission

Fit of DSGE model to data(Schorfheide, 2000)

Good forecasting propertiesSmets and Wouters (2003, 2007)

New Keynesian model

Sims critique

VAR models (Sims 1980)

Lucas critique

RBC model (Kydland-Prescott, 1982)

Theme B: Estimation

24


Methods of fitting DSGE model to data

Calibration Kydland and Prescott, 1982

GMM estimation of equations Hansen, 1982

IRF matching Christiano, Eichenbaum and Evans, 2005

Maximum likelihood estimation Ireland, 2004

Bayesian estimation Schorfheide, 2000; Smets and Wouters, 2003

25


1. Maximum likelihood step-by-step

2. Maximum likelihood in Dynare

3. Bayesian estimation in Dynare

Underlying model is based:

Ireland P., 2004. TECHNOLOGY SHOCKS IN THE NEW KEYNESIAN MODEL.

Review of Economics and Statistics 86(4): 923–936 (http://irelandp.com/pubs/tshocksnk.pdf)

With the codes that are based on materials available on:

Peter Ireland webpage: http://irelandp.com/programs.html

26

Plan for today


27

Underlying model

Habits ℎ� � m B�� " ℎ�� ℎ��(C

Dynamic IS ℎ� � + ℎ�� " no f� " +BR�� C � ��p " ��p " ��(

Phillips curve R� � �qr sR� � A+ R�� t qt

t hu�� hvℎ�

MP rule f� � !f� � 1 " ! 2wR� � 2�� 2x� �� " �� ( � ��y

Shocks ��z � !z��z � ��z for f ∈ |}, �, ��GDP QoQ ��( � �� " �� (

PGDP QoQ ��( � �= � R�

FEDrate f��( � �z � 4f�Notes: a model with nonstationary technology, /� � �e � /� � ��(

Maximum likelihood estimation

28


To derive likehood of observation �� conditional on information set ℐ� ��, ��, … , �� given parameter �ℒB��|ℐ�, �C

Write down the (linear) DSGE model in a state-space representation:

}� � �}� � �� , �� ∼ �B0, ΦC state transition equation

�� }� measurement equation

where matrices �, ��, � and Φ are functions of deep parameters �.

Apply Kalman filter to derive:

��|ℐ� ∼ �B��|�� , Σ�|�� C29

Likelihood function of DSGE models


DSGE model in a state-space representation:

}� � �}� � �� , �� ∼ �B0, ΦC state transition equation

�� }� measurement equation

Kalman filter

Prediction stage:

}�|� � �}�|� Σ�|� � �Σ�|�� Φ��

��|�� }�|� Σ�|�� Σ�|��

KL|�L� ∼ �B�L|L�K , �L|L�K C Kalman gain and update stage:

�� Σ�|�� Σ�|��

}�|� � }�|� � ��B�� " ��|�� C Σ�|� � B� " ��CΣ�|�30

Likelihood function of DSGE models


The likelihood of all observations �:� � �, ��, … , �� given parameter � is:

ℒ �:� � � Π�� ℒB��|ℐ�, �Cwhere ℒ �� ℐ�, � � ��B��|�� , ��|�� C is the pdf of normal distribution.

How to find the maximum value of ℒ �:� � ?

fmincon funtion in Matlab:

problems if cliffs, spikes, non-continuities

csminwel by Sims:

probably most efficient for simple cases

Monte-Carlo based optimization routine:

very time consuming but (nearly) guarantees to find the global maximum

31

Maximum likelihood estimation of DSGE models


ML estimation – additional topics:

We use Kalman filter for gaussian shocksAvailable extensions for non-gaussian shocks

We consider here linear modelsAvailable extensions for nonlinear models

Shocks in measurment equation

Maximum likelihood not so popular due

to "flat regions"

32

Maximum likelihood estimation of DSGE models

B. Introduction to Bayesian analysis

33


Bayes theorem

For events A and B the Bayes theorem is:

� 5 U � � U 5C�B5� U

Explanation:

� 5, U � � 5 U � U � � U 5 �B5C

34


Bayes theorem in econometrics

For parameters � and data � the Bayes theorem implies:

� � � � � � �C�B��

�B�C - prior pdf of parameters

� � � - probability of data given �� - posteriori pdf

�B�C - marginal likelihood of data (does not depend on �)

To derive posterior of � we substitute � � � by likelihood ℒB�|�C:

� � � ∝ ℒ � �C�B�

35


Bayes rule in econometrics: illustration

� � � ∝ ℒ � �C�B�

36


Conjugate prior

� � � ∝ ℒ � �C�B�In some class of models the posterior distribution is in the same

family as the prior distribution. In this case we say about conjgate prior

Example:

�B�C Beta distribution

ℒ � � Likelihood of binomial distribution

� � � Beta distribution

37


Conjugate prior: Beta + Binomial

� ∼ U£c¤ ., A

+ � � 00�q

7¤Y � � 0q0�q ¥B0�q�C

� � � Γ . � AΓ . Γ A �0 1 " � q

0 § � § 1

38


Conjugate prior: example

Two students (A and B) like to play chess. They have already played F Times and

student A won � times (and lost F " �C. Let � be the parameter that describes

the probability of student A success.

Prior: � � � ¨ 0©�q©¨ 0© ¨ q© �0© 1 " � q© � ∼ U£c¤ .�, A�

Likelihood: ℒ �|� � F� �ª 1 " � B«ªC �|� ∼ UBF, �C

Posterior: � �|� ∝ � 0©�ª 1 " � q©�«ª �|� ∼ U£c¤ ., A. � .� � � ; A � A� � BF " �C� �|� � ¨ 0�q

¨ 0 ¨ q �0 1 " � q

Notice: in formula for �B�|�C we omitted ¨ 0©�q©

¨ 0© ¨ q© and F� . Why?

39


Metropolis-Hastings Markov Chain

Monte Carlo (MCMC) algorithm

In most cases we can calculate � �|� for any value of �, but cannot derive the distribution of �|� or transformation ®B�|�), e.g. IRF. In this cases we resort to numerical methods, e.g. Metropolis Hastings MCMC

1. Set the initial value of parameter � z for f � 02. Draw �∗ � � z � s�, where � ∼ F 0, Σ and s is a step length

3. Draw ¯ ∼ ° �,4. Calculate / � � �∗ � /� � z � and compare it to ¯

If / § ¯ then � z� � �BzC If / H ¯ then � z� � �∗

5. Repeat steps 2-4 F(zy times

6. Using the sample � z for f � F�²³´ � 1, … , F(zy calculate descriptive statistics for �|� or ®B�|�)

40

Bayesian estimation of DSGE models

41


Bayesian estimation of DSGE

Why Bayesian estimation of DSGE models

Helps to avoid the problem of "flat likelihood"

A nice method to add "extra" information about deep parameters

Competitive forecasts

What kind of priors for individual parameters (in Dynare):

Beta: beta_pdf

Gamma: gamma_pdf

Normal: normal_pdf

Uniform: uniform_pdf,

Inverse gamma: inv_gamma_pdf

42



Two steps in Bayesian estimation of DSGE models:

Find the posterior mode

the same algorithms as in ML estimation

Approximate posterior distribution using MCMC techniques

a need to set step length, covariance matrix, numer of draws, etc.

Posterior distribution of parameters calculated with MCMC:

+B® �C � � µ ®B�C�B�|�C��Remarks:

Parameter space of DSGE models usually large

many MCMC draws needed to achieve convergence

Calibration of selected parameters warranted

equivalent to assigning zero variance prior

43


Bayesian estimation of DSGE in Dynare

estimation(OPTIONS)[VARIABLE_NAME];

datafile FILENAME: name of your data file (with observables)

mode_compute = INTEGER: algorithm to find the posterior mode

mode_check: plot likelihood and posterior

mh_nblocks = INTEGER: number of Metropolis-Hastings chains

mh_replic = INTEGER: number of MH draws

mh_drop = INTEGER: burn-in sample

mh_jscale = DOUBLE: step length

mode_file = FILENAME: starting values for mode calculation

nobs = INTEGER: set the number of observations

bayesian_irf: returns impulse response functions

44


Posterior output in Dynare

45


Bayesian IRF

46


Marginal data density - MDD

Marginal data density (also called marginal likelihood):

� � d � µ � � �, d � � d �� Derived wih numerical MCMC simulations

MDD used to compare a posteriori probabilities of two models:

¶·z¸ � � dz �� d � � � � dz � dz

� � d � dwhere ¶· is the posterior odds ratio.

Important: comparison valid only for equal data sets!!!

47



MCMC convergence diagnostics (two or more chains needed)

Brooks and Gelman convergence diagnostics

Help to check whether distribution of posterior is the same:

B: between two (or more) chains (should converge to 0)

W: within the same chain (should converge to a constant)

Red line: W (should stabilize)

Blue line: W+B (should converge to W)

48


Post-estimation commands:

Post-estimation commands:

shock_decomposition [VARIABLE_NAME]

stoch_simul [VARIABLE_NAME]

forecast (OPTIONS. . . ) [VARIABLE_NAME]

49

0 20 40 60 80 100 120 140-1.5

-1

-0.5

0

0.5

1

Initial values

ni_s

ni_m

ni_d

Theme C: Forecasting

50


Introduction

The ultimate goal of a positive science is to develop a theory or hypothesis

that yields valid and meaningful predictions about phenomena not yet

observed. Theory is judged by it's predictive power.

A hypothesis can't be tested by its assumptions. What is important is

specifying the conditions under which the hypothesis works. What matters

is it's predictive power, not it's conformity to reality.

Milton Friedman, 1953. The Methodology of Positive Economics.

in Essays in Positive Economics: University of Chicago Press.

51


Economic forecasting - introduction

Types of time series forecasts Qualitative / model-based Be.g. from VAR/DSGE modelC Quantitative / expert based Be.g. survey forecast, SPFCGeneral characteristics of time series forecasts: Forecasting is based on the assumption that the past predicts the future

Think carefully if the past is related to what you expect about the future

Forecasts are always wrongHowever, some models/methods might work better or worse than the other

Forecasts are usually more accurate for shorter time periodsBut, economic theories are more informative for longer horizon

52


Introduction

Common opinion in early 2000s:

DSGE models are too stylized to achieve reasonable t to the data

A break-through came with Smets and Wouters (2003):

richly specified DSGE model can generate better forecasts than BVARs

Later studies confirm that DSGE model-based forecasts competitive

with time series models or professional forecasters:

Adolfson et al. (2007, EconRev); Lees at al. (2007); Edge et al. (2010); Edge

and Gurkaynak (2010); Christoffel et al (2011, Oxf. HandEconFct); Del Negro,

M., Schorfheide (2012, HandEconFct); Gurnkaynak et al. (2013, CEPR WP),

Wolters (2015. JAE), Bekiros & Paccagnini (2016, JoF),

Including my contribution: Rubaszek and Skrzypczynski 2008 (IJF), Kolasa,

Rubaszek, Skrzypczyński (2012, JMBC), Kolasa & Michal Rubaszek (2015 IJCB);

Kolasa & Michal Rubaszek (2015 IJF); Ca' Zorzi, Kolasa and Rubaszek (2017,

JIE); Kolasa & Rubaszek (2018, IJF), Rubaszek (2020, IJF R&R)

53


Introduction

Increasing popularity of DSGE models in policy making institutions is

reflected by the fact that they are used to assist the forecasting exercise

in many central banks: EDO/SIGMA/FRBNY (FED), NAWM (ECB), European

Commission (QUEST), ToTEM (BoC), BEQM (BoE), Ramses (Riksbank),

Nemo (Norges Bank)

See presentations from ECB conference "DSGE models and forecasting" https://www.ecb.europa.eu/pub/conferences/html/20160922_dgse_models_forecasting.en.html

An increasing popularity of DSGE models in academia is also reflected by

a chapter in the Handbook of Economic Forecasting on DSGE-model

based forecasts (Del Negro and Schorfheide, 2013)

54


Plan for today

How to produce a forecast from a DSGE model

1. Generating density/point forecasts from a DSGE model

2. Constructing a fan chart

3. Decomposing a forecast

4. Conditional forecast

55


56

Underlying model




t hu�� hvℎ�

MP rule f� � !f� � 1 " ! 2wR� � 2�� 2x� �� " �� ( � ��y


PGDP QoQ ��( � �= � R�



Given:

the value of model parameters � the state-space representation of the model

}� � �B�C}� � ��B�C �� state transition equation

�� B�C}� measurment equation

future path of structural shocks: ��:��Æ � |��, ��, … , ��Æ � the value of }�B�C obtained from Kalman smoother

We can recursively calculate a forecast for observed variables:

��:��Æ � |��, ��, … , ��Æ �

57

Generating a forecast from a DSGE model


An algorithm to generate point forecast at mean/median/mode

1. Assume that over the forecast horizon shocks are zero: ��m � 0 for ℎ H 0(their expected value at time T)

2. Calculate �∗ as posterior mean/median/mode for �3. Apply Kalman smoother to calculate }�B�∗C4. Calculate ��:��Æ

IMPORTANT: this is not full Bayesian approach!!!

58

Point forecast at mean / median / mode


To calculate FB forecast we need to approximate the predictive density:

�B��:��Æ|�:� C � µ �B��:��Æ|�:� , �C�B�|�:� C�� We do it by drawing from �B��:��Æ|�:� C using the following algorithm:

1. Draw �z from �B�|�:� C for f � 1, 2, … , F2. Calculate }�B�zC with Kalman smoother

a. Draw d paths for shocks: ��:��Æz,¸ B�zC for Ç � 1,2, … , db. Calculate forecast ��:��Æz,¸

Using d ? F draws obtained with the procedure described above we can

approximate the predictive density

The FB point forecast (at mean) is calculated with numerical integration:

+B��:��Æ|�:�C � 1d ? F È È ��:��Æz,¸

¸z59

Full Bayesian point forecast


Based on d ? F draws ��:��Æz,¸ from predictive density it is strightforward to

calculate interval forecast

¶ #U��m,0 § ��m § °U��m,0 � 1 " .and put it on the graph for various . in the form of the fan chart

60

Fan chart

2015 2016 2017 2018 2019 2020 2021 2022-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5fanchart for GDP QoQ

perc

ent,

QoQ


Sometimes we have some external information that can be used in

forecasting, which can increase forecast accuracy (Kolasa, Rubaszek,

and Skrzypczynski, 2012; Del Negro and Schorfheide, 2013, Wolters,

2015)

To calculate conditional forecast we can choose a sequence of shocks ��|��Æ that generate the desired forecast for a variable of interest

In general, there is no unique way of doing it - we need to choose which

shock we want to use for conditioning

Sometimes the choice is natural: use monetary policy shocks to

condition the forecast on a given interest rate path

IMPORTANT: it makes a big difference whether we treat the shocks as

anticipated or unanticipated

61

Conditional forecast


Conditional forecast

62


VMA representation of DSGE model is:

�� Ψ��+Ψ�� Ψ�� …, where Ψ� � Ê�,z¸ ´?y

For f�m variable we can therefore write down:

�z� � �z � ∑ Ê�,z¸�¸�+Ê,z¸�¸,� � Ê�,z¸�¸,��y� � …

Given that +� ��m � 0, we can also decompose forecast into:

+�B�z,��ÆC � �z � ∑ ÊÆ,z¸�¸�+ÊÆ�,z¸�¸,� � ÊÆ��,z¸�¸,��y� � …

IMPORTANT: DSGE forecast is a dynamic transition from �� to its

posterior mean � (see discusssion in Ca'Zorzi, Kolasa, Rubaszek, 2017JIE)

63

Historical decomposition


64

Historical decomposition

Theme D: Forecasting competitions

65


Introduction

Common opinion in early 2000s:

DSGE models are too stylized to achieve reasonable t to the data

A break-through came with Smets and Wouters (2003):

richly specified DSGE model can generate better forecasts than BVARs

Later studies confirm that DSGE model-based forecasts competitive

with time series models or professional forecasters:

Adolfson et al. (2007, EconRev); Lees at al. (2007); Edge et al. (2010); Edge

and Gurkaynak (2010); Christoffel et al (2011, Oxf. HandEconFct); Del Negro,

M., Schorfheide (2012, HandEconFct); Gurnkaynak et al. (2013, CEPR WP),

Wolters (2015. JAE), Bekiros & Paccagnini (2016, JoF),

Including my contribution: Rubaszek and Skrzypczynski 2008 (IJF), Kolasa,

Rubaszek, Skrzypczyński (2012, JMBC), Kolasa & Michal Rubaszek (2015 IJCB);

Kolasa & Michal Rubaszek (2015 IJF); Ca' Zorzi, Kolasa and Rubaszek (2017,

JIE); Kolasa & Rubaszek (2018, IJF), Rubaszek (2020, IJF R&R)

66


Introduction

67


Plan for today

Forecasting competition: BVAR vs DSGE

1. Generate recursive forecasts from DSGE/BVAR

2. Compare the quality of point forecasts

3. Compare the quality of density forecasts

4. Make some nice graphs

68


69

Underlying model




t hu�� hvℎ�

MP rule f� � !f� � 1 " ! 2wR� � 2�� 2x� �� " �� ( � ��y


PGDP QoQ ��( � �= � R�


Forecast error: some theory

70


Forecast error in known DSGE/VAR model

Assume we know DGP, i.e. the parameters and the specification of DSGE/VAR.Hence, we know the parameters of infinite moving average representation�� Ê�� Ê��+ Ê�� + ÊÏ��Ï … �� ∼ FB0, �C

Forecast from known DGP is called optimum forecast. We cannot obtain more accurate forecast from another model. Forecast error of optimal forecast is solely due to futures shocks Brandom errorC:

��m " ��m|� � Ê��m � Ê��m+⋯ + Êm�� The resulting variance of forecast is:

+ ��m " ��m|� � � Ê�Ê�� ÊÊ� +⋯ + ÊmÊm�

71


Forecast error in estimated DSGE/VAR model

Assume that we don't know the true DGP but use a model d instead The variance of our forecast is:

Component A: error of "optimum forecast" Bsee previous slideCComponent B: estimation / misspecification error we want to minimize this valueComponent C: equals to 0 if we cannot forecast future shock

72


Estimation / misspecification error

Let us focus on the estimation / misspecification error and model complexity+| ��m|� " ��m|�Õ ��

I. Large / complex models many parameters � large estimation error Bhigh varianceC many explanatory variables � good specification Blow biasC

II. Small / simple models few parameters � small estimation error Blow varianceC few explanatory variables � potential misspecification Bhigh biasC

Which effect dominates? We don't know and need to check it

73


Illustration of the variance / bias trade-off

74



Let as assume that the true DGP is ARB1C:�� ! �� " � � ��

We have a sample of 180 monthly observations B15 yearsC for �� and would like to decide on one of the three competing models:RW, Random walk: �ØÙ � 0 and !ØÙ � 1HL, 5-year half life model: �ÆÚ � �Û and !ÆÚ � 0.5/Ü�AR, estimated AR model: �ÝØ and !ÝØ are estimated

Which model performs best? It depends on the value of !75



76

Source: Ca' Zorzi M., Mućk J., Rubaszek M., 2016. RER forecasting and PPP: This time the Random Walk loses, Open Economies Review

Ex-post forecast errors

77


About ex-post forecast

We usually work with models that performed well in the past In ex-post forecast we ask a question :how accurate forecasts the model would deliver if it was used in the past We evaluate ex-post forecasts to be sure about model reliability An important issue is the use of "real time data, RTD"

78


About ex-post forecast

We compare forecast ��,mÞ from model dz to realization ��m to assess: the absolute quality of forecasts from model dzMFE, effciency/unbiasedness tests, sequential forecasts, PIT the relative quality of forecasts from models dz and d

RMSFE/MAE, predictive scores BLPS/CRPSC

Various forecasting schemes rolling scheme recursive schemes fixed schemes

A very important choice relates to the split of the sample intoestimation and evaluation subsamples

79


Recursive forecasting scheme - illustration

80

Source: Barbara Rossi, 2014. Density forecasts in economics and policymaking, CREI WP 37


Point forecasts accuracy measures

RMSFE - Root Mean Squared Forecast Error: dà�+m � 1

ám È ��m " ��,mâ ��m

��where Tm � á " á " ℎ � 1MSFE - Mean Squared Forecast Error:

dà�+m � 1ám È ��m " ��,mâ ��m

��MAFE - Mean Absolute Forecast Error:

d5�+m � 1ám È |��m " ��,mâ |

�m

��

81



Diebold-Mariano test for equal forecast accuracy

Forecast errors from two competing models £�,m � ��m " ��,mâand £��,m � ��m " ��,mâ

The quadratic loss function ��,m � £�,m� " £��,m�The null of equal forecast accuracy (RMSFE) ã�: +B��,mC � 0

Test statistic:

äd � ��,mà/ám

∼ F 0,1

where à � ∑ æziçzÚz�Ú is the estimate of ``long-term’’ variance

Important: loss function does not necessary need to be quadratic!

82



MFE - Mean Forecasts Error

d�+m � 1ám È B��m " ��,mâ C

�m

��

83

Source: M. Kolasa & M. Rubaszek & P. Skrzypczyński, 2012. Putting the New Keynesian DSGE Model to the Real‐Time Forecasting Test,

Journal of Money, Credit and Banking



84



RMSFE / DM test example



RMSFE – graphical illustration

85

Source: Ca' Zorzi M. & Kocięcki A. & Rubaszek M., 2015. Bayesian forecasting of real exchange rates with a Dornbusch prior, Economic Modelling

Notes: Each line represents the ratio of RMSE from a given method to RMSE from the random walk, where values below unity indicate better

accuracy of point forecasts. The straight and dotted lines stand for VAR1 and VAR2, respectively. The forecast horizon is expressed in months.



Efficiency / unbiasedness test – graphical illustration

86





Efficiency / unbiasedness test

For regression:

��m � .� � .��,mâ � ��,mwe test whether .� � 0 and . � 1.

[ the alternative specification is £�,m � .� � .��,mâ � ��,m in which we test .� � 0 and . � 0 ]

87





88

Source: Ca’ Zorzi M. & Kolasa M. & Rubaszek M., 2017. Exchange rate forecasting with DSGE models, Journal of International Economics


Density forecasts accuracy measures

PIT – Probability Integral Transform

¶�á�,m � é ��,mâ ¯ �¯��ê

*� ¶�,mâ ��m ∈ $0,1,

where ��,mâ B·C and ¶�,mâ B·C is the forecast pdf and cdf, respectively.

89



PIT – Probability Integral Transform For a well calibrated model the series ¶�á�,m should be drawn from ��ä ° 0,1 We can check it through QQ plot or histogram

90





LPS – Log Predictive Score

#¶à�,m � log ¶à�,m � logB��,mâ ��m Cwhere ��,mâ BC is the forecast for density distribution.

91



Amisano-Giacomini B2007C test AG test allows to compare LPS from two competing modelsThe loss differential #�,m � #¶à�,m " #¶à��,mThe null of equal forecast accuracy ã�: +B#�,mC � 0Test statistic: �5 � ÚÛ�,ê

í/�ê → F 0,1where à is the HAC estimate of the ``long-term’’ variance for #�,m

Interpretation of average LPS difference between models: #¶à " #¶à�

average percentage difference in data fit to predictive density

* Amisano, G., Giacomini, R., 2007. Comparing density forecasts via weighted likelihood ratio tests. Journal of Business & Economic Statistics 25, 177-190.92



93

Source: Kolasa M. & Rubaszek M., 2018. Does the foreign sector help forecast domestic variables in DSGE models?, International Journal of Forecasting

LPS / AG test: example


This course is realized with support of funding from the European Union’s

Horizon 2020 research and innovation programme under the Marie

Skłodowska-Curie grant agreement No 734712

94

Documents

Forecasting with DSGE models