Upload
others
View
29
Download
0
Embed Size (px)
Citation preview
IntroductionBacktesting Principles
Testing strategiesRecommandations
Backtesting Value-at-Risk Models
Christophe Hurlin
University of Orléans
Séminaire Validation des Modèles Financiers. 29 Avril 2013
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Introduction
The Value-at-Risk (VaR) and more generally the Distortion RiskMeasures (Expected Shortfall, etc.) are standard risk measuresused in the current regulations introduced in Finance (Basel 2), orInsurance (Solvency 2) to x the required capital (Pillar 1), or tomonitor the risk by means of internal risk models (Pillar 2).
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Introduction
Denition
Let frtgTt=1 be a given P&L series. The daily (conditional) VaR fora nominal coverage rate α is dened as
Pr[ rt < VaR t jt1(α) Ft1] = α
where Ft1 denotes the set of information available at time t 1.
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Introduction
Who does use VaR? What for?
Bank risk manager Measure rm-level market, credit, op. risk
Bank executives Set limits (management)
Banking regulators Determine capital requirements
Exchanges Compute margins
Regulators Forecast systemic risk (CoVaR)
Industry Ex: EDF, spot prices of electricity
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
"Disclosure of quantitative measures of market risk, suchas value-at-risk, is enligthening only when accompaniedby a thorough discussion of how the risk measures werecalculated and how they related to actual performance",Alan Greenspan (1996)
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Introduction
DenitionBacktesting is a set of statistical procedures designed to check ifthe real losses are in line with VaR forecasts (Jorion, 2007).
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Introduction
Whatever the type of use of VaR, the VaR forecasts aregenerated by an internal risk model.
This model is used to produced a sequence of pseudo out-ofsample VaR forecasts for a past period (typically one year)
The backtesting is based on the comparison of the observedP&L to these VaR forecasts.
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Outlines
1 How to test the validity of a VaR model?2 What are the backtesting strategies?3 What are the good practices?
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Backtesting Principles
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Backtesting Principles
Remark 1: Ex-post VaR is not observable, so it is impossible tocompute traditional statistics or criteria such as MSFE.
Remark 2: There is no proxy for the VaR contrary to the volatility(realized volatility, Andersen and Bollerslev 1998)
Patton, A.J. (2011) Volatility forecast comparison using imperfect volatility
proxies, Journal of Econometrics, 260, 246-256.
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Backtesting Principles
Backtesting procedures are based on VaR exceptions
Denition
We denote It (α) the hit variable associated to the ex-postobservation of an α% VaR exception at time t :
It (α) =
(1 if rt < VaR t jt1(α)0 else
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Backtesting Principles
Christo¤ersen (1998) : VaR forecasts are valid if and only if theviolation process It (α) satises the following two assumptions:
1 The unconditional coverage (UC) hypothesis.
2 The independence (IND) hypothesis.
Christo¤ersen P.F. (1998), Evaluating interval forecasts, International Economic
Review, 39, pp. 841-862.
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Backtesting Principles
Denition (unconditional coverage hypothesis)The unconditionnal probability of a violation must be equal to theα coverage rate
Pr [It (α) = 1] = E [It (α)] = α.
If Pr [It (α) = 1] > α, the risk is under-estimated
If Pr [It (α) = 1] < α, the risk is over-estimated
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Backtesting Principles
Denition (independence hypothesis)VaR violations observed at two di¤erent dates must beindependently distributed.
It (α) and Is (α) are independently distributed for t 6= s
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Backtesting Principles
Figure: Illustration: violationscluster
0 50 100 150 200 2506
4
2
0
2
4
6
8 VaR(95%) P&L
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Backtesting Principles
Figure: Illustration: violationscluster
0 50 100 150 200 2506
4
2
0
2
4
6
8 VaR(95%) P&L
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Backtesting Principles
Denition (conditional coverage hypothesis)The violation process satises a di¤erence martingale assumption.
E [ It (α) j Ft1] = α
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Backtesting Principles
Remark: These assumptions can be expressed as distributionalassumptions.
Under the UC assumption, each variable It (α) has a Bernouillidistribution with a probability α.
Itt (α) Bernouilli (α)
Under the IND assumption, these variables are independent, andthe number of violations has a Binomial distribution.
T
∑t=1It (α) B (T , α)
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategiesWhat are the backtesting strategies?
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies
Let us consider a sequence of daily VaR out-of-sample forecastsVaR t jt1(α)
Tt=1
and the corresponding observed P&L.
How to test the validity of the internal risk model?
Hurlin C. and Pérignon C. (2012), Margin Backtesting,Review of Futures Market, 20, pp. 179-194
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies
Testing strategies:
1 Frequency-based tests
2 Magnitude-based tests
3 Multivariate tests
4 Independence tests
5 Duration-based tests
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: frequency-based tests (1/5)
Figure: BIS "Tra¢ c Light" System
Note: VaR(1%, 1 day), 250 daily observations
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: frequency-based tests (1/5)
Denition
Christo¤ersen (1998) proposes a Likelihood Ratio statistic for UCdened as:
LRUC = 2 lnh(1 α)TH αH
i+2 ln
h(1H/T )TH (H/T )H
id!
T!∞χ2 (1)
where H = ∑Tt=1 It (α) denotes the total number of exceedances.
For a nominal risk of 5%, the null of UC can not be rejected ifand only if H < 7 for T = 250 and α = 1%.
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: frequency-based tests (1/5)
Example
Berkowitz and O-Brien (2002) consider the VaR forecasts of six UScommercial banks
Berkowitz, J., and O-Brien J. (2002), How Accurate are theValue-at-Risk Models at Commercial Banks, Journal ofFinance.
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: frequency-based tests (1/5)
Figure: Bank Daily VaR Models
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: frequency-based tests (1/5)
Figure: Violations of Banks99% VaR
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: frequency-based tests (1/5)
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: frequency-based tests (1/5)
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies
Testing strategies:
1 Frequency-based tests
2 Magnitude-based tests
3 Multivariate tests
4 Independence tests
5 Duration-based tests
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: magnitude-based tests (2/5)
All these tests do not take into account the magnitude of thelosses beyond the VaR
Example
Consider two banks that both have a one-day 1%-VaR of $100million. Assume each bank reports three VaR exceptions, but theaverage VaR exceedance is $1 million for bank A and $500 millionfor bank B.In this case, standard backtesting methodologies would indicatethat the performance of both models is equal and acceptable.
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: magnitude-based tests (2/5)
Figure: Daily VaR and P/L for SocGen 2008
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: magnitude-based tests (2/5)
Figure: Daily VaR and P/L for SocGen 2008
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: magnitude-based tests (2/5)
The Risk Map
Colletaz G., Hurlin C. and Perignon C. (2013), The RiskMap: a new tool for Risk Management, forthcoming inJournal of Banking and Finance
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: magnitude-based tests (2/5)
We propose a VaR backtesting methodology based on the numberand the severity of VaR exceptions: this approach exploits theconcept of "super exception".
DenitionWe dene a super exception using a VaR with a much smallercoverage probability α0, with α0 < α. In this case, a superexception is dened as a loss greater than VaRt (α0).
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: magnitude-based tests (2/5)
Figure: VaR Exception vs. VaR Super Exception
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: magnitude-based tests (2/5)
Solution
Given VaR exceptions It (α) and VaR super exception It (α0), wedene a Risk Map that jointly accounts for the number and themagnitude of the VaR exceptions
Let us consider a given UC test with a statistic Z (α) based on theviolations sequence fIt (α)gTt=1 .
H0 : E [It (α)] = α
H1 : E [It (α)] 6= α.
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: magnitude-based tests (2/5)
Num
ber o
f VaR
Exc
eptio
ns (N
)
Nonrejection area for teston VaR exceptions
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: magnitude-based tests (2/5)
Based on the same UC test, it is possible to test for themagnitude of VaR exceptions, via the VaR super exceptionsfIt (α0)gTt=1
H0 : EItα0= α0
H1 : EItα06= α0
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: magnitude-based tests (2/5)
Nonrejectionarea for teston VaR superexceptions
Number of VaR Super Exceptions (N’)
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: magnitude-based tests (2/5)
We can also test jointly for both magnitude and frequency of VaRexceptions:
H0 : E [It (α)] = α and EItα0= α0
Multivariate approach
Perignon C. and Smith, D. (2008), A New Approach toComparing VaR Estimation Methods, Journal ofDerivatives
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: magnitude-based tests (2/5)
0 1 2 3 4 5 6 7 80123456789
101112131415
Number of VaR Super Exceptions (N')
Num
ber o
f VaR
Exc
eptio
ns (N
)
Nominal risk 5%
Nominal risk 1%
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: magnitude-based tests (2/5)
Figure: Backtesting Bank VaR: La Caixa (2007-2008)
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: magnitude-based tests (2/5)
0 1 2 3 4 5 6 7 80123456789
101112131415
Number of VaR Super Exceptions (N')
Num
ber o
f VaR
Exc
eptio
ns (N
)
Nominal risk 5%
Nominal risk 1%
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies
Testing strategies:
1 Frequency-based tests
2 Magnitude-based tests
3 Multivariate tests
4 Independence tests
5 Duration-based tests
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: multivariate tests (3/5)
Intuition: Testing the validity of the VaR model for M coveragerates, with M > 1.
Perignon C. and Smith, D. (2008), A New Approach toComparing VaR Estimation Methods, Journal ofDerivatives
Hurlin C. and Tokpavi, S. (2006), BacktestingValue-at-Risk Accuracy: A Simple New Test, Journal ofRisk
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: multivariate tests (3/5)
Perignon and Smith (2008) consider the null:
H0,MUC : E [It (α)] = α and EItα0= α0.
Let us denote:
J0,t = 1 J1,t J2,t
J1,t =
1 if VaR t jt1(α0) < rt < VaR t jt1(α)0 otherwise
J2,t =
1 if rt < VaR t jt1(α0)0 otherwise
.
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: multivariate tests (3/5)
Denition (Perignon and Smith, 2008)
The multivariate unconditional coverage test is a LR test given by:
LRMUC = 2 lnh(1 α)H0
α α0
H1 α0H2i+2 ln
"1 H0
T
H0 H0T H1T
H1 H2T
H2#.
where Hi = ∑Tt=1 Ji ,t , for i = 0, 1, 2, denote the count variable
associated with each of the Bernoulli variables.
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: multivariate tests (3/5)
Hurlin and Tokpavi (2006):
A natural test of the CC is the univariate Ljung-Box test of
H0,CC : r1 = ... = rK = 0
where rk denotes the k th autocorrelation:
LB(K ) = T (T + 2)K
∑k=1
br2kT k
d!T!∞
χ2 (K )
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: multivariate tests (3/5)
Denition (Hurlin and Tokpavi, 2006)
Let Θ = fθ1, .., θmg be a discrete set of m di¤erent coverage ratesand Hitt = [Hitt (θ1) : Hitt (θ2) : ... : Hitt (θm)]
0
Hitt (θi ) =
(1 θi if rt < VaR t jt1(θi )θi else
Under the null of CC (martingale di¤erence):
H0,CC : E[HittHit
0tk
= 0m 8k = 1, ...,K
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies
Testing strategies:
1 Frequency-based tests
2 Magnitude-based tests
3 Multivariate tests
4 Independence tests
5 Duration-based tests
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: independence tests (4/5)
LR tests
Christo¤ersen (1998) assumes that the violation process It (α) canbe represented as a Markov chaine with two states:
Π =
1 π01 π011 π11 π11
πij = Pr [ It (α) = j j It1 (α) = i ]
DenitionThe null of CC can be dened as follows:
H0,CC : Π = Πα =
1 α α1 α α
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: independence tests (4/5)
LR tests
Christo¤ersen (1998) assumes that the violation process It (α) canbe represented as a Markov chaine with two states:
Π =
1 π01 π011 π11 π11
πij = Pr [ It (α) = j j It1 (α) = i ]
DenitionThe null of IND can be dened as follows:
H0,IND : Π = Πβ =
1 β β1 β β
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: frequency-based tests (4/5)
The corresponding LR statistics are dened by:
LRIND = 2 lnh(1H/T )TH (H/T )H
i+2 ln [(1 bπ01)n00 bπn0101 (1 bπ11)n10 bπn1111 ] d!
T!∞χ2 (1)
LRCC = 2 lnh(1 α)TH (α)H
i+2 ln [(1 bπ01)n00 bπn0101 (1 bπ11)n10 bπn1111 ] d!
T!∞χ2 (2)
By denition:LRCC = LRUC + LRIND
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: independence tests (4/5)
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: independence tests (4/5)
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: frequency-based tests (1/5)
Figure: Violations of Banks99% VaR
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: independence tests (4/5)
Regression based tests
Engle and Manganelli (2004) suggest another approach basedon a linear regression model. This model links current marginexceedances to past exceedances and/or past information.
Let Hit (α) = It (α) α be the demeaned process associatedwith It (α):
Hitt (α) =1 α if rt < VaR t jt1(α)α otherwise
.
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: independence tests (4/5)
Regression based tests
Consider the following linear regression model:
Hitt (α) = δ+K
∑k=1
βkHittk (α) +K
∑k=1
γkztk + εt
where the ztk variables belong to the information set Ωt1(lagged P&L, squared past P&L, past margins, etc.)
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: independence tests (4/5)
Regression based tests
The null hypothesis test of CC corresponds to testing the jointnullity of all the regression coe¢ cients:
H0,CC : δ = βk = γk = 0, 8k = 1, ...,K .
since under the null :
E [Hitt (α)] = E [It (α) α] = 0() Pr [It (α) = 1] = α
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: independence tests (4/5)
Denition (Engle and Manganelli, 2004)
Denote Ψ = (δ β1 ...βK γ1 ...γK )0 the vector of the 2K + 1
parameters in this model and Z the matrix of explanatory variablesof model, the Wald statistic, denoted DQCC , then veries:
DQCC =bΨ0Z 0Z bΨα (1 α)
d!T!∞
χ2 (2K + 1)
where bΨ is the OLS estimate of Ψ.
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: independence tests (4/5)
Regression based tests
Extension: A natural extension of the test of Engle and Manganelli(2004) consists in considering a (probit or logit) binary modellinking current violations to past ones
Dumitrescu E., Hurlin C. and Pham V. (2012),Backtesting Value-at-Risk: From Dynamic Quantile toDynamic Binary Tests, Finance
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: independence tests (4/5)
Denition (Dumitrescu et al., 2012)
We consider a dichotomic model:
Pr [ It (α) = 1 j Ft1] = F (πt ) .
where F (.) denotes a c.d.f. and the index πt satises thefollowing autoregressive representation:
πt = c +q1
∑j=1
βjπtj +q2
∑j=1
δj Itj (α) +q3
∑j=1
γjxtj ,
where l(.) is a function of a nite number of lagged values ofobservables, and xt is a vector of explicative variables.
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: independence tests (4/5)
Regression based tests
H0 : β = 0, δ = 0,γ = 0 and c = F1 (α) .
since under the null of CC:
Pr(It = 1 j Ft1) = F (F1(α)) = α.
The Dynamic Binary (DB) LR test statistic is:
DBLRCC = 2ln L(0,F1(α); It (α),Zt ) ln L(θ, c ; It (α),Zt )
d!
T!∞χ2 (dim(Zt ))
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies
Testing strategies:
1 Frequency-based tests
2 Magnitude-based tests
3 Multivariate tests
4 Independence tests
5 Duration-based tests
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: duration-based tests (5/5)
The UC, IND, and CC hypotheses also have some implicationson the time between two consecutive VaR margin exceedances.
Let us denote by dv the duration between two consecutiveVaR margin violations:
dv = tv tv1
where tv denotes the date of the v th exceedance.
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: duration-based tests (5/5)
Under CC hypothesis, the duration process dv has a geometricdistribution:
Pr [dv = k ] = α (1 α)k1 k 2 N.
This distribution characterizes the memory-free property ofthe violation process It (α) with E (dv ) = 1/α
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: duration-based tests (5/5)
Denition
Christo¤ersen and Pelletier (2004) use under the null hypothesisthe exponential distribution:
g (dv ; α) = α exp (αdv ) .
Under the alternative hypothesis, they postulate a Weibulldistribution for the duration variable:
h (dv ; a, b) = abbdb1v exph (adv )b
i.
H0,IND : b = 1 H0,CC : b = 1, a = α
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: duration-based tests (5/5)
Drawback: we have to postulate a distribution for the durationunder the alternative (misspecication of the VaR model).
Solution: Candelon et al. (2001) propose a J-test based onorthonormal polynomials associated to the geometric distribution.
Candelon B., Colletaz G., Hurlin C. et Tokpavi S. (2011),"Backtesting Value-at-Risk: a GMM duration-basedtest", Journal of Financial Econometrics,
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: duration-based tests (5/5)
Candelon et al. (2001)
In the case of continuous distributions, the Pearson distributions(Normal, Student, etc.) are associated to some particularorthonormal polynomials whose expectation is equal to zero.
These polynomials can be used as special moments to test fora distributional assumption (see. Bontemps and Meddahi,Journal of Econometrics, 2005).
In the discrete case, orthonormal polynomials are dened fordistributions belonging to the Ords family (Poisson, Pascal,hypergeometric, etc.).
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: duration-based tests (5/5)
Candelon et al. (2001)
DenitionThe orthonormal polynomials associated to a geometricdistribution with a success probability β are dened by thefollowing recursive relationship, 8d 2 N:
Mj+1 (d ; β) =(1 β) (2j + 1) + β (j d + 1)
(j + 1)p1 β
Mj (d ; β)
jj + 1
Mj1 (d ; β) ,
for any order j 2 N , with M1 (d ; β) = 0 and M0 (d ; β) = 1 and:
E [Mj (d ; β)] = 0 8j 2 N, 8d 2 N.Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: duration-based tests (5/5)
Candelon et al. (2001)
Example
We can show that if d follows a geometric distribution ofparameter β, then:
M1 (d ; β) = (1 βd) /p1 β
withE [M1 (d ; β)] = 0
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: duration-based tests (5/5)
Candelon et al. (2001)
Our duration-based backtest procedure exploits these momentconditions.
More precisely, let us dene fd1; ...; dNg a sequence of Ndurations between VaR violations, computed from thesequence of the hit variables fIt (α)gTt=1 .Under the CC assumption, the durations di , i = 1, ..,N, arei .i .d . geometric(α). Hence, the null of CC can be expressedas follows:
H0,CC : E [Mj (di ; α)] = 0, j = f1, .., pg ,
where p denotes the number of moment conditions.
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: duration-based tests (5/5)
Candelon et al. (2001)
DenitionThe null hypothesis of CC can be expressed as
H0,CC : E [M (di ; α)] = 0,
where M (di ; α) denotes a (p, 1) vector whose components are theorthonormal polynomials Mj (di ; α) , for j = 1, .., p. Under someregularity conditions:
JCC (p) =
1pN
N
∑i=1M (di ; α)
!| 1pN
N
∑i=1M (di ; α)
!d!
N!∞χ2 (p)
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Frequency-based testsMagnitude-based testsMultivariate testsIndependence testsDuration-based tests
Testing strategies: duration-based tests (5/5)
Candelon et al. (2001)
DenitionUnder UC, the mean of durations between two violations is equalto 1/α, and the null hypothesis is
H0,UC : E [M1 (di ; α)] = 0.
with a test statistic equal to
JUC =
1pN
N
∑i=1M1 (di ; α)
!2d!
N!∞χ2(1)
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Test, test and testCheck the P&L dataThe power of your tests may be low...Estimation risk
Backtesting
Recommandations
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Test, test and testCheck the P&L dataThe power of your tests may be low...Estimation risk
Backtesting
Recommandation 1: Test, test and test
Recommandation 2: Check the P&L data
Recommandation 3: The power of your tests may be low..
Recommandation 4: Take into account the estimation risk
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Test, test and testCheck the P&L dataThe power of your tests may be low...Estimation risk
Backtesting
Recommandation 1: Test, test and test
Each type of test (frequency, severity, independence,conditional coverage, multivariate test etc..) captures onetype of potential misspecication of the VaR model.
It is important to use a variety of tests
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Test, test and testCheck the P&L dataThe power of your tests may be low...Estimation risk
Backtesting
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Test, test and testCheck the P&L dataThe power of your tests may be low...Estimation risk
Backtesting
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Test, test and testCheck the P&L dataThe power of your tests may be low...Estimation risk
Backtesting
Recommandation 2: Check the P&L data
Frésard, L., C. Perignon, and W., Anders (2011), ThePernicious E¤ects of Contaminated Data in Risk Management,Journal of Banking and Finance.
1 A large fraction of US and international banks validate theirmarket risk model using P&L data that include fees andcommissions and intraday trading revenues.
2 Distinction between dirty P/L and hypothetical P/L (JP.Morgan, Romain Berry 2011).
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Test, test and testCheck the P&L dataThe power of your tests may be low...Estimation risk
Backtesting
Recommandation 3: The power of your tests may be low..
DenitionThe power of a backtesting test corresponds to its capacity todetect misspecied VaR model.
Pr [Rejection H0 j H1]
Example
Berkowitz, J., Christo¤ersen, P. F., and Pelletier, D., 2013,Evaluating Value-at-Risk Models with Desk-Level Data.Management Science.
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Test, test and testCheck the P&L dataThe power of your tests may be low...Estimation risk
Backtesting
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Test, test and testCheck the P&L dataThe power of your tests may be low...Estimation risk
Backtesting
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Test, test and testCheck the P&L dataThe power of your tests may be low...Estimation risk
Backtesting
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Test, test and testCheck the P&L dataThe power of your tests may be low...Estimation risk
Backtesting
Hurlin C. et Tokpavi S. (2008), Une Evaluation desProcédures de Backtesting : Tout va pour le Mieux dans leMeilleur des Mondes", Finance
Idea: we use 6 di¤erent methods (GARCH, RiskMetrics, HS,CaviaR, Hybride, Delta Normale) to forecast a VaR(5%) on thesame asset (GM, Nasdaq), and we apply the backtests (LR, DQ,Duration based tests) on a set of 500 samples (rolling window) ofT = 250 forecasts.
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Test, test and testCheck the P&L dataThe power of your tests may be low...Estimation risk
Backtesting
Example
LRCC tests: for 47% of the samples, we dont reject (at 5%) thenull for any of the six VaR forecats. In 71% of the samples, wereject at the most one VaR.
Example
DQCC tests: for 20% of the samples, we dont reject (at 5%) thenull for any of the six VaR forecats. In 51% of the samples, wereject at the most one VaR.
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Test, test and testCheck the P&L dataThe power of your tests may be low...Estimation risk
Backtesting
The power of a consistent test tends to 1 when the samplesize tends to ininity.
Recommandation: increase at the maximum the sample sizeof your backtest.. (T = 500, 750 or more.)
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Test, test and testCheck the P&L dataThe power of your tests may be low...Estimation risk
Backtesting
Recommandation 4: Take into account the estimation risk
The risk dynamic is usually represented by a parametric orsemi-parametric model, which has to be estimated in apreliminary step. However, the estimated counterparts of riskmeasures are subject to estimation uncertainty.
Replacing, in the theoretical formulas, the true parametervalue by an estimator induces a bias in the coverageprobabilities.
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Test, test and testCheck the P&L dataThe power of your tests may be low...Estimation risk
Backtesting
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Test, test and testCheck the P&L dataThe power of your tests may be low...Estimation risk
Backtesting
Escanciano and Olmo (2010, 2011) studied the e¤ects ofestimation risk on backtesting procedures. They showed howto correct the critical values in standard tests used to assessVaR models.
Escanciano, J.C. and J. Olmo (2010) Backtesting ParametricValue-at-Risk with Estimation Risk, Journal of Business andEconomics Statistics.
Escanciano, J.C. and J. Olmo (2011) Robust Backtesting Testsfor Value-at-Risk Models. Journal of Financial Econometrics.
Christophe Hurlin Backtesting
IntroductionBacktesting Principles
Testing strategiesRecommandations
Test, test and testCheck the P&L dataThe power of your tests may be low...Estimation risk
Backtesting
Estimation Adjusted VaR
Gouriéroux and Zakoian (2013) a method to directly adjust theVaR to estimation risk ensuring the right conditional coverageprobability at order 1/T :
Prrt < EVaR t jt1 (α)
= α+ oP (1/T )
Gouriéroux C. and Zakoian J.M. (2013), Estimation AdjustedVaR, forthcoming in Econometric Theory.
Christophe Hurlin Backtesting