Upload
willis-gallagher
View
215
Download
0
Tags:
Embed Size (px)
Citation preview
Quantifying Model Specification Riskin PracticeActuarial Teachers’ and Researchers’ Conference
Edinburgh, 2nd December 2014
Alan Forrest
RBS Group Risk Analytics Independent Model Validation
Information Classification –
2
Disclaimer
Disclaimer
The opinions expressed in this document are solely the author’s and do not necessarily reflect those of The Royal Bank of Scotland Group or any of its subsidiaries.
Any graphs or tables shown are based on mock data and are for illustrative purposes only.
3
Overview
Background
Background to Model Risk – the risk of using a model.
Model Risk Quantification
Model specification risk as a Data-shift Problem.
Quick estimates of specification risk using geometric and information-theoretic approaches to the data-shift problem.
4
Model Risk Background
The US Regulator (Fed / OCC 2011-12a )
“The use of models invariably presents model risk, which is the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports.”
– Using a model presents a risk.
FSA - Turner Review - March 2009
“Misplaced reliance on sophisticated maths”
– The assumptions and limitations of the models were not communicated adequately to the pricing and lending decision-makers.
BoE - The Dog and the Frisbee – Haldane, August 2012
“… opacity and complexity… It is close to impossible to tell whether results from [internal risk models] are prudent.”
– If we cannot say why we trust a model, are we right to use it?
5
Fed / OCC 2011-12a
“Model Risk should be managed like other types of risk.”
– Identify– Quantify / assess– Act / manage– Monitor
Focus on specification risk:
The part of model risk connected with model selection.
– Model risk also includes risks of model implementation, use and interpretation.
Quantification of specification risk:
How differently could the model have been built under different conditions?
Model Risk Background
6
Example Model Risk
A Probability of Default model is proposed for implementation
The model includes a factor W that has 20% missing values.
The missing values have been filled in, all with the same “mean” value, and the preferred model has been built with this imputed factor.
Missing values tend to be associated with older accounts.
7
Example Model Risk
Hypothetical alternative model builds – based on different development data:
With “missing” as a special class.
With a different method of imputation
Explore the possible bias over time by forcing (at random) new missing values among more recent accounts.
8
Quantifying Model Risk – Bottom Up
The key to specification risk quantification is sensitivity analysis;
and most sensitivity analysis can be expressed as a Data-shift Problem.
If the data used to build a model shifts, how far and in what way does the model shift?
In practice, we could have 100 data-shifts to test, and models require much resource to build.
Modellers need a quick and reliable way of assessing the likely impact of data shifts without needing to build models:
– Prioritising analysis;– Getting immediate assurance on shifts that are immaterial;– Making the appropriate model changes.
9
Original data
Model space
Shifted data
Type 3 (or type 0) Error
Over-fitting Over-sensitive / discontinuous
Just right
Geometry and Model Sensitivity
The sensitivity of a model to data-shift has a geometric interpretation.
Model fitting looks “geometric”: the model is found “closest” to the data.
Geometry and curvature expresses data-shift sensitivity.
10
Simplifications and Conventions
Banking Book Credit Risk Models allow the following assumptions:
All factors and outcomes are categorical / classed / discrete
The development data-set is completely described by its frequency table: i.e. whole number entries in a finite contingency table.
The space of all data is finite dimensional.
Fixed-effects regression based on an exponential family of distributions
Includes all the classical regressions: logistic, multinomial, poisson etc.
Maximum Likelihood Estimation (MLE)
11
Simplifications and Conventions
A model is a description of the data on which it is developed:
a model is another point in data space.
The preferred model is chosen from a limited set of possible descriptions – the model space.
This model space is a subspace of data space, chosen for its convenience, simplicity or usefulness.
What about inputs and outputs?
Some dimensions of the data-space are inputs, others are outputs.
If the model space covers marginally all input populations, then* the development data and the MLE optimised description have the same input population distributions: this allows a model outcome to be defined for any input population.
* non-trivial theorem
13
Information Principles
Dual optimisation principles for exponential families
The red model spaces are generated multiplicatively / ‘tropically’.
Data points are drawn to the MLE model along the blue spaces, which are linear.
Principle of Maximum Likelihood– The model that maximises likelihood, for
data d, is the point, m, in red space that minimises KL divergence I(d,m).
Principle of Maximum Entropy– If m’ is any model in the red space then,
within each blue space, the red point m minimises KL divergence, I(m,m’) .
Principles of Inference– if m is the MLE fit to data d, and m’ is
another model from the model space, then I(d,m’) = I(d,m) + I(m,m’) .
14
Geometry and Model Sensitivity
Geometry of the data space
The natural metric is ds2 = w dxw2 / xw
– Locally equal to Kullback-Leibler divergence and to Hellinger distance.
– “Local Chi-squared”.– “Boot-strapping geometry”.
Isometric to a portion of a sphere.– 2x = u2 connects this space isometrically with
Euclidean space.
The model fitting foliations are orthogonal in this metric.– The model space curvature reflects true model
sensitivity.
15
Efficient Sensitivity Analysis
A sensitivity principle implied by boot-strapping
For large development samples, the standard error ellipsoid is sufficient to describe model sensitivity to data shifts.
Model Prediction Error Ellipsoid = ds-ball
Model Standard Error Ellipsoid = image of data
ds-ball
Data Bootstrapping Ellipsoid = ds-ball
Scale = Chi-sq (df = dimension) / (2N)
16
Geometry and Model Sensitivity
0.2536
0.0054 +
0.0030
0.0443
Example: Sensitivities for factor with 20% missing values– Distances (squared) between hypothetical alternative datasets, computed in spherical
metric from marginals illustrated.– Additional distance estimated by KL information value relative to marginals.
17
Managing Sensitivities
Model Risk - 20% missing values - example revisited :
PD has been built from a pool of 12 classed factors:– Dimension of the data space (roughly = number of cross-tab cells), D = 50,000.
PD model built by MLE on sample of N = 500,000 records.
Bootstrap scale is D/2N = 0.05 .
Sensitivity TestData Shift
distance-squaredObservation Action
Force missing values among new
accounts0.008
Correlation of missingness with age of accounts is unlikely to cause significant
change in model build.No need to investigate.
Missing at Randomv. Mean Allocation 0.045
These two imputation options are likely to result in same factor selections, but
different factor weights.Low priority
Missing as a separate category 0.254
Model build is likely to be materially different if missing is treated as a
separate category. High priority
18
Conclusions
Quantifying Model Risk
Quantitative model risk assessment is needed for– consistent management and maintenance of a bank’s models, and– effective communication of model weaknesses and limitations to decision-makers and
users.
The key to bottom up quantitative model risk assessment is sensitivity analysis.
The key to practical sensitivity analysis is data-shift.
Data-shift is a deep problem, well-known in geometric statistics, information theory and artificial intelligence, and rich in mathematical interest.
Simplified approximate solutions to the data-shift problem can be used in practice to quantify and prioritise model risk assessment in banking book credit risk.
19
An incomplete bibliography
The geometric approach to statistics is rich and well-established, classically starting with Rao in the 1940s
Its connection with information and KL divergence is also developed in great generality
Centsov (1965, et seq.), making geometric the original developments of Kullback and Leibler.
Efron (1978 et seq.), Lauritzen (1980s), Critchley et al. (1993 et seq.), etc.
Amari et al. (1982 et seq.) develops Akiake’s insights using differential geometry
The application to the data shift problem is comparatively recent but growing
Recent developments in Machine Learning, by Kanamori, Shimodaira (2009) and others, are particularly relevant to sensitivity analysis.
Hulse et al. (2013) recently explored its implications in Financial Modelling.