MODEL RISK MANAGEMENT: BEST PRACTICES FOR MATLAB MODELS
Information, data and drawings embodied in this presentation are strictly confidential and are supplied on the understanding that they will be held confidentially and not disclosed to third parties without the prior written consent of QuantUniversity LLC.
Sri Krishnamurthy, CFAFounder and CEOwww.QuantUniversity.comAugust 14th 2016
AG
END
A
About QuantUniversity
Stress Testing and Model Risk: A Brief Introduction
The Model Risk Analytics Solution
1
2
3
4
Agenda
Case Study5
Challenges to implementing Model Risk Management
ANALYTICS, RISK MANAGEMENT & STRESS TESTING FOR FINANCIAL INSTIT UTIONS- ADVISORY SERVICES- PLATFORM TO STRESS TESTING, ANALYTICS AND RISK ASSESSMENT- ARCHITECTURE REVIEW, TRAINING AND AUDITS
INTR
OD
UC
TION
• Founder of QuantUniversity. • Advisory and Consultancy for Financial Analytics
and Model Risk• Prior Experience at MathWorks, Citigroup and
Endeca and 25+ financial services and energy customers.
• Charted Financial Analyst and Certified Analytics Professional
• Teaches Analytics in the Babson College MBA program and at Northeastern University, Boston
Sri KrishnamurthyFounder and CEO
4
STRESS TESTING AND ANALYTICS– A BRIEF INTRODUCTION
STRESS TESTS IN
THE N
EWS
“Of the 33 banks the Fed tested—which included the largest U.S. banks, like Bank of America, Citigroup, and Wells Fargo.. all were deemed strong enough to weather a severe economic meltdown without any help from the government.”
MO
DEL
VA
LIDA
TION
“Model risk is the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports. “ [1]
“Model validation is the set of processes and activities intended to verify that models are performing as expected, in line with their design objectives and business uses. ” [1]
Ref:[1] . Supervisory Letter SR 11-7 on guidance on Model Risk
Model Risk and Validation
STR
ESST
ESTING
AN
DM
OD
ELR
ISKM
AN
AG
EMEN
T
Regulatory efforts
EIOPA: Europe-wide stress test for the insurance sector
What’s driving Stress Testing and Model Risk Management efforts?
https://eiopa.europa.eu/Publications/Surveys/eiopa-14-215_stress_test_2014_specifications.pdf
STR
ESST
EST&
SC
ENA
RIO
TEST
Stress Tests and Scenario Tests
Figure courtesy: http://www.actuaries.org/CTTEES_SOLV/Documents/StressTestingPaper.pdf
DEFIN
ITION
SDefinitions
1. Scenarios :
“A scenario is a possible future environment, either at a point in time or over a period of time.”
“Considers the impact of a combination of events“
2. Sensitivity Analysis:
“A sensitivity is the effect of a set of alternative assumptions regarding a future environment. “
3. Stress Testing:
• Analysis of the impact of single extreme events (or risk factors)
• Extreme but plausible
Ref: http://www.actuaries.org/CTTEES_SOLV/Documents/StressTestingPaper.pdf
STR
ESST
ESTING
AN
DM
OD
ELR
ISKM
AN
AG
EMEN
T
Model verification:
• Migration of code from an older code-base/ from another language
• Regression Testing
Model Deployment:
• Model tests to be done prior to deployment
Model Replication:
• Models that replicate alternate models or functionality/strategy
Model Back testing:
• Test how a model performs under different conditions with historical data
What’s involved with Model Risk Management efforts?
MO
DEL
LIFECYC
LEM
AN
AG
EMEN
TK
EYTO
MR
MHow’s Model Risk Management implemented in the company?
Ref: The Decaloguehttp://quantuniversity.com/w9.html
CHALLENGES TO IMPLEMENTING MODEL RISK MANAGEMENT
CH
AR
AC
TERISTIC
SO
FS
TRESS
AN
DS
CEN
AR
IOTESTIN
GCharacteristics of Stress and Scenario testing
1. Difficult to build parametric models – Simulation driven approach necessary
2. Parameter space can explode easily
3. Large number (10s of thousands) of tests needed
4. Human intervention required
5. Company needs are different : Customization required
6. Significant coordination and engagement with multiple groups needed
Ref: http://www.actuaries.org/CTTEES_SOLV/Documents/StressTestingPaper.pdf
IMP
LEMEN
TING
SYSTEM
ATIC
STRESS
TESTING
ISC
HA
LLENG
ING
Challenges to systematic stress testing
1. Model Implementation
• Does it actually work for all intended use cases?
2. Model parameter testing
• How many parameters ?
• How many scenarios ?
3. Model Applicability
4. Model Benchmarking against Reference Implementation
• R vs Proprietary vs MATLAB
5. Model Migration (version)
• Regression Testing v1.0 to v2.0
6. Model use case validation
• Can we use the results to make decisions?
MO
DEL
LIFECYC
LEM
AN
AG
EMEN
TIS
STILLEV
OLV
ING
Challenges in Model Life Cycle Management
1. Model risk management considered as a regulatory/compliance task
2. No one-size-fits-all solution
3. Lack of transparency, standardization and centralized model management
4. Disparate and disconnected systems (SVN, Sharepoint, Excel sheets, custom systems)
5. Difficult to track and audit changes
6. Metrics to quantify model risk not established
MODELRISK ANALYTICS SOLUTION
LEV
ERA
GE
TECH
NO
LOG
YTO
SCA
LEA
NA
LYTICS
Leverage technology to scale analytics
1.64 bit systems : Addressable space ~ 8TB
2.Multi-core processors : Explicit and Implicit Multi-threading
3.Parallel and Distributed Computing : Leverage commodity/specialized hardware to scale problems
4.Micro services-based solution for fast deployment and replication
5.Cloud Computing
Ref: Gaining the Technology Edge: http://www.quantuniversity.com/w5.html
AD
DR
ESSING
STRESS
TESTIN
GC
HA
LLENG
ESHow can we address challenges in Stress testing?
1. Leverage hardware for scaling
1. Enable developers to plan for stress tests
2. Enable developers to scale without having to rewrite code
2. Enable developers to run their code
1. Locally
2. On Enterprise Servers
3. Cloud
AD
DR
ESSING
STRESS
TESTIN
GC
HA
LLENG
ESHow can we address challenges in Stress testing?
3. Facilitate exhaustive and robust testing without having to get lost in details
1. Enable testers to specify their tests and configurations
2. Enable benchmarking and comparisons with other models
4. Facilitate deployment of models built using different frameworks including legacy code
1. Provide a choice of models (which could be built in different languages)
2. Enable benchmarking and comparisons with models built using other languages
AD
DR
ESSING
MO
DEL
LIFEC
YCLE
MA
NA
GEM
ENT
CH
ALLEN
GES
How can we address challenges of Model Lifecycle Management?
1. Provide a single comprehensive environment to manage model validation tests
1. Enable integration with source control systems
2. Enable model validators to create, run and assess tests
2. Facilitate all stages of model lifecycle management in a single environment
1. Allow access to all model related artifacts (documentation, tests and results)
2. Provide ability to collaborate with all stakeholders responsible for model risk
3. Enable audit trails and analytics to manage model risk
MO
DELR
ISKAN
ALYTIC
SS
YSTEMModelRisk Analytics System
MO
DEL
RISK
AN
ALYTIC
SS
OLU
TION
Model Risk Analytics Solution
1. Distributed System to support Model Verification, Validation, Stress and Back Testing
2. Supports MATLAB and other open source packages like R, Python, SVN
3. Architecture leverages state-of-the-art asynchronous computing modes
4. Package management
1. Using Custom Client/ Images
2. Using Docker(Different containers for different apps)
5. Supports synchronous and asynchronous modes
6. Built for the cloud but can work on any cloud/internal infrastructure (private or public)
CASE STUDY
Value-at-Risk & Conditional Value-at-Risk
• VaR : The predicted maximum loss of a portfolio with a specified probability level (e.g., 95%) over a certain period of time (e.g. one day)
• CVaR (Expected Shortfall) : The expected value of the loss given that the loss exceeds VaR
Ref: Optimization Methods in Finance by Gerard Cornuejols, Reha Tutuncu, Cambridge University Press Image courtesy: http://www.imes.boj.or.jp/english/publication/mes/2002/me20-1-3.pdf )
How to Implement VaR and CVaR?
Methodology:
• Historical
• Variance-Covariance method
• Monte-Carlo simulations
Models are implemented in:
• MATLAB – Production Model
• Python – Alternate model developed by a 3rd-party vendor
• R – Reference/Benchmark model from a paper
Methods to compute VaR and CVaR• Historical method
Image Courtesy: http://www.investopedia.com/articles/04/092904.asp
1. Compute Daily Returns and sort them in ascending order
2. For a given confidence level (e.g. 95%) , find VaR α (X) such that:
P(X<= VaR α(X)) = α
3. Compute CVaR by taking the average loss of the tail
Methods to compute VaR and CVaR
• Variance-Covariance Method
Image Courtesy: http://www.investopedia.com/articles/04/092904.asp
1. Compute Daily Returns and fit a Normal distribution to obtain mean and Standard Deviation (µ & σ)
2. For a given confidence level (e.g. 95%) , find VaR α (X) such that:
P(X<= VaR α(X)) = αExample 95% => -1.65* σ
3. Compute CVaR by taking the average loss of the tail
(See Yamai and Yoshibahttp://www.imes.boj.or.jp/english/publication/mes/2002/me20-1-3.pdf )
Methods to compute VaR and CVaR
• Monte-Carlo Simulations
Image Courtesy: http://www.investopedia.com/articles/04/092904.asp
1. Compute Daily Returns and fit a Normal distribution to obtain mean and Standard Deviation (µ & σ)
2. Run n Monte-Carlo simulations with random numbers drawn from a normal distribution described by (µ & σ)
3. For a given confidence level (e.g. 95%) , find VaR α (X) such that:
P(X<= VaR α(X)) = α
4. Compute CVaR by taking the average loss of the tail
Model Summary
VaR Model
Historical Method
Python
R
MATLAB
Variance-Covariance
Method
Python
R
MATLAB
Monte-Carlo Simulations
Python
R
MATLAB
Given:1. Historical Daily price time series for a specified time period2. Constituents of a 3-asset long-only portfolio
Compute:VaR and CVaR
Model Verification criteria
1. Model Benchmarking• MATLAB production model vs Alternate Python model vs
Reference R model
2. Parameter sweeps• Different Confidence Intervals (90%, 95%, 99%)
3. Model Convergence• How many simulations needed ? (100, 500, 1000)
4. How do different methods compare?• Historical vs Variance-Covariance vs Monte-Carlo methods
VarModel.m
ModelRisk Studio facilitates easy structuring of tests
Input parameters can be specified for each model
ModelRisk Analytics allows you to run the model in multiple environments
The Model Testing results can be viewed in an interactive dashboard
Model Issues can be tracked, risks evaluated and addressed appropriately
FEA
TUR
ESComing soon!
ModelRisk Model Definition Language• To specify model parameters
ModelRisk Engine Optimization to leverage infrastructure in the cloud
• Resource constraints: • Budget & Time constraints
• Priority queues and jobs• Dynamic scaling and load balancing
ModelRisk Audit to enable periodic testing and ongoing monitoring of your models
MO
DEL
RISK
AN
ALYTIC
SB
ETAP
RO
GR
AM
If you are interested in being a part of the Beta program:
If you are interested in knowing more about the Model Risk Analytics systemOr Want to be part of the Beta program,
email: [email protected] to be added to the interest list
Model Risk Analytics Beta program
MO
DEL
RISK
MA
NA
GEM
ENT
WO
RK
SHO
PAlso available:
A 2-day workshop in Model Risk Management with hands-on case studies
Email: [email protected] to be added to the interest list
Stress Testing and Model Risk
Management for Quants
AN
OM
ALY
DETEC
TION
WO
RK
SHO
PComing soon:
A 2-day course in methods and best practices in Anomaly DetectionSeptember 19th, 20th , New York
Visit www.analyticscertificate.com/Anomaly for more details
Anomaly DetectionCore Techniques and
Best Practices
WH
ITEPA
PER
S
To get copies of these articles, go to:http://www.quantuniversity.com/articlerequest.html
OREmail: [email protected]
Thank you!
Sri Krishnamurthy, CFA, CAPFounder and CEO
QuantUniversity LLC.
srikrishnamurthy
www.QuantUniversity.com
Contact
Information, data and drawings embodied in this presentation are strictly a property of QuantUniversity LLC. and shall not bedistributed or used in any other publication without the prior written consent of QuantUniversity LLC.