Upload
jtnylson
View
220
Download
0
Embed Size (px)
Citation preview
7/28/2019 Celso Mug Nela
1/20
Validation in Risk Management:Approaches, Issues, and Observations
Kenneth G. Fulton
Division of Banking Supervision and Regulation
Board of Governors of the Federal Reserve System
**************************Febraban 2nd Annual Operational RiskConference
June 5, 2009
7/28/2019 Celso Mug Nela
2/20
Agenda
z Validation more than models
z U.S. Final Rule
requirementsz Framework validation
z Data validation
z Model Validation
z Supervisory review
z Pillar 2
2
I think you should be more
explicit here in step number 2
This presentation contains the views and
opinions of the speaker, and does not
represent a policy statement of the FederalReserve Board of Governors.
7/28/2019 Celso Mug Nela
3/20
Validation
z Validation in this context is a set of processesor activities designed to assess whether riskmanagement tools are working as intended
Applies to quantitative and qualitative tools (not justmodels)
z Validation is a control mechanism that helps
to assess uncertainty in risk management Tools for assessing risk always contain inherent
uncertainty
Validation helps minimize uncertainty (but cannotremove it)
3
7/28/2019 Celso Mug Nela
4/20
Characteristics of Validation
Validation:
z Should provide an effective challenge to risk
management toolsz Does not rely on a single method
z Is a process, not an event
z Should cover all aspects of risk management
(including underlying components)
z Requires judgment and is a combination ofquantitative and qualitative assessment
4
7/28/2019 Celso Mug Nela
5/20
Governance Framework
5
7/28/2019 Celso Mug Nela
6/20
Components of Validation
z Validation activities generally fall into threecategories:
Review of the development process and conceptual foundation
Ongoing monitoring of process, which includes verifyingsoundness of approach and comparing to benchmarks
Analysis of outcomes (such as backtesting)
z
All three types of validation activities should beapplied to all elements of the risk measurement
process
Applied to the process that translates inputs into outputs, as wellas to the inputs and outputs themselves
z Key exercises for analyzing and checking outcomes
include sensitivity analysis and stress testing
6
7/28/2019 Celso Mug Nela
7/20
Keys to Successful Validation
z Senior management is responsible for ensuring thatvalidation is working properly Through internal audit or other means
Should set clear expectations for acting on validation results
z Validation tends to be more effective if consideredbefore tools are developed not after the fact
z Good practice to have a firm-wide validation policy
z Many institutions have various levels of validation(conceptual / partial / full) Can be appropriate when full validation is not possible
But lack of full validation should always be transparent
If a tool has not been fully validated, there should be greateruncertainty about the estimates it produces
7
7/28/2019 Celso Mug Nela
8/20
Keys to successful validation (cont.)
z Objectivity in validation is highly important Does not necessarily have to mean structural independence
But less structural independence may call for more controls toalleviate potential conflicts of interest
On its own, independent structure does not translate to effectivevalidation
z Documentation is vital to good validation Without it, evidence of objective validation is lacking
z Effective validation requires competent staff with theright skills, the right resources, the right incentives,and appropriate stature within the organization
Need ability to effectively challenge developers Need to identify issues and have a process for resolution
8
7/28/2019 Celso Mug Nela
9/20
Model Validation Techniques
z Benchmarking
Benchmarking involves assessing the consistency ofthe estimated parameters with those obtained by other
estimation techniques, and potentially using other datasources.
z Backtesting
Backtesting generally involves comparing realized (expost) values with estimated (ex ante) parameters for acomparable and homogeneous data set.
z Judgmental Comparison
Comparison with other judgments and models
Comparison with other external information9
7/28/2019 Celso Mug Nela
10/20
Validating Vendor Models
z Vendor models should be subject to the samevalidation standards as internal models
Validation includes not just the vendor model itself
9but also application of the vendor model to theindividual bank, its exposures, and risk profile
Documentation standards also apply Simple market acceptance does not suffice
Two key questions regarding vendor models:
9Does the model do what it is designed to do?9Does the model do what the bank is employing it to
do?
10
7/28/2019 Celso Mug Nela
11/20
U.S. Final Rule Requirements
The [bank] must validate, on an ongoing basis, its advancedsystems. The [bank]s validation process must beindependent of the advanced systems development,
implementation, and operation, or the validation processmust be subjected to an independent review of itsadequacy and effectiveness. Validation must include:
i. An evaluation of the conceptual soundness of (includingdevelopmental evidence supporting) the advancedsystems;
ii. An ongoing monitoring process that includes verification ofprocesses and benchmarking; and
iii.An outcomes analysis process that includes back-testing.
11
7/28/2019 Celso Mug Nela
12/20
Framework Validation
Examples:
z A broad assessment of the conceptual soundness of theAMA framework (e.g., Do the tools, processes, and
oversight systems the institution has designed andimplemented to measure and manage operational risk fitwith the quantitative techniques designed to quantify
operational risk?).z Are all the risk management tools subject to ongoing
monitoring and evaluation of their effectiveness? Does
the institution have robust mechanisms to periodically testwhether a particular tool or process remains longereffective?
12
7/28/2019 Celso Mug Nela
13/20
Data Validation
Examples:
z Reconcile to source systems
z Review by a risk control functionz Review by internal audit
Does internal audits participation as an internal control
compromise its independence?
z Review by business line peers
z
Comparison with other data elementsz Comparison with experience or expertise
Backtesting of BEICFs to internal loss data
z Assumptions are also part of the data13
7/28/2019 Celso Mug Nela
14/20
Framework for Supervisory Review
14
7/28/2019 Celso Mug Nela
15/20
Pillar 2
15
Pillar 2:
Supervisory
Review and
Internal Capital
Adequacy
Assessment
Process
(ICAAP)
Pillar 3:
Market Discipline
Pillar 1:
Minimum Capital
Requirements
7/28/2019 Celso Mug Nela
16/20
Pillar 2 Supervisory Review & ICAAP
Supervisory guidance issued 7-15-08
z Identify and measure material risk
Including, but not limited to: credit, market, operational,interest rate, and liquidity risks
z Set and assess capital adequacy goals in
relation to riskz Ensure the integrity of the ICAAP
ICAAP is not just an economic capital calculation
16
7/28/2019 Celso Mug Nela
17/20
Implications for Operational Risk
z Overall process for operational risk may bebased heavily on AMA
Pillar 2 approach may use a different confidencelevel (99.9th percentile for Pillar 1) or insuranceoffset (limited to 20% of operational risk capital for
Pillar 1)z The AMA is not as rules-based as the
AIRB
No prescribed formulas
Institutions design framework that best fits theirorganization
17
7/28/2019 Celso Mug Nela
18/20
Pillar 2 Validation Required
z Each bank should ensure that the componentsof its ICAAP, including any models and their
inputs, are subject to the banks validation
policies and procedures.
z
A bank should be cognizant that bothquantitative and qualitative approaches have
their own inherent biases and assumptions
that affect risk assessment. Accordingly, abank should recognize the biases and
assumptions embedded in, and the limitations
of, the approaches used. 18
7/28/2019 Celso Mug Nela
19/20
Supervisory References
z Risk Model Validation ; OCC Bulletin 2000-16http://www.occ.treas.gov/OCC_current.htm
z Model Governance ; FDIC Supervisory Insights, Winter 2005
http://www.fdic.gov/regulations/examinations/supervisory/insight
s/siwin05/index.html
z Guidelines on the implementation, validation and assessment of
Advanced Measurement (AMA) and Internal Ratings Based (IRB)
Approaches, Committee of European Bank Supervisors, April2006 http://www.c-ebs.org/getdoc/5b3ff026-4232-4644-b593-
d652fa6ed1ec/GL10.aspx
z Supervisory Guidance: Supervisory Review Process of Capital
Adequacy (Pillar 2) Related to the Implementation of the Basel IIAdvanced Capital Framework, July 2008
http://www.federalreserve.gov/newsevents/press/bcreg/bcreg200
80715a1.pdf
19
http://www.occ.treas.gov/OCC_current.htmhttp://www.fdic.gov/regulations/examinations/supervisory/insights/siwin05/index.htmlhttp://www.fdic.gov/regulations/examinations/supervisory/insights/siwin05/index.htmlhttp://www.c-ebs.org/getdoc/5b3ff026-4232-4644-b593-d652fa6ed1ec/GL10.aspxhttp://www.c-ebs.org/getdoc/5b3ff026-4232-4644-b593-d652fa6ed1ec/GL10.aspxhttp://www.c-ebs.org/getdoc/5b3ff026-4232-4644-b593-d652fa6ed1ec/GL10.aspxhttp://www.c-ebs.org/getdoc/5b3ff026-4232-4644-b593-d652fa6ed1ec/GL10.aspxhttp://www.c-ebs.org/getdoc/5b3ff026-4232-4644-b593-d652fa6ed1ec/GL10.aspxhttp://www.c-ebs.org/getdoc/5b3ff026-4232-4644-b593-d652fa6ed1ec/GL10.aspxhttp://www.c-ebs.org/getdoc/5b3ff026-4232-4644-b593-d652fa6ed1ec/GL10.aspxhttp://www.c-ebs.org/getdoc/5b3ff026-4232-4644-b593-d652fa6ed1ec/GL10.aspxhttp://www.fdic.gov/regulations/examinations/supervisory/insights/siwin05/index.htmlhttp://www.fdic.gov/regulations/examinations/supervisory/insights/siwin05/index.htmlhttp://www.occ.treas.gov/OCC_current.htm7/28/2019 Celso Mug Nela
20/20
Kenneth Fulton