CRM Scorecard Re Calibration Methodology BGH

Bank of Good Hope

Credit Risk Management

Scorecard Recalibration

Technical Proposal

1

PIC Solutions Presentation Outline

The BGH Business Problem

The BGH Business Goals and Objectives

PIC Solutions: Technical Proposal

Case Study: CRM Scorecard Recalibration

Objectives of the PIC Technical Proposal

Benefits of the PIC Technical Solution

Implementation of the PIC Technical Solution

Discussion

22

The BGH Business Problem Steady Decline of the Credit Card Portfolio

Competition from other Financial Institutions

Outdated Credit Risk Management Scorecards

Deterioration of the CRM Scorecard Performance

Limited Analytical Resources for Model Development

Unacceptably Long Model Development Times

Lack of Confidence in the Decision Analytics Function

Lack of Synergy between Sales and Risk

33

The Required Technical Solution

Updated and Well - Performing CRM Scorecards

Staff Training: SAS and Model Development

Simple Robust and Effective Solution: Easy to Understand Easy to Implement Easy to Maintain

Better Analytical Support for: Credit Risk Management ( Acquisitions to Collections ) Risk Appetite Analysis ( Business Development )

Classification of Customers for Risk-Based Pricing

More Effective and Efficient Credit Risk Management44

PIC Solutions Presentation Outline The BGH Business Problem


PIC Solutions: Technical Proposal





Discussion

55

The BGH Business Goals & Objectives

BGH wish to grow the Credit Card Portfolio by using Effective Decision Analytics

They want to implement their Risk Appetite Strategy and Risk-Based Pricing to grow the business and to transform the Credit Card Portfolio.

Existing Staff need to be trained in SAS and Model Development Techniques

The Project Management Office must be established.

BGH are looking for a cost – effective solution and rapid implementation.

They want to see measurable results within 6 months.66



PIC Solutions Technical Proposal





Discussion77

The PIC Solutions Technical Proposal

1. Comprehensive Scorecard / Model Evaluation: Data Quality Analysis Model Development Methodology Model Validation Methodology Model / Scorecard Implementation Scorecard Monitoring Reports

2. Credit Card Portfolio Analysis

3. CRM Scorecard Recalibration

4. Updated CRM Scorecard Implementation

5. Risk Appetite Analysis

6. Risk-Based Pricing Strategy

8


What is Scorecard Recalibration?

Why should we Recalibrate CRM Scorecards? How should we Recalibrate CRM Scorecards?

When should we Recalibrate CRM Scorecards?

How Often should we Recalibrate CRM Scorecards?

Does Scorecard Recalibration New Scorecard Development ?

Will CRM Scorecard Recalibration solve our Analytical Problems?

How Much does Scorecard Recalibration Cost?

99


Scorecard Recalibration

High Level Overview: The Objectives of Scorecard Recalibration

The Benefits of Scorecard Recalibration

The Scorecard Recalibration Process

Detailed Description: The Scorecard Recalibration Methodology

Analytical Support for Scorecard Recalibration

The Scorecard Recalibration Toolkit

1010

Scorecard Recalibration Objectives

Identify those Scorecards where there has been significant

deterioration in the performance of the Scorecard

Explore the potential value in recalibrating the existing Scorecards

using the most recently available population data without changing

the Structure of the Scorecard and its Component Variables.

If necessary, Recalibrate the Scorecard Model as required in an

incremental and phased approach.

Facilitate the rapid development of CRM Strategy based on the

Recalibrated Scorecard

1111

Scorecard Recalibration Objectives

Overcome the Analytical Resource Constraint.

Evaluation of the CRM Scorecard on Current Data

Updating the Model Parameters

Incremental Modification of the Model as required

Improvement of the Scorecard / Model Performance

Reduction in NEW Model Development Costs

Rapid Scorecard Development

Rapid Scorecard Implementation

Robust, Effective and Simple Solution1212

Scorecard Recalibration Benefits

Model Risk Reduction

Credit Risk Management and Business Support

Reduction in Model Development Costs

Reduction in Model Development Project Time

Resource Development

Tactical Flexibility

Incremental Change Management

Strategy Development

Facilitates Portfolio Transformation1313

So what is the Real Problem ? Most Credit Risk Scorecards are based on a Statistical Model.

The Statistical Model is based on several assumptions.

The Statistical Model extracts information from the Data.

The quality of the data will affect the power of the model.

The quality of the model development process is critical.

The performance of the model is measured by several statistics.

The application of the scorecard affects the statistical measures.

The business and risk assumptions may no longer be valid.

The intended target population is always changing.

The Scorecard is Out of Date BEFORE it is Implemented.

1414

CRM Scorecard Models

Most Credit Risk Scorecards are based on a Statistical Model:

The Model uses the information contained within a linear combination of the Predictor Variables.

The individual Predictor Variables in the Model have a Linear or Monotonic relationship with the Binary Target Variable

The Model is fitted to the Portfolio Population using Binary Logistic Regression

The Model estimates the probability that the Binary Outcome Variable will take on one of its two values ( Default or Not-Default)

The Model Development process produces a set of weights

corresponding to the attributes within each characteristic variable which is included in the Model.

The values / range of the Attribute Weights reflect the Predictive and / or Classification Power of the respective Characteristics.

1515

CRM Scorecard Model Assumptions

The fundamental assumptions behind all predictive credit risk scoring systems are:

The past behaviour of one group of customers can be used to

predict the future behaviour of another group of customers with

similar credit risk profiles (characteristics ).

The future macro-economic and political environment ( in which

the Model will be applied ) will be generally the same as the past

economic environment in which the Model was developed.

The rate of change within the bank’s credit risk portfolio over a

given time period is gradual, smooth and continuous.

Small changes in Input produce consistent changes in Output. 1616

CRM Scorecard Model Assumptions

Unfortunately, these assumptions are often NOT TRUE;

The world is always changing and such change is seldom gradual,

smooth and continuous.

Forecasting the future is a difficult task, especially when shocks to

the system disrupt trends and increase volatility.

Many Events or Factors that influence Customer Default Behaviour

cannot be predicted from the Customer’s Credit Risk Profile.

Given that Credit Risk Scorecards are built using historical data in

order to help predict future behaviour, all scoring systems will lose

Predictive Power over time and eventually require redevelopment.

To maintain and enhance the performance of a scoring system, its

components must be validated on an ongoing basis so that any

deterioration can be identified and corrected.

1717

CRM Scorecard Performance Measures

The Performance of a Credit Risk Scorecard is best evaluated using measures that are specifically related to its application within the Credit Risk Management System:

Rank-Ordering Classification Prediction Explanation

The standard diagnostic tools can be used to measure the Reliability and Predictive Power of a Credit Risk Scoring system across the following three dimensions:

Scorecard and Characteristic Discriminatory Power Scorecard and Characteristic Stability Cut-off Score Performance ( requires Cost Information )

1818


Scorecard and Characteristic Discriminatory Power:

The Model Validation system measures the Power of the scoring system ( and each of the characteristics in the Scorecard ) to separate the "good risk" and "bad risk" sub-populations.

Various statistical performance measures, such as the Gini and

the Kolmogorov-Smirnov (K-S) test, are used to measure the ability of the score to separate “goods” and “bads” and provide the expected "lift" between low and high scoring segments.

The Gini is best used to compare the Power of two or more Binary Decision Scorecard Models derived from the same portfolio population where the costs of the two decision errors are the same for each model.

1919


Scorecard and Characteristic Discriminatory Power:

However, the Gini is NOT a coherent measure of the Power of a Binary Decision Scorecard Model unless the costs of each decision option are

included in the Model Evaluation process.

Type 1 Error: Decision: Reject a Good Customer Cost is related to the loss of potential revenue gain,

Type 2 Error: Decision: Accept a Bad Customer Cost is related to the loss due to possible bad debts.

This is even more important when the Gini measure is used to compare two or more Scorecards each from a different portfolio / population where the decision costs are quite different.

20

Portfolio Type 1 Error Cost Type 2 Error Cost

Hire Purchase

Mortgages

20

CRM Scorecard Monitoring

Scorecard and Characteristic Stability: Economic climates and business environments are constantly changing and this

can influence both the structure and the composition of a credit risk portfolio. These changes will impact the population distributions (with respect to key credit

risk drivers) as well as the interactions between the factors and consequently the changes affect the performance of a credit risk scoring system.

To determine if a scoring system can continue to be used effectively, it is

necessary to measure the stability of the target population relative to both: the Model Development Population the Current Scorecard.

The Population Stability Index measures the degree of change within the target

population, both for the overall scoring system and for each characteristic in the Scorecard.

High Population Stability Index values indicate the population has changed and may suggest that the entire scoring system or a specific component needs Recalibration or redevelopment.

2121

CRM Scorecard Challenges

Given the small size of some Retail Credit portfolios as well as the resulting

low number of “Events” vs. “Non-Events” in the Binary Outcome Variable

that is being Modelled;

The Model Development Method has to overcome several challenges:

Business Requirements for Portfolio Segmentation.

Strong Predictor Variables which may not be “Good for Business”.

Sparse data sets with respect to Predictor Variables and / or the Target Outcome Variable.

“Clumpy” distributions of the “Event” across one or more Predictor Variables.

Uncertain Data Quality during periods of economic flux and portfolio volatility.

Insufficient (historical) portfolio data for robust Scorecard Development.

Scorecard Validation Data requirements may aggravate the above problems.

2222

OK; So what is the Solution?

Accept the limitations of Credit Risk Scorecards.

Evaluate the Statistical Model Assumptions.

Engage the Business in Credit Risk Model Development.

Do Exploratory Data Analysis and Data Preparation for MD

Use a well structured Model Development Methodology

Use a well structured Model Validation Methodology

Select appropriate model performance measures.

Evaluate the application of the scorecard using monitoring.

Frequently Update and Recalibrate each CRM Scorecard.

Evaluate the factors which impact on the CRM Scorecard.

2323

CRM Scorecard Recalibration Methodology

If the Scorecard does not adequately validate on the most recent Portfolio Data, then we will provide recommendations to address any deficiencies that are identified during the analysis.

We are able to assist you with the necessary remedial action, which can include one or more of the following solutions:

Simple re-weighting of the existing Scorecard characteristics and their

attributes.

Optimisation of the attribute structure of the Scorecard characteristics.

Identification and replacement of one or more problematic variables.

Low Default Portfolio Analysis and Model Recalibration

Complete New Scorecard Development.

2424

CRM Scorecard Recalibration Methodology

The Specific Scorecard Recalibration Methodology that is applied will depend on:

The type of Scorecard that is being investigated:

– Application Scorecards ( May impact BASEL II PD Models )– Reject Inference, – Categorical Variables – Integer Scores

– Behaviour Scorecards ( May impact BASEL II PD Models )

– Collections Scorecards

– Recoveries Scorecards

The Structure of the Business Portfolio

The Application of the Scorecard within Risk Management

The Availability of Credit Bureau Data

2525

CRM Scorecard Recalibration Process

Phase 0: ( New Validation )Evaluate the Existing Scorecard on Recent Data

Phase 1: ( New Weights )Keep the Existing Scorecard Structure Intact

Phase 2: ( New Attributes ) Optimise The Attribute Structure of The Variables

Phase 3: ( New Variables ) Replace Weak or Problematic Scorecard Variables

Phase 4: ( New Target Definition ) Low Default Portfolios, RELAX the BAD Definition

Phase 5: ( New Model ) Develop a completely NEW Scorecard Model

2626

CRM Scorecard Recalibration: Phase 0

Phase 0: (Evaluate the Existing Scorecard )

Retain the same underlying Statistical Model.

Retain the same Binary Target Variable Definition

Retain the same Scorecard Characteristic Variables

Retain the same Attribute Structures within each Characteristic

Retain the same Scorecard Weights for each Attribute.

Construct a suitable Data Set for OOT Model Validation.

Evaluate the Existing Scorecard / Model on Recent Data2727


Existing Scorecard Recalibration:Compute New Scorecard Weights

Retain the same underlying Statistical Model


Use the previously constructed Data Set for OOT Model Validation

Keep the Existing Scorecard Structure Intact – Retain the same Scorecard Characteristic Variables– Retain the same Attribute Structures within each

Characteristic Variable

Compute New Scorecard Weights ( Points ) for each Attribute (and therefore for each Characteristic Variable)

2828


Optimise the Attribute Structure for each Characteristic Variable



Keep the Existing Scorecard Structure Intact ( Characteristics )

Retain the same Scorecard Characteristic Variables

Optimise the Attribute Structures within each Characteristic Variable

Compute New Scorecard Weights ( Points ) for each Attribute (and therefore for each corresponding Characteristic Variable)

2929


Replace Problematic Variables



Modify the Existing Scorecard Structure ( Characteristics and / or Attributes )

Replace the Weak or Problematic Characteristic Variables

Optimise the Attribute Structures within each Characteristic Variable


3030


LDP Construct a New Target Variable Definition

Relax the Binary Target Variable Definition to create more positive cases (Defaults) and then apply Phases 1 , 2 & 3.

Modify the Existing Scorecard Structure ( Characteristics and / or Attributes )

Replace the weak or problematic Characteristic Variables

Optimise the Attribute Structures within each Characteristic


3131


Build a NEW Scorecard Model

Follow BGH Group Standards for CRM Models

Construct the Binary Target Variable Definition

Construct the Characteristic Variables and Attributes

Optimise the Attribute Structure within each Variable

Select the best Characteristic Variables for the Model

Fit the Binary Logistic Regression Model

Compute the Scorecard Weights for each Attribute and Characteristic Variable included in the model

Evaluate and Validate the Scorecard Model

3232








Discussion 3333

Case Study: HP Application Scorecard

The Hire Purchase Application Scorecard was developed on 2005-2007 data and implemented in Nov 2009

Recent monitoring of the Scorecard has shown a significant

deterioration in the scorecard in terms of: Discriminatory Power (Gini coefficient) Significance of Scorecard Characteristics Risk Ranking by Score

The Existing Scorecard includes a variable which is having a detrimental effect on the size of the portfolio

Business Requires an Urgent Fix while waiting for the complete Redevelopment of the Existing Suite of Scorecards

3434

Existing HP Application Scorecard

Gini: 11.9Mar 09

35

Variable Attribute BandScore

Points

Total Monthly Income <=418 115

>418 170

Salary & BGH Flag Non BGH Account Holder 121

BGH holder - non salary transfer 160

BGH holder & salary transfer 195

Age < 26 or missing 78

26 - 48 or 53+ 96

48 - 53 126

# of Bank Accounts < 1 108

>= 1 149

Nationality South African 58

Other 86

Photo Card Other 98

Photo card 140

Monthly Expenses > 0 156

<= 0 200

35

ROC Curve for Model Evaluation

0

20

40

60

80

100

0 20 40 60 80 100% Goods

% B

ads

0

20

40

60

80

100

0 20 40 60 80 100% Goods

% B

ads



0

20

40

60

80

100

0 20 40 60 80 100% Goods

% B

ads

0

20

40

60

80

100

0 20 40 60 80 100% Goods

% B

ads


0

20

40

60

80

100

0 20 40 60 80 100% Goods

% B

ads

ROC Curves for Comparison of 2 Scorecards

Existing HP Scorecard Evaluation

Gini: 11.90

Performance of the Existing Scorecard on the Present 2011 Portfolio

• Scorecard no longer performing on the current population

• The BGH flag variable has a negative effect on the portfolio composition

• The Photo Card Variable is not used and data is no longer captured

4141

Recalibration Phase 01- Results

Gini: 44.65

42

Variable Attribute Score Points

Intercept 392Total Monthly Income <=418 -22

>418 23Age <26 or missing 3

26-48 or 53+ 048-53 -2

Monthly Expense <=0 0>0 0

Salary & BGH Flag Non BGH Account Holder -32BGH Holder- non salary transfer -12BGH Holder- salary transfer 163

Nationality South African -36Other 13

Photo Card Flag No Photo Card 3Photo Card -48

Number of Bank Accounts >=1 1<1 -23

42

Recalibration Phase 2 - Results

Gini: 40.38 Gini: 44.65

43


Intercept 392Total Monthly Income <=245 -48

245-299 -22300-449 14

450+ 19<=26 -1027-29 -1429+ 5<=0 40-79 -3180-99 -16100+ 14

Salary Transfer & BGH Customer Non BGH Account Holder -32BGH Holder- non salary transfer -12BGH Holder- salary transfer 162

Nationality Malaysian -31Other 11

Photo Card Flag No Photo Card 3Photo Card -47

Number of Accounts >=1 1<1 -23

Age

Monthly Expense

43

Recalibration Phase 3 - Results

Gini: 49.6

44


Intercept 391

Total Monthly Income <=244 -56

245-299 -26

300-449 17

450+ 23

Monthly Expenses <=0 5

0-79 -43

80-99 -23

100+ 20

Number of Accounts >=1 -25

<1 11

Salary Mode Salary Transfer -17

Non Salary Transfer 34

Nationality South African -32

Non-SA African -55

Other 16

N -25

Y 51

Worktype Salary 5

Self -Employed & Other -124

Standing Instruction Set-Up Flag

44

Final Recalibrated Scorecard - Results

All the attributes within each scorecard characteristic now have significant points

Age variable no longer significant

Photo Card removed

Maximum separation between attribute points to give optimum distribution of scores

BGH & Salary transfer has been replaced by Salary Mode, and worktype

Standing instruction set up has also become significantGini: 49.6

4545

Recalibration Phase 1 - 3: Results

Gini: 40.38 Gini: 44.65

Model Phase Gini KS

Current 0 11.90

Re-Aligned 1 40.38

AttributesOptimised

2 44.65

New Variables

3 49.60

4646

Final Recalibrated Scorecard

ROC Curve

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.2 0.4 0.6 0.8 1

Cum % Goods

Cu

m %

Ba

ds

New

Random

Original

KS

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

170 220 270 320 370 420 470 520

Score

Cu

m %

Acc

ou

nts

Cum % Bads

Cum % Goods

Score Band

Total Accounts

% Accounts Cum % Goods BadsCum % Bads

Bad Rate

low-327 545 11% 11% 462 83 37.22% 15.23%328-370 551 11% 21% 508 43 56.50% 7.80%371-376 458 9% 30% 443 15 63.23% 3.28%377-418 687 13% 44% 665 22 73.09% 3.20%419-424 663 13% 57% 634 29 86.10% 4.37%425-437 199 4% 61% 194 5 88.34% 2.51%438-451 489 10% 70% 476 13 94.17% 2.66%452-482 521 10% 80% 516 5 96.41% 0.96%483-500 689 13% 94% 681 8 100.00% 1.16%501-high 324 6% 100% 324 0 100.00% 0.00%Total 5126 100% 4903 223 4.35%

Scorecard ranks bad accounts• 1st score band has bad rate 15 %

• 37% of bad accounts captured in 1st score band

• 56% of bad accounts captured in first two score bands

4747








Discussion

4848








Discussion

4949





PIC Solution Objectives

Benefits of our Technical Solution

Implementation of the Technical Solution

Discussion5050

Implementation of the Technical Solution

CRM Scorecard Evaluation ( Phase 0 )

CRM Scorecard Recalibration ( Phases 1 to 5 )

CRM Scorecard Validation ( Phase 6 )

Implementation of the Recalibrated Scorecards

Monitoring of the New Scorecard Performance

Implementation of the CRM Strategy

Monitoring of Portfolio Transformation

5151

CRM Scorecard Implementation

Cut-off Score Performance:

A scoring system can continue to rank order the population and provide acceptable discriminatory Power, yet fail to produce the desired credit risk management performance.

Scorecard Cut-Offs are specific score values that are selected when

designing decision rules to automate credit risk decisions that are based on the Scorecard.

Shifts in Scorecard performance by score range may demand adjustments to the score cut-offs and the associated decisions.

Scorecard validations include updated score performance tables for each key portfolio segment so that score cut-offs can be adjusted in line with both credit risk and business strategy.

Cut-Off Analysis should also include the Utility-Costs of each decision

that is based on whether or not the score is above the cut-off value.

5252

Cut-Off Analysis – New Scorecard

We have decreased the overall Bad Rate within the Approved Groups from 4.4% to 3.5%, while the Approval Rate has been decreased from5126 to 5012

Current Cutoff- 293

Approve Decline SummaryGoods 4,480 423 4,903 Bads 152 71 223 Total 4,632 494 5,126

Bad Rate 3.3% 14.4% 4.4%Goods 359 150 509 Bads 21 23 44 Total 380 173 553

Bad Rate*** 5.5% 13.2% 7.9%Goods 4,839 573 5,412 Bads 173 94 267 Total 5,012 667 5,679

Bad Rate 3.5% 14.1% 4.7%

Summary

New Scorecard

Existing Scorecard

Approve

Decline**

5353


Keeping the approval rate at 90%, we can reduce the bad rate to 3.6%

Insert $ value of scorecard

Current Approval Rate- 90%(Score-291)




Bad Rate 3.6% 16.5% 4.8%

New Scorecard

Existing Scorecard

Approve

Decline**

Summary

5454


Keeping the Bad Rate at 4.4%, we can increase the Accept Rate to 97%

Current Bad Rate- 4.4% (Score-192)




Bad Rate 4.3% 20.4% 4.7%

New Scorecard

Existing Scorecard

Approve

Decline**

Summary

5555

Strategy Results (Credit Limit)Difference Report – after 12 months

30.72%

21.54%

25.47%

7.06%

26.59%

4.05%

-1.51%

-29.49%

CHALLENGERCHAMPION CHANGE

Total Current Balance (DR)

Average Current Balance (DR)

Cash Sales

Merchandise Sales

Finance Charges

Increase in % Current Balances

Reduction in % 2Cyc Balances

Over-limit Accounts

CHALLENGERCHAMPION CHANGE

Total 2 Cycle Balance

Total 3 Cycle Balance

Total 4+ Cycle Balance

Roll Rates 2 – 3 Cycle (Accts)

Roll Rates 2 – 3 Cycle (ZAR)

Roll Rates 3 – 4+Cycle (Accts)

Roll Rates 3 – 4+ Cycle (ZAR)

Total Current Balance

Strategy Results (Delinquency)Difference Report – after 12 months

-10.69%

-30.77%

-33.78%

-27.42%

-31.00%

-39.74%

-29.12%

30.72%

Conclusion

How will PIC deliver the required value to BGH?

Practical Vehicles for Delivery: Programme Management

Multiple Projects Project Integration

Project Based Management Terms of Reference Project Design Project Scope Project Plan Project Resources

58

Conclusion

What direct benefits can the PIC Methodology Provide to the Bank of Good Hope?

More Effective and More Efficient Credit Risk Management Account Acquisition Account Management Account Collections

Reduction in Credit Risk Loss Increase in Customer Profitability Practical Training and Skills Development

59

Conclusion

How will PIC deliver the required value to BGH?

General Principles:

Consultancy ( We will listen and understand)

Engagement ( We will empower you )

Collaboration ( We will work together )

Knowledge Transfer ( We will share our expertise )

Partnership ( All of the above )

60

Conclusion 1. Doing the Right Thing

Data Driven Quantitative Score Foundation CRM Policy Business Strategy

2. For the Right Reason Risk Appetite Framework Business Objectives

3. At the Right Time Quantitative Analysis Economic Factors Business Knowledge

6161

Questions

and

Discussion

62

Questions and Discussion

The Effective Use of CRM Scorecards

CRM Scorecard Model Development

CRM Scorecard Model Validation

The PIC Technical Proposal: Recalibration

Project Management and Delivery

Development and Implementation in SAS

Staff Training and Development

63

CRM Scorecard Recalibration

The Scorecard Recalibration Methodology

The Objectives of Scorecard Recalibration

The Benefits of Scorecard Recalibration

The Scorecard Recalibration Process: DETAILS

Analytical Support for Scorecard Recalibration

The Scorecard Recalibration Toolkit

6464

Scorecard Recalibration Phase 0

Evaluate the Existing Scorecard on Recent Data

Data Management:

Request / extract the most recent data set that is required for the specific scorecard evaluation and Recalibration analysis.

If necessary, import the RAW Data into SAS to construct a SAS Data Set.

Obtain and / or construct the Metadata Dictionary from the Business Intelligence Unit.

Conduct Exploratory Data Analysis and Data Cleaning ( Data Quality ).

Verify or Construct the Good-Bad Binary Dependent Variable ( Model Development Definition ).

If necessary, construct the Scorecard Characteristic Variables and the corresponding component attributes ( categories ) in the SAS Data Set.

65

Scorecard Recalibration: Phase 0

Evaluate the Existing Scorecard on Recent Data

1. Score each account in the portfolio using the existing Scorecard;

Translate the Existing Scorecard (Characteristics, Attributes, Weights and Intercept) into SAS Program Code.

Compute the Score for each account in the most recent Population Data Set.

2. Scorecard Model Evaluation:

Use the GINI MACRO and the Scores from the Current Data Set to compute the Gini in order to Measure the Power of the Scorecard to separate “Goods” and “Bads”.

3. Model Performance Measures: [ Classification and Prediction ]

Gini > BGH Group Standard ( Specific Portfolio + Model Type ) ?

66


Evaluate the Existing Scorecard on Recent Data Request / extract the most recent data set required for the specific analysis.

Population Stability Analysis:

Compute the Population Stability Index ( Relative to the Scorecard Model ) to

determine whether the Current Portfolio Population has drifted away from the

Model Development Population that was used to construct the scorecard?

Scorecard Characteristic Stability Analysis:

Compute the Characteristic Stability Index, ( Relative to each Individual

Characteristics in the Scorecard ) to determine whether the Current Portfolio

Population has drifted away from the Model Development Population that

was used to construct the scorecard?

Construct the Existing Scorecard Baseline Performance Report..

6767


Actions after completing Phase 0

Successful Evaluation: Recalibration Decision: STOP

If the Scorecard Performance as measured in the above steps is GOOD or adequate

by BGH Standards and there is no evidence of any fundamental problems with the

Scorecard then STOP and do not proceed to Recalibration Phase 1.

Unsuccessful Evaluation: Recalibration Decision: GO

If the Scorecard Performance as measured in the above steps is NOT GOOD or

adequate by BGH Standards OR there is evidence of fundamental problems with

the Scorecard then proceed to Recalibration Phase 1.

6868

Scorecard Recalibration Phase 01a

Keep the Existing Scorecard Structure Intact

Data Management: ( in addition to Phase 0 )

Based on the Outcome of the Characteristic Stability Analysis and the Scorecard Model Evaluation Phase 0;

Conduct Exploratory Data Analysis and Data Cleaning

Verify or Reconstruct the Good-Bad Binary Dependent Variable

Verify or reconstruct the Scorecard Characteristic Variables and their component attributes.

69

Scorecard Recalibration Phase 01b


For each Nominal or Categorical Characteristic Variable;

Allocate integer values to each distinct category in the Characteristic Variable.

Compute the cross tabulation frequencies of each category with the Good-Bad variable.

Use these frequencies to compute the Weights of Evidence for each attribute.

Map the Weights of Evidence onto the (linear integer scale) created above.

Scorecard Model Evaluation:

Fit a Logistic Regression Model to the WOE-Transformed Scorecard Variables.

Evaluate the Full Logistic Regression Model ( All Variables )

70

Scorecard Recalibration Phase 01c


Model Performance Measures:

Evaluate the Scorecard Variables by comparing the AIC and SBC values for the Intercept Only Model with the corresponding AIC and SBC values of the Full Model.

Classification Model Chi-Square: p-value < 0.05 is GoodGini > BGH Group Standard ( Specific Portfolio + Model Type ) ?KS > BGH Group Standard ( Specific Portfolio + Model Type ) ?

Prediction H-L Goodness of Fit: p-value > 0.05 is Good Parameter Estimates should be large and p values should be small Compare the p-values together with the Information Values

– P-value small and large IV Strong– P-value large and small IV Weak

71



Actions Required for Case A: Successful Recalibration

If the performance of the recalibrated Model is significantly better than that of the original development Model and there has been a significant population shift since development, ( PSI is relatively large) then:

Confirm that each individual characteristic variable in the Model is statistically

significant and Predictive.

Recalibrate the existing Scorecard using the Logistic Regression Model

parameter values obtained from the most recent population data set.

Align the recalibrated Scorecard to the BGH standard

Do Cut-off analysis to select the appropriate cut-off score value in order to

construct the relevant credit risk decision rule.

72



Case B: Unsuccessful Recalibration in P_01 [ Go To Phase 2 ]

If the performance of the Recalibrated Model is NOT significantly better than that of the existing Development Model, then:

Investigate the Predictive Power of the individual Characteristic Variables included in the existing Scorecard (relative to the current population data.)

Investigate the Attribute Structure of each Characteristic Variable that is included in the existing Scorecard (relative to the current population data.)

Identify Weak Predictor Variables that could be excluded from the Scorecard Model.

Identify Problematic Predictor Variables that could be excluded from the Scorecard Model: Bad for Business Data No Longer Collected or Available Product Change Data Quality (Missing Data, Outliers, Errors, Collinearity, Insufficient Spread)

73

Scorecard Recalibration Phase 1d

Go to Recalibration Phase 2.

If the Recalibrated Model from Phase 1 is significantly

better than the existing model but there are one or

more weak or problematic characteristic variables in

the Phase 1 Recalibrated Model.

If the performance of the Recalibrated Model from

Phase 1 is not adequate or there are structural

problems with the Phase 1 Recalibrated Model.

74


Optimise The Attribute Structure of The Variables

Retain the Existing Scorecard Characteristic Variables. For each Continuous Scorecard Characteristic Variable:

Optimise the Attribute Structure ( Bins or Categories ) in order to maximise the univariate Predictive Power of the Characteristic Variable in relation to the Binary Default Variable. (measured by the Gini)

Classify the observed values of the Characteristic Variable into the appropriate ordered set of attribute categories ( Bins ).

Map an ordered set of integer values to the corresponding ordered set of distinct attribute categories ( Bins ) within the Characteristic Variable.

For each Categorical Scorecard Characteristic Variable: Investigate the Attribute Structure of the Categorical Characteristic Variables. Where it is feasible to modify the attribute structure of a categorical

characteristic variable, apply the same optimisation process as described above.

75



Retain the Existing Scorecard Characteristic Variables.

For all Scorecard Characteristic Variables:

Compute the frequencies of each New Attribute Category in a cross tabulation with the Binary Default Variable.

Use these frequencies to compute the Weights of Evidence for each attribute.

Map the Weights of Evidence onto the ( linear integer scale ) created in steps 2 or 3

Fit a Binary Logistic Regression Model to the WOE-Transformed Scorecard Characteristic Variables:

Force all the existing Variables into the Full Model Perform Stepwise Variable Selection to compare the Full Model

with the Subset Model

76



(Retain the Existing Scorecard Characteristic Variables.) Evaluate the Logistic Regression Model using the SAS Output:

AIC and SBC should decrease with the addition of useful explanatory Variables into the Regression Model during stepwise selection.

Model Chi-Square: p-value < 0.05 is Good

Gini and KS should exceed BGH Group Standards

H-L Goodness of Fit: p-value > 0.10 is Good

Parameter Estimates should be large and p values should be small.

Compare the p-values with the corresponding Information Values

77



(Retain the Existing Scorecard Characteristic Variables.)

Case A: Successful Recalibration

If the performance of the recalibrated Model ( based on the new attribute structures ) is significantly better than that of the development Model, then do the following:

Confirm that each individual characteristic variable in the Model is statistically significant and Predictive.

If so then retain the existing Scorecard Characteristic Variables. Apply the new Attribute Structures within each Scorecard Characteristic Variable. Recalibrate the existing Scorecard using the parameter values obtained from the

Logistic Regression Model that has been fitted to the most recent data set. Align the recalibrated Scorecard to the BGH standard Conduct Cut-off analysis to select the appropriate cut-off score value(s) for the

credit risk decision rule(s).

78

Scorecard Recalibration Phase 02e


(Retain the Existing Scorecard Characteristic Variables.)

Case B: Unsuccessful Recalibration in P_02 [ Go To Phase 3 ]

If the performance of the Recalibrated Model ( using the modified attribute structures ) is NOT significantly better than that of the Development Model, then:

Existing Scorecard Investigate the Univariate Predictive Power of the individual Characteristic

Variables included in the existing Scorecard (current population data). Investigate the Multivariate Predictive Power of various groups of Characteristic

Variables included in the existing Scorecard (current population data).

Modified Scorecard Investigate the Univariate Predictive Power of the individual Characteristic

Variables NOT included in the existing Scorecard (current population data). Investigate the Multivariate Predictive Power of various groups of Characteristic

Variables NOT included in the existing Scorecard (current population data).

79

Scorecard Recalibration Phase 02f

Go to Recalibration Phase 3.

If the Recalibrated Model from Phase 2 includes

one or more weak or problematic characteristic

variables which should be replaced.

If the performance of the Recalibrated Model from

Phase 2 is not adequate.

There are structural problems with the Phase 2

Recalibrated Model.

80


Replace Weak or Problematic Scorecard Variables

If the performance of the Phase 2 Recalibrated Model is NOT significantly better than that of the Development Model, then: Investigate the Predictive Power of the individual Characteristic Variables included in

the existing Scorecard (relative to the current population data.)

Investigate the Attribute Structure of each Characteristic Variable that is included in the existing Scorecard (relative to the current population data.)

Identify Weak Predictor Variables that could be excluded from the Scorecard Model.

Identify Problematic Predictor Variables that could be excluded from the Scorecard Model: Bad for Business Data No Longer available Product Change Data Quality (Missing Data, Outliers, Errors, Collinearity, Insufficient Spread)

81



Identify those Characteristic Variables in the Recalibrated Model from Phase 2 that are weak Predictors and do not contribute to the Predictive Power of the Scorecard Model:

Univariate Analysis ( Xk: k = 1 to n ) WOE Information Value Variance

Bivariate Analysis Time Series ( Xk , Time ) k = 1 to n Logistic Regression ( Y , Xk ) k = 1 to n

Multivariate Analysis Logistic Regression ( Y , X1 ,X2 ,X3 ,...., Xn) Gini and KS AIC and SBC Interactions / Correlation / Collinearity

82



Investigate alternative Explanatory Variables that are NOT in the

Recalibrated Model from Phase 2 but which could be used to replace

the weak or problematic Characteristic Variables in the Scorecard.

Use the same analytical techniques as described in previous slides in

order to ensure that the NEW Variables are compared with the OLD

Variables against the same measures so that the potential benefits of

swapping the two sets of Variables is evident.

Replace the weak Predictor Variables presently in the Recalibrated Model

from Phase 2 with strong(er) Predictor Variables that were identified in the

above Phase 3 analysis.

83



The ranking and selection of NEW Variables for inclusion in the Phase 3

Recalibrated Scorecard should be based on the following factors:

Variables that are well behaved Statistically Variables that satisfy the Logistic Regression Model Assumptions Variables that have Univariate Predictor Power Variables that have Multivariate Predictor Power Variables from different Variable Clusters ( Risk Factors ) Variables that make good Business Sense ( Behaviour and Portfolio

Transformation)

If necessary, supplement the Recalibrated Scorecard with a set of Expert

Decision Rules that are based on key risk drivers which are not explicitly

included as Explanatory Variables in the Model.

84


Replace Weak or Problematic Scorecard Variables Construct WOE-Transformed Scorecard Variables for Model Development:

Fit a Logistic Regression Model to the WOE-Transformed Scorecard Variables: Force all the Variables from the Existing Scorecard into the Full Model Perform Stepwise Variable Selection to compare the Full Model with the

Subset Model(s) Evaluate the NEW Logistic Regression Model using the SAS Output:

AIC and SBC should decrease with the addition of useful Explanatory Variables into the regression model during Stepwise Variable Selection.

Model Chi-Square: p-value < 0.05 is Good Gini and KS should exceed BGH Group Standards H-L Goodness of Fit: p-value > 0.10 is Good The Model Parameter Estimates should be large and p values should be small. Compare the Parameter p-values with the corresponding Information Values

85

Scorecard Recalibration Phase 03f


Case A: Successful Recalibration : STOP

If the performance of the Recalibrated Model from Phase 3 is significantly better than that of the Phase 2 Recalibrated Model from, then do the following:

Confirm that each individual Characteristic Variable in the Model is Statistically significant and Predictive.

If so then retain the existing Phase 3 Scorecard Characteristic Variables.

Apply the new Attribute Structures within each Characteristic Variable.

Recalibrate the existing Phase 3 Scorecard using the parameter values obtained from the logistic regression Model that has been fitted to the most recent data set.

Align the Phase 3 Recalibrated Scorecard to the BGH standards

Conduct Cut-off analysis to select the appropriate cut-off score for the credit risk decision rule.

86

Scorecard Recalibration Phase 03g


Case B: Unsuccessful Recalibration If the Recalibrated Model from Phase 3 is NOT significantly better than the Phase 2 Recalibrated Model then do the following: Existing Phase 3 Scorecard Variables

Investigate the Univariate Predictive Power of the individual Characteristic Variables included in the existing Scorecard (current population data).

Investigate the Multivariate Predictive Power of various groups of Characteristic Variables included in the existing Scorecard (current population data).

Potential Phase 3 Scorecard Variables Investigate the Univariate Predictive Power of the individual Characteristic

Variables NOT included in the Phase 3 Scorecard (current population data). Investigate the Multivariate Predictive Power of various groups of Characteristic

Variables NOT included in the Phase 3 Scorecard (current population data).

Go to Recalibration Phase 4 OR Phase 5....

87

Scorecard Recalibration Phase 03h

Go to Recalibration Phase 4

If the Recalibrated Model from Phase 3 is NOT significantly

better than the Phase 2 Recalibrated Model

AND

there are not enough Default Cases for robust model development in

relation to the number of Predictor Variables in the Model.

OR

If the performance of the Recalibrated Model from Phase 3 is NOT

adequate

OR

there are structural problems with the Phase 3 Recalibrated Model,

(insufficient data, time based discontinuities or too few defaults ).8888

Scorecard Recalibration Phase 03h


If the Recalibrated Model from Phase 3 is NOT significantly better

than the Phase 2 Recalibrated Model BUT there are sufficient

Default Cases for robust model development.

AND If the performance of the Recalibrated Model from Phase 3 is

NOT adequate OR there are structural problems with the Phase 3

Recalibrated Model, such as insufficient data, time related

discontinuities or known data quality issues.

89

Scorecard Recalibration: Phase 04a

Low Default Portfolios, RELAX the BAD Definition

Investigate the feasibility of relaxing the default definition, from 90 days to 60 days.

Investigate the impact of relaxing the default definition, from 90 days to 60 days.

Investigate the possible segmentation of the portfolio and the corresponding distributions of defaults across and within the various segments.

Apply the Effective Portfolio Sample Size Heuristic to the Portfolio based on the number of defaults and the information contribution of additional Goods in the sample.

Explore Characteristic Variables that may be more appropriate for the relaxed Bad Definition.

90



Investigate alternative Predictor Variables for possible inclusionin the S[60] Scorecard

Univariate Analysis ( Xk: k = 1 to n ) WOE Information Value Variance

Bivariate Analysis Time Series ( Xk , Time ) k = 1 to n Logistic Regression ( Y , Xk ) k = 1 to n

Multivariate Analysis Logistic Regression ( Y , X1 ,X2 ,X3 ,...., Xn) Gini and KS AIC and SBC Interactions / Correlation / Collinearity

91



Select candidate predictor characteristics for inclusion inthe S[60] Scorecard Model:

Variables that are well behaved Statistically

Variables that satisfy the Logistic Regression Model Assumptions

Variables that have Univariate Predictor Power

Variables that have Multivariate Predictor Power

Variables from different Variable Clusters (Risk Factors )

Variables that make good Business Sense ( Behaviour and Portfolio

Transformation)

92



Compare the structure and performance of the two Scorecard

Models S[60] and S[90] based on the respective two Default

Definitions.

Standard Model: S[90]

Relaxed Model: S[60]

Consider a mapping from the S[60] model outcome to the S[90]

model outcome based on the probability : P[Y 90|Y=60 ]

93



If the Recalibrated Model from Phase 4 is NOT significantly

better than the Phase 3 Recalibrated Model although there are now

sufficient Default Cases for robust model development.

OR

If the performance of the Recalibrated Model from Phase 4 is

NOT adequate

OR

If there are structural problems with the Phase 4 Recalibrated

Model, such as non-compliance with BGH Behaviour Standards

94


A: Develop a completely NEW Scorecard Model

B: Compare the NEW Scorecard Model with:

1. The Original Model

2. The Phase 1 Model




95


Validation of the Final Logistic Regression Model

1. Model Development Sample : Performance Evaluation

2. Hold-Out Sample : Performance Evaluation

3. Out-OF-Time Sample : Performance Evaluation

4. Data Quality and Model Risk

5. Model Development Methodology

6. Business Requirements

96


Implement the Final Recalibrated Model

1. Transformation of the Final Model into a Behaviour Scorecard

2. Behaviour Strategy Development

3. Behaviour Scorecard Approval ( Model, Scorecard, Strategy )

4. Behaviour Scorecard Implementation

5. Behaviour Scorecard Monitoring

6. Portfolio Transformation

97

Scorecard Recalibration Toolkit

With respect to the CRM Scorecard Model Recalibration Toolkit; we have developed a suite of SAS Programs that will enable us to:

Evaluate the present Scorecard relative to the Current Population as opposed to the Development Population [ Quarterly ]

Recalibrate the Existing Scorecard (and its Statistical Model) to the Current Population. ( No modification to the Internal Structure )

Optimise the Internal Attribute Structure of the Characteristics in the present Scorecard relative to the Current Population

Identify weak Explanatory Variables in the present Model which are no longer performing relative to the Current Population or which can no longer be used for business reasons.

Investigate alternative Explanatory Variables which could be used to revitalise the present Scorecard Model by replacing weak Variables.

9898

Documents

CRM Scorecard Re Calibration Methodology BGH