Sampling Methodologies

Bank Supervision and Examination Process

Sampling Methodologies

Comptroller’s Handbook

August 1998

EP-SM

Comptroller of the CurrencyAdministrator of National Banks

EP

Comptroller's Handbook i Sampling Methodologies

Sampling Methodologies Table of Contents

Introduction 1Background 1OCC Policy 2Sampling Objectives 3Sampling Methods 5

Nonstatistical Sampling S Judgmental 5Statistical Sampling S Proportional 5Statistical Sampling S Numerical 6Application of Methods of Sampling 6

Statistical Sampling Assurance 9Precision 9Reliability 11Sampling Risk 12

Sampling Plans 12Population Selection 13Sample Design and Selection 14Sample Evaluation and Interpretation 19Probability Statement 25Report of Examination (ROE) Comments 27

Examination Objectives 29

Examination Procedures 30Proportional Sampling 30Numerical Sampling 32

Appendix 34A. Proportional Sampling Evaluation Worksheet 34B. Numerical Sampling Evaluation Worksheet 37C. Statistical Sampling Overstatements 38D. Statistical Sampling Understatements 39

Glossary 40

Comptroller's Handbook 1 Sampling Methodologies

Sampling Methodologies Introduction

Background

It is usually impractical or impossible to review all items or files whenexamining an area of bank operations, especially if the volume of informationis large. Examiners use sampling to observe a random subset to learn aboutthe multitude of items from which they are drawn. Upon drawing statisticalinferences from this subset, they can state with a certain level of confidencethat the inferences apply to the population as a whole.

Benefits of using statistical sampling include:

C The ability to quantify results and relate them to the entire portfolio beingreviewed.

C The ability to quantify sampling risk (i.e., the risk that the sample is notindicative of the entire portfolio).

C Effective use of limited examiner resources.

This booklet discusses statistical sampling in general, as well as specificmethods of designing, selecting, and evaluating statistically valid samples.

The two types of statistical sampling methods discussed most prominently inthis booklet are numerical sampling and proportional sampling. In numericalsampling, each item in a population is equally likely to be drawn. Inproportional sampling, the likelihood of an item being selected isproportional to the item’s size.

An examiner’s choice of a sampling method depends on the specificobjectives of the supervisory activity. Sometimes, examiners may choose asampling method that is not statistical — that is, they may want to rely onjudgment or specific knowledge about a population in selecting files forreview. Examiners do not use the results from a judgmentally derived sampleto draw conclusions about a larger population. Often, the specificknowledge or judgment used to derive the nonstatistical sample is used todevelop statistically valid samples that divide the population into groups. Each group, or stratum, can then be reviewed, intensively if necessary. Suchsampling improves the examination process.

This booklet provides guidance on which sampling methods are best forspecific areas of examination interest. Although the discussion is gearedtoward sampling loan portfolios, other types of items — investments,deposits, and off-balance-sheet accounts — can be sampled. By sampling,

Sampling Methodologies 2 Comptroller's Handbook

examiners can test the effectiveness of processes, policies, controls,management information systems, or risk management practices.

OCC Policy

The OCC generally does not require the use of any specific samplingmethodology. Unless specifically addressed in a handbook section or otherpolicy guidance, the use of sampling in a supervisory activity, as well as thetype of sampling used, is left to the discretion of the examiner-in-charge (EIC). The “Sampling Objectives” portion of this booklet addresses matters thatexaminers must consider in determining an appropriate sampling approach aspart of a bank’s supervisory strategy.

From time to time, OCC senior management may recommend a specificsampling method for examining certain on- or off-balance-sheet accounts. They may also provide guidance on which parameters to use in selecting asample. Such guidance may be part of a year’s OCC operating plan, as wellas other OCC issuances. For example, the examination procedures for fairlending and community bank consumer compliance have specific guidanceon sampling methods.

When conducting statistical sampling, examiners should use the following guidelines:

C The populations to be sampled should be the portfolios identified in thebank’s approved supervisory strategy. If the focus is on the internallyrated nonclassified portion of the bank’s loan portfolio, the populationshould generally exclude Shared National Credit (SNC) loans. However,there may be circumstances (e.g., testing for process, policy orunderwriting exceptions) in which examiners are justified in includingSNC loans in the sample population. The population should also exclude“credit basket” loans (small business loans requiring limiteddocumentation as outlined in the March 30, 1993 “Interagency PolicyStatement on Documentation of Loans”).

— If the bank’s management information system accurately reportslegally binding commitments, the sample should be selected on thecommitment amount. When the sample can be selected by eitherthe note (an individual loan) or the borrower, examiners shouldchoose the borrower.

C The sampling plans for banks that are part of multibank holdingcompanies should be consistent for all national banks in the company. For example, if the company strategy calls for using proportionalsampling when reviewing the quality of commercial loans in one of thefive national bank subsidiaries, the EIC should use proportional samplingon the commercial loans at each of the other subsidiary banks.


C The EIC should deem statistical sampling reasonable and feasible withrespect to the national bank being examined. The size and compositionof the sample should be commensurate with the risk characteristics of thepopulation being tested. Common sense and judgment are critical indetermining the focus and extent of testing. The examiner shouldconsider any known resource constraints on the bank or examiningpersonnel relating to time, personnel, and costs. These include, but arenot limited to, actual or projected sample sizes, availability of examiningand bank staff, availability of data processing hardware and software, andany other pertinent factors.

C Although proportional and numerical sampling can be done manually, anautomated sampling program or module can be more efficient and lessburdensome. Examiners should try to use a bank’s own automatedsampling program, if available. Examiners may also use other availableautomated programs.

C Because precision and reliability levels (i.e., statistical assurance) affectthe size of samples, the OCC recommends that examiners use thefollowing guidelines. For proportional sampling, examiners should useprecision levels of 5 percent, 10 percent, 15 percent, or 20 percent andreliability levels of 80 percent, 86 percent, 90 percent, or 95 percent. Fornumerical sampling, examiners should use precision levels of 5 percentor 10 percent and reliability levels of 90 percent or 95 percent. Thisbooklet discusses statistical assurance in a later section.

Sampling Objectives

Supervisory strategies for national banks should specifically note if theexamination process will use sampling. Based upon the objectives of thesupervisory activities, the strategy should identify the portfolios to sample, thetype of sampling to use, and the purpose of the sampling. If examinationplanning activities indicate a previously unidentified need for sampling, theEIC should change the strategy to note this.

In determining the type of sampling to use, examiners consider the quantity,quality, and nature of the population to be reviewed; the bank’s riskmanagement systems; the bank’s appetite for risk; the objectives and benefitsof the different sampling methods; the purpose and objective of the sample;and resource constraints. Examiners may choose which of the three samplingmethods — judgmental, proportional, or numerical — will best meet theobjectives of supervisory strategy.


The following list highlights some strategic objectives that examiners maywant to meet by using sampling.

C Commercial loans, commercial real estate loans, floor plan, andreceivable financing:

S Validate risk management and internal problem loan identificationsystems or test process effectiveness.

S Identify additional credit risk exposure in the sampled portfolio.

S Determine compliance with underwriting standards.

S Identify administrative weaknesses (e.g., financial statement or otherdocumentation exceptions).

S Assess the adequacy of the allowance for loan and lease losses(ALLL).

C Retail loans (e.g., consumer paper, credit card, check credit, residential

real estate, and home equity lending):

S Validate adherence to appropriate performance reporting standardsby checking extension or re-aging history.

S Determine lien perfection status.

S Assess compliance with underwriting standards through a review ofoverrides, extensions, renewals or other loans.

S Determine compliance with laws and regulations (e.g., consumercompliance).

C Municipal investments or derivatives:

S Validate risk management systems.

S Identify credit or market risk exposure.

S Determine compliance with investment standards.

S Identify administrative weaknesses (e.g., financial statement or otherdocumentation exceptions).


C Deposit accounts:

S Verify the accuracy of the interest rate, terms, and conditions whencompared to the disclosed contractual information.

S Determine compliance with a bank’s deposit policies.

S Identify weaknesses in a bank’s deposit operations.

S Determine compliance with applicable laws and regulations.

C Consumer compliance:

S Determine compliance with fair lending laws and regulations.

S Determine compliance with truth-in-lending and other consumercompliance disclosure requirements.

S Evaluate the reliability of a bank’s compliance management system.

Sampling Methods

Nonstatistical Sampling S Judgmental

Judgmental sampling is sampling without statistical measurement. The OCChas historically used, and continues to use, nonstatistical judgmentalsampling in its examinations. Using sound judgment and knowledge of abank’s policies, controls, and systems, examiners identify the bank’s areas ofgreatest risk exposure and select items for review.

Judgmental sampling allows examiners to review an identified percentage(coverage) of a specific population. Although examiners cannot statisticallyrelate the results of this sample to the entire population of items, they canidentify specific exceptions. The results of the judgmental sample areconsidered when examiners evaluate the quality of the population reviewed. Examiners should comment on exceptions, identifying the root causes ofthose exceptions. For example, exceptions noted in a review of loandocumentation on instalment loan extensions could be the result of bankpersonnel’s lack of training or inadequate knowledge of bank policy.

Statistical Sampling S Proportional

Proportional sampling is appropriate when the dollar amount of items isrelevant to the objective of a procedure, particularly when those dollaramounts vary significantly. In proportional sampling, the population to besampled is defined by dollar amount. Large dollar amounts have a greater


chance of being selected than smaller ones; all items greater than a certainamount and a representative number of smaller items are selected. Proportional sampling is useful to examiners evaluating the quality of a loanportfolio because of the effect larger dollar items can have on asset quality. Sometimes, a strategic objective will call specifically for this approach.

The objective of the sample is to review items for a specific characteristic,which is also called the feature of interest. The OCC has historicallyendorsed the use of proportional sampling to discover additional classifiedloans in commercial loan portfolios. Proportional sampling also isappropriate when examiners want to test population items for monetaryerrors (i.e., incorrect finance charges, incorrect late charges) or complianceerrors (i.e., compliance with underwriting standards, incomplete disclosurestatements).

Statistical Sampling S Numerical

In numerical sampling, the population to be sample is defined by the numberof items. Numerical sampling is usually used to reveal the presence (orabsence) of a defined characteristic in a portfolio of items with similarcharacteristics. Each item in the population has the same probability ofselection as any other. Therefore, examiners may evaluate the results ofapplying numerical sampling and the related examination of selected itemsonly in terms of the number of errors or exceptions. This statistical samplingmethod is appropriate for examination procedures in which the frequency oferrors, exceptions, or another feature of interest is of primary concern and thedollar amount of the exception is not considered relevant.

This method is a valid sampling procedure for determining adherence to arequirement, such as a bank’s underwriting standards or interagencyclassification policies for delinquent loans. The OCC has historicallyendorsed the use of numerical sampling to evaluate retail credit portfoliossuch as instalment loans and residential real estate loans. Examiners can usenumerical sampling to gain a broad perspective about the adequacy of abank’s administrative controls and practices. Sample results can indicate theeffectiveness of these systems but do not provide a quantitative measure ofthe results. Depending upon the sample objectives, results are indicated bythe level of compliance with a bank’s administrative controls, banking lawsand regulations, bank policy, or OCC policy.

Application of Methods of Sampling

Following are suggestions for applying statistical sampling plans and specificprocedures to areas of examination interest:


Program Examples of Sampling Procedures

Due From Banks Use proportional sampling to select accountswhose reconcilements are to be reviewed.

Investment Securities Use proportional sampling to selectmunicipal securities and money marketholdings to determine compliance withinvestment standards.

Bank Dealer Activities Use proportional sampling to select tradingaccount securities for examination.

Loan Portfolio Use proportional or numerical sampling toManagement validate internal loan review.

Commercial Loans Use proportional or numerical sampling toAccounts Receivable evaluate credit quality, compliance withFloor Plan underwriting standards, accuracy ofCommercial Real Estate internal risk rating.Commercial Lease FinancingIndirect Dealer Lines

Instalment Loans Use numerical sampling to test:Credit Cards S The accuracy of MIS, such as bank-Check Credit prepared past due, problem loanHome Equity listing, and insider loans.Residential Real Estate S Renewals, deferrals and extensions for

compliance with policy, degree ofusage, accuracy of reporting.

S Score overrides for compliance withpolicy guidelines and fair lending lawsand regulations, documentation,reasonableness of credit decision.

S Recently extended loans forcompliance with underwriting policy,credit criteria, laws and regulations.

Other Real Estate Owned Use proportional sampling to test for policyadherence, when appropriate, based onnumber and volume.

Deposit Accounts Use numerical sampling to select accounts totest computation of interest, early withdrawalpenalty, compliance with laws andregulations.


Fiduciary Accounts Use numerical sampling to test compliancewith trust agreements.

Off-balance-sheet accounts Depending upon the sample objectives, useeither proportional or numerical sampling.

Consumer Compliance Use numerical sampling to identify a samplefor Truth-in-Lending or Truth-in-Savingsreviews, or to test the integrity of HMDA andCRA small business lending reports.

Community Reinvestment Act Use numerical sampling to select sample forcomparison of credit extended inside andoutside a bank’s assessment area or toperform a lending test.

Examples of how to sample for specific objectives:

Objective Suggested Method

Identify unrecognized Proportional sample of all loans that are classified loans in not internally rated.commercial loan portfolios.

Validate current internal risk Proportional or numerical sample of allratings and identify commercial loans, both those that areclassified loans in commercial internally rated and those that are not.loan portfolios.

Test for compliance with Numerical or proportional sample fromunderwriting practices and, new assets booked since the last policy, e.g.,credit criteria, examination for a defined period (e.g., 30documentation, pricing to 90 days).and terms.

Test for compliance of credit Numerical sample of new loans thatscore loan overrides with scored below a cutoff, but werepolicy and fair lending laws approved in the recent past (e.g., 30 to 90and regulations by checking days).compliance with guidelines,documentation of reason foroverride, and appropriatenessof credit decision.


Test loan renewals and Numerical sample from the entireextensions for compliance population to test usage and accuracy ofwith policy, degree of usage, reporting, or numerical sample from theand accuracy of reporting. bank’s renewal and extension listing to test

for compliance with policy.

Determine accuracy of the Numerical sample from the entire bank’s management population and trace reportable items toinformation system. appropriate listings, e.g., past due, renewals

and extensions, re-agings, prepayments, andinsider loans.

Identify risk exposure Proportional sample of all municipal bondsin the municipal bond and evaluate the sample for asset quality.Portfolio.

Determine accuracy of the Numerical sample from the entirebank’s external reports, such population to test the accuracy of reportedas HMDA and CRA small data.business reports.

Determine compliance of Numerical sample of deposit accountsdeposit accounts with policy booked within the last six months or sinceguidelines, deposit contract the last examination and determineterms, and applicable laws compliance with bank policy. Ensureand regulations. that rates and terms of the deposit are in

agreement with the contract and anyapplicable laws and regulations.

Determine compliance of Numerical sample of all accounts orfiduciary accounts with trust accounts booked since the lastagreements. examination and evaluate the accounts to

ensure that the accounts are being handled inaccordance with the trust agreement.

Statistical Sampling Assurance

Precision

“Precision” is the examiner’s tolerance for exceptions in the sample. Normally, precision implies a range of acceptable values. Most examinationprocedures are designed to test exceptions to a bank’s risk managementsystems, i.e., its policies, practices, and procedures. Since few exceptionsshould exist, sampling procedures are concerned primarily with the upperprecision limit. Examiners want to know the greatest effect (the upper


precision) that sampling exceptions will have on the bank’s condition. Theyshould adequately document in working papers their reasons for selecting acertain precision value. Examiners should consider quantity of risk, directionof risk, and quality of risk management in determining precision levels to use. In designing samples, the precision limit affects the sample size; the smallerthe precision limits, the larger the size of the sample selected.

Proportional Sampling

When examiners use proportional sampling, precision is set using apercentage of Tier 1 capital plus the allowance for loan and lease losses(ALLL). The larger the quantity or number of exceptions the EIC can tolerate,the larger the precision limit should be. Examiners should base the precisionlimit for proportional samples on their knowledge of a bank’s financial andoperating conditions.

Examiners should consider:

C The composition of the bank’s portfolio. For example, the examiner maytolerate more exceptions in a loan portfolio segment supported by liquidor marketable collateral than in one secured by real estate constructionloans.

C The bank’s risk management systems and controls. The better the bank’srisk management system, the more exceptions an examiner may tolerate(in the belief that the bank will identify deficiencies).

C The amount of previously identified classified loans. If a bank has a high

level of classified loans, the examiner may select a lower precision levelbecause she or he is less tolerant of additional classified loans in theportfolio.

C The local economy. If the local economy is strong and stable, theexaminer may tolerate more exceptions than if the economy isexperiencing a decline.

Individual bank circumstances may support a precision limit of 5 percent, 10percent, 15 percent, or a maximum of 20 percent for proportional sampling. (Any value less than 20 percent can be chosen; for discussion purposes in thisbooklet, increments of five are used.) When an examiner can tolerate fewexceptions, a precision of 5 percent is normally chosen. When an examinercan tolerate a large rate of exceptions, a precision of 20 percent is normallychosen. Precision levels greater than 20 percent are not recommended.

For example, assume that since its last examination, a bank began extendinga significant amount of new loans to the cable television industry. The bank’sloan review division has not performed an in-depth review of this area before


the scheduled examination date. Several of these credits appear on thedelinquency report. Based on reported delinquencies in a portfolio not yetinternally reviewed, the examiner may select a precision of 5 percent and thesample size will be relatively large. On the other hand, when a bank’s internal classified figure is low, it issufficiently capitalized, the examiner has confidence in the internal loanreview system, sound control systems are in place, and the nonclassifiedcommercial loan portfolio is to be reviewed, a precision of 20 percent and arelatively small sample may be appropriate.

Numerical Sampling

An examiner sets a precision limit in numerical sampling by deciding howmany exception items can be tolerated in the sample population. The moreexceptions the examiner can tolerate, the higher the precision limit shouldbe. Because noncompliance indicates weakness in policies, systems, orcontrols, bankers and examiners should have a low tolerance for exceptionsin that area. Examiners should consider setting the precision level at 5percent or 10 percent.

Reliability

“Reliability” is the level of confidence in sample results. Selecting areliability level will affect the size of a sample: the higher the reliability level,the greater the number of items examiners will review. Examiners shoulddocument in working papers why they selected a certain reliability level. They should not select reliability levels without considering the bank’sfinancial condition, risk profile and risk management systems. The moreconfidence examiners have in the institution and its risk managementprocess, the fewer OCC examiner resources should be needed to evaluatetransactions.

Reliability factors are numbers that reflect reliability levels (95 percentreliability equates to a 3.0 factor; 80 percent reliability equals a 1.6 factor). Reliability factors are used to determine sample sizes and precisionadjustment factors. When exceptions are found, examiners use reliabilityfactors to adjust previously selected precision limits to evaluate sample resultsand perform statistical sampling projections. (See examples in this booklet’ssection “Sampling Plans” under “Sample Design and Selection” and“Numerical Sample Evaluation Worksheet Example.”) The reliability factorsin this booklet, which are taken from Poisson probability distributions (seeglossary for definition), are approximations of the probability of finding atleast one exception in a sample of selected items at a given reliability leveland error rate. Reliability factors and their associated reliability levels andprecision adjustment factors are noted in appendixes C and D and arecommon references for audit sampling purposes.


Examiners should consider four different reliability levels when usingproportional statistical sampling S 95 percent, 90 percent, 86 percent, and 80percent. These levels allow examiners some flexibility in structuring thesample based on an assessment of the factors mentioned at the beginning ofthis section and the resultant tolerance for exceptions. These levels alsoallow examiners to control the associated degree of sampling risk (definedbelow).

Two reliability levels should normally be considered for numerical sampling— 90 percent and 95 percent. Because the objective of numerical samplingis generally to determine whether the bank adheres to a policy, system, orcontrol, the OCC desires a high degree of confidence (i.e., minimal samplingrisk) in the results.

Sampling Risk

Sampling risk is the difference of one minus the reliability level. With 95percent reliability, the sampling risk is 5 percent. This means that 5 percentof the time the results of the sample may not be truly indicative of the entireportfolio. With 80 percent reliability, the sampling risk is 20 percent. Thismeans that 20 percent of the time, or one time in five, the results of thesample may not be truly indicative of the entire portfolio. Because theobjective of sampling is to test for deviations from a bank’s policies,procedures or practices, the OCC wants to maintain sampling risk at levelsthat allow a high degree of confidence in sample results and their validity. Sampling risk greater than 20 percent is not acceptable.

Sampling Plans

When sampling is used, examiners devise a sample plan to make the mostefficient and effective use of resources to meet the objectives of the review. Each sample plan design involves the same concepts:

C Population selection. C Sample design and sample selection. C Sample evaluation and interpretation of results.

The following comments and examples are provided to illustrate the samplingplan design process for the three sampling methodologies. Although thecomments and examples below relate to the commercial and instalment loanportfolios, examiners can use sampling in any asset, liability, or off-balance-sheet account.


Population Selection

In population selection, examiners decide which portfolios to sample. Theydo so by considering the approved strategy for the bank and the objectives ofthe examination. Strategies may call for examiners to review the entirecommercial loan portfolio or focus on particular types of loans. Types aregrouped by credit, industry affiliation, or bank lending division. Selecting avery broad category, such as the entire commercial loan portfolio, can resultin a very large sample size. When possible, examiners should considerfurther segmenting such large portfolios to reduce the workload. To makestatistically valid conclusions, examiners must define or stratify (group) theselected population as much as possible by their characteristics.

For OCC purposes, assets generally can be grouped into two categories. Inone category are assets that typically use common, uniform underwritingstandards (e.g., one-to-four-family residential real estate loans, consumerinstalment loans, credit card loans, home improvement loans, home equityloans, and overdraft lines of credit). In the other category are assets withbroader common characteristics, such as type of credit, industry, or bankdivision (e.g., commercial real estate loans, real estate construction ordevelopment loans, municipal investment securities, private placements, oiland gas loans, and mortgage warehousing).

Example of Population Selection — Judgmental Sampling

The examiner is concerned about the accuracy of the bank’s risk gradingsystem for commercial real estate loans identified as problem credits. Injudgmental sampling, an examiner may decide to review files for commercialreal estate loans of $500,000 or more that are internally classified. Theexaminer reviews all files selected.

Example of Population Selection — Proportional Sampling

The examiner wants to determine whether the nonclassified loans in a bank’sasset-based lending division pose unrecognized risk. Proportional samplingcan be used effectively to select a statistical sample of loans internally ratedpass. This portfolio is considered “grouped” because the population includesall nonclassified loans in this particular division.

Example of Population Selection — Numerical Sampling

The examiner is concerned about the number of overrides in a bank’sinstalment loan portfolio. The examiner can use numerical sampling to testthe extent of compliance with the bank’s override policy.


Sample Design and Selection

In sample design and selection, the examiner determines which items shouldbe in the sample and bases selection of the sample items on selectedreliability and precision levels. Examiners can conduct a statistical samplemanually or by using an automated sampling program. Sample Selection — Judgmental Sampling

The examiner uses sound judgment to determine and select items for review. For example, the selected loans might include ten loans extended since thelast examination, five significant doubtful-rated classified loans, seven large-dollar substandard loans, two new loans to insiders, and any other type ofloan that the examiner decides to review. The examiner determines whetherthe internal loan rating is accurate (by reviewing the loans’ quality) andwhether the underwriting standards are safe and sound.

Sample Selection — Proportional Sampling

Sample design, with a proportional statistical sample, consists of selecting thereliability and precision.

To select sample items from loan portfolios, the examiner may use the samecriteria followed by the bank’s internal loan review. Alternatively, theexaminer can sample from every loan in the population or, to eliminate aloan review of small dollar loans, the examiner can specify that the sample istaken of all loans exceeding a certain dollar amount.

When examining the internally rated, nonclassified commercial loanportfolio, loans to be excluded from the sample are ones extended as part ofthe minimal loan documentation program, i.e., “credit basket” loans. Additionally, shared national credits might be excluded from samplesdesigned to test risk rating accuracy because they are reviewed separately aspart of a specific program.

With the internally rated, nonclassified commercial loan portfolio, examinersmight consider using a precision level of 20 percent and a reliability level of80 percent. These levels can be justified when the bank has adequatecapital, acceptable internal loan review, sound risk management systems, andgood asset quality.

Once examiners establish precision and reliability, the sample is selected. The size for a proportional sample (based on selecting the sample by note) isestimated as:


population amount in $ = sample size (# of items)

monetary interval

The monetary interval is calculated as:

(Tier 1 capital + ALLL) x precision level as a decimal

reliability factor (From table 1, appendix C)

NOTE: (Tier 1 capital + ALLL) x precision level is also known asmonetary precision.

Stated another way, the equation for estimating the sample size is as follows:

reliability factorpopulation amount in $ x = sample size

monetary precision

Examples of calculations: An examiner is examining a regional bank with$345 million in Tier 1 capital plus the ALLL. The nonclassified commercialloan portfolio, excluding “basket” and, if warranted, SNC loans, is $1.3billion.

1. With a desired precision level of 20 percent and a reliability level of 80percent, the equation for calculating the monetary interval is:

$345MM x .20 $69MM = = $43MM

1.6 1.6

The estimated sample size is equal to the dollar amount of the samplepopulation divided by the monetary interval. (If the result is a fraction, therule of thumb is to round down to the next whole number.)

$1,300MM = 30 loans

$43MM


2. If the desired precision level is 5 percent and the reliability level is 95percent, the equation for the monetary interval is:

$345MM x .05 $17.25MM = = $5.75MM

3 3

The estimated sample size is equal to the sample population divided bythe monetary interval.

$1,300MM = 226 loans

$5.75MM

Once the appropriate sample design is completed, sample items need to beselected. The examiner must begin the sample selection from a randomstarting point. This starting point is a randomly chosen number between zeroand the monetary interval.

For example, if the monetary interval is $5,750,000, the random numericalstarting point must be a number between zero and 5,750,000. The randomnumber may be obtained from various methods, such as a table of randomnumbers or the serial number of a dollar bill. Automated sampling softwarewill normally have a random number generator feature.

Discussed below are two sample methods for selecting items using a standardcalculator and the bank’s trial balance or other report listing samplepopulation items. For illustration, the following examples use 462,021 as thesample’s random numerical start.

Cumulative total. Beginning with the first item on the trial balance, thismethod involves adding the dollar amount of each item to an ongoing total. The item whose balance results in the subtotal equaling or surpassing therandom numerical start (e.g., $462,021) is chosen as the first sample item. The remaining sample items are selected by continually adding itemamounts from the trial balance and selecting each item whose balance causesthe subtotal to equal or exceed each increment of the monetary interval (i.e.,$6,212,021, $11,962,021, etc.).

Negative/Positive. Beginning with the random numerical start entered intoan adding machine as a negative number (e.g., -462,021), the dollar amountsof the trial balance items are added until they subtotal zero or a positivenumber. The item that triggers this event is selected as the first sample item. From the subtotal, the random numerical start is then continually subtracteduntil the subtotal again results in a negative number. Then the dollar


amounts of the remaining trial balance items are added until the subtotal isagain zero or a positive number. The item that triggers the zero or positivesubtotal is the next sample item selection. This process is repeated to selecteach subsequent sample item until the population of items is exhausted.

Sample Selection — Numerical Sampling

The sample size for a numerical sample is determined by the reliability andprecision levels selected.

reliability factor (from table 1)sample size =

precision (as a decimal)

The following chart shows the sample sizes associated with OCC desiredprecision and reliability levels for numerical sampling.

Sample Sizes

Reliability - 90% Reliability - 95%

Precision 5% 46 60

Precision 10% 23 30 When reliability is 95 percent (the corresponding reliability factor is 3.0, from appendixes C and D) and precision is 5 percent, the examiner selects 60items.

When reliability is 90 percent (the corresponding reliability factor is 2.3, from appendixes C and D) and precision is 10 percent, the examiner selects 23items. Although a sample of fewer than 30 items can be selected andreviewed, examiners should not project sample results based on such a smallsample because of the high degree of error inherent in such a small samplepopulation.

Examiners should use a bank’s automated sampling software, if available, toperform numerical sampling. However, a numerical sample can be selectedmanually.

When selecting a numerical sample manually, examiners should divide thepopulation size by the sample size to determine the sampling interval.


population size in number of items = sampling interval

sample size

(If the result is a fraction, the rule of thumb is to round down to the nextwhole number.)

For example, if there are a total of 1,000 override instalment loans, thesampling interval is 33 when reliability is 95 percent and precision is 10percent (30-item sample size).

1,000 = 33

30

The item that coincides with the random starting point (using a randomnumber selected from a dollar bill, random number table, or other source) isthe first sample item selected for review. The examiner continues countingand selecting subsequent items that coincide with the sampling interval.

The sampling interval for numerical sampling can also be derived using otherselection techniques. These techniques include measured interval, specificposition, and terminal digit, and are applied after determining the randomstarting point.

C Measured interval uses a set distance (e.g., inches) to select items forreview. Each subsequent sample item will be the same measured intervalfrom the last sample item. If the distance from the first sampled item on atrial balance to the second is five inches, the examiner measures everyfive inches on the trial balance and selects a sample item.

C Specific position uses a set position on each page of a list to select itemsfor review. If, for example, the first selected sampled item was the lastitem on a page, the examiner could select that same item from eachsubsequent page.

C Terminal digit selects sample items using the same terminal digits, i.e., theone or more digits furthest to the right. Those items having the sameterminal digits S for example, 0089, 0189, 0289 . . . 9989 S would beselected.


Sample Evaluation and Interpretation

In sample evaluation and interpretation, examiners review the sample resultsand draw conclusions about the entire population of data. By designing asample plan, examiners attempt to control the risk of a significantdeteriorating condition remaining undetected. The evaluation of the samplehelps examiners achieve that objective.

Statisticians recommend that a sample be at least 30 items. When smallersamples are used to make projections about the population from which theyare drawn, the degree of error is too great. Examiners can still pull samples ofless than 30 items, but they should not be used to make projections about thelarger population.

Judgmental Sampling

Sample results from nonstatistical sampling cannot be projected beyond theloans sampled. However, exceptions discovered when evaluating thesecredits may suggest a larger problem. The results of a judgmental sample areconsidered when examiners evaluate the quality of the population fromwhich the sample was selected. Examiners should comment on exceptionsand identify the nature and possible root causes of the exceptions. Forexample, an individual bank employee’s lack of training or inadequateknowledge of bank policy could cause exceptions. Examiners should discussexceptions and root causes with bank management.


One of the OCC’s objectives in performing a proportional statistical samplein a commercial loan portfolio is to identify exceptions. An exception in thiscase is, for example, a loan classified by the OCC but not classified by thebank. Such exceptions can either be full exceptions (the entire loan amountis classified) or partial exceptions (only a portion of the loan is classified).

When reviewing a sample of loans a bank has not classified, an examinermight consider screening loans using credit file information. Doing so couldreduce file review time. Since none of the loans in the sample has beenclassified by the bank, only a minimal file review might be warranted. However, screening requires experience and knowledge (see the“Classification of Credit” section of the Comptroller’s Handbook). Whenscreening loans, examiners could answer the following questions:

C Is the purpose of the loan identified?

C Is the source of repayment identified and is the loan paying as agreed?

C Is the loan secured by marketable or liquid collateral?


C Is the financial statement current and does it support the source ofrepayment?

C Does the nature of the business warrant further consideration?

Based on the answers to these questions, examiners decide whetheradditional loan file analysis is necessary. If screening suggests that the loanshould be classified, examiners must analyze it more extensively usingtraditional methods.

In this example of reviewing pass loans for rating accuracy, OCC examinersassign loan ratings (pass, special mention, substandard, doubtful, loss) to allsample loans reviewed. The degree of classification (substandard, doubtful,or loss) is not factored into a statistical probability evaluation; the significantfactor is that the OCC considers the loan to be classified and the bank’sinternal loan review system has not classified it. If the OCC classifies a loanthat the bank does not, the difference is an “exception of overstatement.”

When an examiner reviews a portfolio containing both classified andnonclassified loans, the evaluation is conducted in a slightly different manner. Even if ratings are somewhat different (i.e., the OCC classifies the loansubstandard, while internal loan review rates it doubtful), no exception existsfor statistical sampling purposes. But an exception could exist if samplingwas done on a population of loans all of which are rated substandard by thebank and a portion of which are rated doubtful by the OCC. If internal loanreview rates a loan as classified and the OCC considers it a pass or specialmention loan, the difference is an “exception of understatement.”

Both understatements and overstatements are considered when interpretingsample results.

If no exceptions are found, the desired statistical assurance (reliability leveland precision level) has been attained and no further evaluation is required. When exceptions are found, examiners should further analyze and evaluatethe exceptions. Examiners should attempt to determine the root causes ofexceptions and whether exceptions are isolated occurrences or a pattern orpractice. Some possibilities to explain an exception could be:

C The inexperience of a bank officer.

C An intentional disregard for policies and procedures.

C Exception items unique to a particular type of asset, division in the bank,or industry.


C Internal reviewers assigning ratings differently because theymisunderstood internal classification definitions.

C Untimeliness of bank rating changes (information was not available at thetime of the internal review).

Examiners could identify other reasons for the exception. A reasonableexplanation for an exception does not mean examiners should exclude itfrom sample results. Examiners should note causes of exceptions and discussthem with bank management. Examiners may recommend that managementconduct its own review of exceptions.

If exceptions are found, the original reliability and precision levels are nolonger valid. Examiners must revise their statistical assurance based uponsample findings, but should not enlarge the statistical sample or alter thereliability level. Enlarging the sample would assume that original sampleexceptions were not representative of exceptions in the population andcorrupt the statistical validity of the sample. However, the type and causes ofexceptions noted may indicate a need for further testing under supervision byrisk.

Since the reliability level is the confidence in the sample, changing reliabilityis not desirable. Therefore, examiners must adjust the precision level afteridentifying exceptions. Appendixes C and D are tables that examiners canuse to arrive at precision-adjusted exception levels. Appendix C determinesthe adjusted precision for overstatement exceptions; appendix D determinesthe adjusted precision for understatement exceptions.

A proportional sample evaluation worksheet will help examiners evaluate thesample (see examples of completed worksheets on the following two pages). By projecting the value of all sample exceptions to the population fromwhich the sample was selected, the worksheets arrive at an estimatedadditional percentage of classified loans in the commercial loan portfolio.

The worksheet can be prepared manually or in an automated format. Instructions for the worksheet can be found in appendix A.


PRO

POR

TIO

NA

L SA

MPL

E EV

ALU

ATI

ON

WO

RK

SHEE

T I

Exam

ple:

A s

tatis

tical

sam

ple

was

con

duct

ed o

f all

com

mer

cial

loan

s ra

ted

”pas

s” b

y th

e ba

nk’s

Inte

rnal

Loa

n R

evie

w (I

LR).

ILR

iden

tifie

d a

tota

l of

$362

,413

,000

cla

ssifi

ed c

omm

erci

al lo

ans

(105

per

cent

of T

ier

1 +

ALL

L).

A r

evie

w o

f the

sam

ple

item

s re

sulte

d in

four

loan

s (a

ll be

low

the

cuto

ff,i.e

., th

e m

onet

ary

inte

rval

) bei

ng c

lass

ified

by

the

exam

iner

whi

ch th

e ba

nk c

onsi

dere

d Pa

ss.

This

exa

mpl

e w

orks

heet

is c

ompl

eted

to s

how

how

the

valu

e of

thos

e ex

cept

ions

rel

ate

to th

e pr

ecis

ion

fact

or.

Tier

1 C

apita

l + A

LLL:

$ 3

45M

M

Rel

iabi

lity:

90

% P

reci

sion

: 1

0 %

Mon

etar

y Pr

ecis

ion(

MP)

$ 3

4.5M

M M

onet

ary

Inte

rval

(MI):

$ 1

5MM

OC

CR

atin

gB

orro

wer

Loan

Bal

ance

(L)

Sam

ple

Exce

ptio

n(S

E)

Sam

ple

Inte

rval

MI/L

=SI

Cut

off

Loan

sPo

pula

tion

Exce

ptio

nSE

xSI=

PE

Ran

kPr

ecis

ion

Adj

ustm

ent

Fact

or(P

)

Prec

isio

n-ad

just

edEx

cept

ion

PE x

P

$2,1

88$2

,188

6.85

56$1

5,00

01

1.59

$23,

850

$6,0

07$2

,857

2.49

71$

7,1

343

1.36

$ 9

,702

$3,8

50$3

,850

3.89

61$1

5,00

02

1.44

$21,

600

$ 2

00$

4

075

.000

0$

3,0

004

1.32

$ 3

,960

TOTA

LS$8

,935

$40,

134

$59,

112

RES

ULT

S:Pr

ecis

ion-

adju

sted

Exc

eptio

ns=

$

59,

112

Pr

ecis

ion

Leve

l of (

as %

of T

ier

1 +

ALL

L)=

+$

3

4,50

0

Add

ition

al P

roje

cted

Cla

ssifi

ed L

oans

==

$

93,

612

B

ank

Rep

orte

d C

lass

ified

Loa

ns=

+$

362

,413

Adj

uste

d Pr

ojec

ted

Cla

ssifi

ed L

oans

==

$ 4

56,0

25

THER

EFO

RE:

With

90

%

rel

iabi

lity,

we

infe

r th

at a

ctua

l cla

ssifi

ed c

omm

erci

al lo

ans

shou

ld n

ot e

xcee

d 13

2 %

of T

ier

1 C

apita

l plu

s A

LLL.


PRO

POR

TIO

NA

L SA

MPL

E EV

ALU

ATI

ON

WO

RK

SHEE

T II

Exam

ple:

Thi

s si

tuat

ion

is s

imila

r to

the

prev

ious

pag

e in

dat

a us

ed.

How

ever

, in

this

exa

mpl

e th

e sa

mpl

e w

as ta

ken

of a

who

le c

omm

erci

al lo

anpo

rtfo

lio c

onsi

stin

g of

bot

h cl

assi

fied

and

nonc

lass

ified

loan

s. T

he s

ampl

e re

sulte

d in

thre

e lo

ans

(all

belo

w th

e cu

toff,

i.e.

, the

mon

etar

y in

terv

al) n

otcl

assi

fied

by th

e ba

nk b

eing

cla

ssifi

ed b

y ex

amin

ers,

and

one

inte

rnal

ly c

lass

ified

loan

bei

ng r

ated

”pa

ss”

by e

xam

iner

s (d

enot

ed b

y am

ount

inpa

rent

hese

s be

low

). T

his

wor

kshe

et r

efle

cts

the

impa

ct o

f the

und

erst

atem

ent a

nd th

e ov

erst

atem

ents

to th

e pr

ecis

ion

fact

or.

Tier

1 C

apita

l + A

LLL:

$ 3

45M

M

Rel

iabi

lity:

90

% P

reci

sion

: 1

0 %

Mon

etar

y Pr

ecis

ion(

MP)

$ 3

4.5M

M M

onet

ary

Inte

rval

(MI):

$ 1

5MM

OC

CR

atin

gB

orro

wer

Loan

Bal

ance

(L)

Sam

ple

Exce

ptio

n(S

E)

Sam

ple

Inte

rval

MI/L

=SI

Cut

off

Loan

sPo

pula

tion

Exce

ptio

nSE

xSI=

PE

Ran

kPr

ecis

ion

Adj

ustm

ent

Fact

or(P

)

Prec

isio

n-ad

just

edEx

cept

ion

PE x

P

$2,1

88$2

,188

6.85

56$1

5,00

01

1.59

$23,

850

$6,0

07$2

,857

2.49

71$

7,1

342

1.44

$10,

273

($3,

850)

($3,

850)

3.89

61($

15,0

00)

1.1

0($

1,5

00)

$ 2

00$

4

075

.000

0$

3,0

003

1.36

$ 4

,808

TOTA

LS$5

,085

$10,

134

$36,

703

RES

ULT

S:Pr

ecis

ion-

adju

sted

Exc

eptio

ns=

$

36,

703

Pr

ecis

ion

Leve

l of (

as %

of T

ier

1 +

ALL

L)=

+$

3

4,50

0

Add

ition

al P

roje

cted

Cla

ssifi

ed L

oans

==

$

71,

203

B

ank

Rep

orte

d C

lass

ified

Loa

ns=

+$

362

,413

Adj

uste

d Pr

ojec

ted

Cla

ssifi

ed L

oans

==

$ 4

33,6

16

THER

EFO

RE:

With

9

0 %

rel

iabi

lity,

we

infe

r th

at a

ctua

l cla

ssifi

ed c

omm

erci

al lo

ans

shou

ld n

ot e

xcee

d 12

6 %

of T

ier

1 C

apita

l plu

s A

LLL.


Numerical Sampling

In our example, the objective of performing a numerical sample on overrideinstalment loans is to determine whether they are safely and soundlyunderwritten and in compliance with any override policy. An exceptionwould be a loan that is not in compliance with policy or not underwrittenprudently.

The same evaluation guidance noted in “Proportional Sampling” above isapplicable to whether exceptions are found or not.

However, examiners use the following numerical evaluation worksheet todetermine the precision-adjusted value of the exceptions.

Numerical Sample Evaluation Worksheet Example

An examiner performed a numerical sample of the instalment loandepartment to determine compliance with the bank’s override underwritingstandards. The sample was designed with a 95 percent reliability and 10percent precision. Two exceptions were found.

Precision (upper limit of the exception rate) P 10%Population size N 1000Reliability factor R 3.0Sample Size n 30Sampling interval I 33Number of errors e 2Sum of the precision adjustment factors = 1.75 + 1.56 = S 3.31

Error Loan PrecisionNumber Number Adjustment

Factors

1 12345 1.75

2 54321 1.56

TOTALS 3.31

33 x (3.0 + 3.31)P' (revised upper precision limit) = = .208

1000


The revised upper precision limit, .208 or 21 percent, exceeds the sampledesign upper precision limit of 10 percent. Therefore, the examiner mustevaluate the errors and attempt to determine their cause.

If this level of exceptions is unacceptable, the examiner should discuss theresults with appropriate bank management and obtain management’scommitment to undertake a review and corrective actions.

Probability Statement

A probability statement is a declaration of statistical assurance. Suchassurance is a combination of precision and reliability. If an exception rate ina population exceeds some material level (precision), the examiner can statewith a certain degree of confidence (i.e., the results have a certain reliability)that the statistical sample will contain a certain number or amount ofexceptions. In other words, results of the statistical sample are used to makeinferences regarding the entire portfolio (population).

The probability statement only applies to the population from which itemswere selected statistically. If examiners eliminated a group of items from thepopulation before the sample selection, they cannot evaluate that group ofitems statistically.

Probability statements cannot be used with nonstatistical judgmentalsampling because the results cannot be statistically related to the entirepopulation.


In our example, where the OCC uses proportional sampling to determineadditional classified loans in a bank’s nonclassified internally ratedcommercial loan portfolio, a probability statement can inform bankmanagement of the estimated maximum amount of classified loans in thatportfolio.

Example with no exceptions: Current internally identified classifiedcommercial loans are 30 percent of Tier 1 capital plus the ALLL. Examinersused an 80 percent reliability level and a 20 percent precision level to reviewnonclassified commercial loans. Sample results disclosed no exceptions. Aprobability statement for this sample would be:

With 80 percent reliability, we infer from our statistical sample that theamount of classified commercial loans will not exceed 50 percent (bank’sidentified 30 percent plus OCC’s precision of 20 percent) of Tier 1 capitalplus the ALLL.


Example with exceptions (see the example in the first worksheet): Currentinternally identified classified commercial loans are 105 percent of Tier 1capital plus the ALLL. Examiners used a 90 percent reliability level and a 10percent precision level to examine nonclassified commercial loans. Fourexceptions were found. After completing the Proportional Loan EvaluationWorksheet, results disclosed the adjusted precision of those exceptions to be17 percent. A probability statement for this sample would be:

With 90 percent reliability, we infer from our statistical sample that theamount of classified commercial loans will not exceed 132 percent of Tier1 capital plus the ALLL (bank’s internally classified 105 percent plus OCCprecision of 10 percent plus adjusted precision value of exceptions 17percent.)

Note: For official purposes in the report of examination and OCC’s electronicinformation systems, the amount and per cent of criticized and classifiedassets should be derived from the bank’s internally identified problem assetsadjusted for OCC findings rather than the statistically extrapolated figures. According to the data in the first worksheet, the amount of classified loans tobe reflected in the report and in OCC electronic data bases would amount tothe more than $362 million identified by the bank plus the $9 millionidentified by OCC examiners as additional classified loans.

Numerical Sampling

When the OCC uses numerical sampling to determine adherence to bankpolicies or controls, or compliance with laws and regulations, examiners canuse a probability statement to inform bank management of the estimatedexception rate in a specified portfolio.

In our example, examiners used numerical sampling to test the accuracy ofthe instalment loan override underwriting standards.

Example with no exceptions: Examiners set the reliability level at 95 percentand precision level at 10 percent. Thirty loans were reviewed and noexceptions discovered. A probability statement for this sample would be:

With 95 percent reliability, our statistical sample results indicate thatexceptions to the override policy in instalment loans do not exceed 10percent of the instalment loan portfolio.

Example with two exceptions: Examiners set the reliability level at 95percent and the precision level at 10 percent. They reviewed 30 loans anddiscovered two exceptions (see the foregoing “Example of a NumericalSample Evaluation Worksheet” ). A probability statement for this samplewould be:


With 95 percent reliability our statistical sample results indicate thatexceptions to the override policy in instalment loans do not exceed 21percent of the instalment loan portfolio.

If the projected level of exceptions is significant or considered unacceptable,examiners should discuss the level with bank management and recommendthat management review the situation, determine the extent of the problem,and implement corrective action.

Report of Examination (ROE) Comments

When examiners use statistical sampling and note significant or unacceptablelevels of exceptions, they should consider including appropriate language inthe applicable ROE comment. The language should include:

C The fact that statistical sampling was used.

C The method of statistical sampling used.

C The number and type of exceptions noted, the underlying root cause ofexceptions, and either suggested improvements or managementcommitments for corrective action.

Examiners should use their own words, rather than statistical language, forreport comments. In addition, report comments must be supported by andbased on a sufficiently large enough sample size (generally a minimum of 30items in keeping with accepted statistical industry standards).

Examples of possible ROE comments, including probability statements:

C Proportional sampling

Currently, bank-identified classified loans in the (describe portfolioreviewed) are (ratio of classified loans to Tier 1 plus the ALLL). Duringour examination we reviewed a proportional statistical sample ofinternally rated, nonclassified (identify portfolio reviewed) loans. Oursample disclosed (number or dollar amount of) exceptions indicating thatadditional unidentified classified loans of percent of Tier 1 capitalplus the ALLL may exist. The causes of the sample exceptions were(reason for the exceptions). We recommend that management (suggestedcorrective action).

Note: For official purposes in the report of examination and OCC’s electronicinformation systems, the amount and percent of criticized and classifiedassets should be derived from the bank’s internally identified problem assetsadjusted for OCC findings rather than the statistically extrapolated figures.


Using the first worksheet’s data, the amount of classified loans to be reflectedin the report and in OCC electronic data bases would amount to the morethan $362 million identified by the bank plus the $9 million identified byOCC examiners as additional classified loans.

C Numerical sampling

During our examination we reviewed a numerical statistical sample of(identify portfolio reviewed) to determine compliance with (describe thespecific feature of interest). Our sample disclosed (number) exceptionsrepresenting percent of the (identify portfolio). The causes of thesample exceptions were (reason for the exceptions). We recommend thatmanagement (suggested corrective action).


Sampling Methodologies Examination Objectives

1. Determine whether sampling is an efficient and effective tool inevaluating areas of examination interest.

2. Select the sampling method that will achieve the objectives of thesupervisory strategy.

3. Use a sample to draw conclusions about the data from which it is taken.


Sampling Methodologies Examination Procedures


Examiners can use these procedures when dollar amounts are of particularimportance and quantifying exceptions or differences in dollars isappropriate, e.g., when sampling the internally rated, nonclassified portionsof a bank’s loan portfolios. 1. Determine sampling objectives by reviewing the approved supervisory

strategy for the bank.

2. Identify and select the population to be reviewed and identify why it isbeing reviewed.

3. Define the feature of interest for which the identified portfolio is beingsampled. For example, in the review of a nonclassified commercial loanportfolio, the feature of interest or exception may be defined as aninternally unidentified classified commercial loan.

4. Identify the desired reliability and precision levels for the sample.

Precision level = %

Reliability level = %

Monetary interval = $

($ Tier 1 Capital + ALLL) x precision as a decimalreliability factor (from appendix C based on reliability %)

If reviewing portions of loan portfolios that the bank has not classified,consider the following:

C Bank loan identification systemS Bank reviews all loans exceeding $

C Internal identified classification level $ %

C Document accuracy of internal identification of past classifications.S How do bank ratings compare with OCC classifications?

C Sample Population $ (Exclude “credit basket” loans and SNC loans if warranted.)


5. Select the sample population. This can be done manually or by using anautomated sampling program.

6. Analyze the selected sample items. When analyzing the nonclassifiedportion of a loan portfolio, consider conducting a minimum file reviewby screening sampled loans as follows:

C Is the loans’s purpose identified?C Is the source of repayment identified and is the loan paying as

agreed?C Is the loan secured by marketable or liquid collateral?C Is the financial statement current and does it support the source or

repayment?C Does the nature of the business warrant further consideration?

If the answers suggest that the loan should be classified, conduct a moreextensive analysis of the loan using traditional methods.

7. Evaluate the sample results (number of sample exceptions )

C If no exceptions are found, the initial reliability and precision levelsare valid.

C If exceptions are found, determine whether they are a result ofinadvertent error or a pattern or practice. Inadvertent exceptionsare still exceptions and should not be excluded from sample results. A pattern or practice determination helps with analysis of thesample but does not affect sample results. Determine, based on thetype and causes of exceptions, whether additional testing iswarranted using the principles of supervision by risk.

C If sample exceptions exist, adjust the initially selected precisionlimit to apply sampling results to the entire population. Seeappendix A for a proportional sample evaluation worksheet that canbe prepared manually or using an automated format.

C If a sample is selected from an entire portfolio (i.e., it includesclassified and nonclassified loans), remember to consider bothoverstatements (loans OCC classified as substandard, doubtful, orloss) and understatements (loans the OCC rates as pass or specialmention and the bank classifies as substandard, doubtful, or loss).

8. Document results in working papers and OCC electronic data bases (ifsignificant).

OCC electronic data base should show:


Bank internally identified classified loans: $ Additional OCC-identified classified loans: $

Enter total in supervisory data asset statisticsapplication $ *

*This figure should not be the statistically extrapolated amount.

9. If warranted, prepare comments for the report of examination, focusingon exceptions and their root causes.

Numerical Sampling

Examiners can use these procedures when sampling loan portfolios or otherbalance sheet accounts to discover whether a defined feature of interest ispresent. Such features include compliance with laws or regulations oradherence to a bank’s underwriting standards.

1. Determine sampling objectives by reviewing the approved supervisorystrategy for the bank.

2. Identify and select the population to be reviewed and why it is beingreviewed.

3. Define the feature of interest for which the portfolio is being sampled(e.g., adherence to the bank’s underwriting standards for extensions inthe instalment loan portfolio.)

4. Identify the reliability and precision levels for the sample.

5. Select the sample population. This can be done manually or by using anautomated format.

6. Analyze the selected sample items.

7. Evaluate the sample results (number of sample exceptions )

C If no sample errors are found, the initial reliability and precisionlevels are valid.

C If exceptions are found, determine whether they are a result ofinadvertent error or a pattern or practice. Inadvertent exceptionsare still exceptions and should not be excluded from sample results. A pattern or practice determination helps with analysis of thesample, but does not affect sample results.

C When sample exceptions exist, adjust the initially selected precisionlimit to project sampling results for the entire population sampled.


Appendix B’s “Numerical Sample Evaluation Worksheet” can beprepared manually to accomplish this. If the sample size is 30 itemsor more, consider formulating a probability statement.

C Compare the revised precision limit with the original precisionlimit. If the revised limit exceeds the original one, evaluate theexceptions and attempt to determine their root cause.

C If the exception level is unacceptable, discuss the results withappropriate bank management and obtain a commitment forcorrective action.

8. Document results in working papers and in the OCC’s electronic database (if significant).

9. If warranted, prepare comments for the report of examination, focusingon the level of exceptions and their root causes.


PRO

POR

TIO

NA

L SA

MPL

E EV

ALU

ATI

ON

WO

RK

SHEE

T

Tier

1 C

apita

l + A

LLL:

$

Rel

iabi

lity:

%

Prec

isio

n:

%M

onet

ary

Inte

rval

(MI):

$

OC

CR

atin

gB

orro

wer

Loan

Bal

ance

(L)

Sam

ple

Exce

ptio

n(S

E)

Sam

ple

Inte

rval

MI/L

=SI

Cut

off

Loan

sPo

pula

tion

Exce

ptio

nSE

xSI=

PE

Ran

kPr

ecis

ion

Adj

ustm

ent

Fact

or(P

)

Prec

isio

n-A

djus

ted

Exce

ptio

nPE

x P

TOTA

LSR

ESU

LTS:

Prec

isio

n-ad

just

ed E

xcep

tions

=$

Pr

ecis

ion

Leve

l of (

as %

of T

ier

1 +

ALL

L)=

+$

Add

ition

al P

roje

cted

Cla

ssifi

ed L

oans

=$

Ban

k R

epor

ted

Cla

ssifi

ed L

oans

=+

$

Adj

uste

d Pr

ojec

ted

Cla

ssifi

ed L

oans

=$

THER

EFO

RE:

With

% r

elia

bilit

y, w

e in

fer

that

act

ual c

lass

ified

loan

s sh

ould

not

exc

eed

% o

f Tie

r 1

capi

tal p

lus

ALL

L.

Sampling Methodologies Appendix A


Proportional Sampling Evaluation Worksheet

Instructions

If at least one exception is found, revise the statistical assurance based onsample findings. Raise the precision limits. Do this by completing thefollowing:

OCC Rating: Enter OCC rating (substandard, doubtful, or loss).

Borrower: For exceptions, enter borrower’s name in Borrowercolumn.

Balance: Enter loan balance in the Balance (L) column.

Sample Enter the dollar amount of the sample exception (SE). Exception: This amount may be different from “Balance” when only

a portion of the loan is classified. Sample exceptions canbe overstatements or understatements. An overstatementis an OCC-classified loan. An understatement is a bank-classified loan that the OCC does not classify. Understatements occur only when the OCC conducts astatistical review of internally classified loans.

Sampling Enter the sampling intervals (SI) for each exception. TheInterval: sampling interval is calculated by dividing the monetary

interval by the loan balance (MI/L). The sampling intervalis the probability of selecting a loan balance (L) as one ofevery SI balances of like amount. The sampling intervalapplies the level of sampling exceptions observed to thepopulation. Sample loan exceptions exceeding the cutoffare always selected; their sampling interval is set at 1.0.

Cutoff Enter the dollar amount of loans equal to or exceeding theLoans: monetary intervals which are exceptions (CL). Because

these loans are selected with certainty, the estimatedpopulation exception is the same as the sampleexception.

Population Obtain the value in this column by multiplying theException: sample exception (overstatement or understatement loans)

by its probability as indicated in the sampling intervalcolumn (SE x SI = PE). This statistically projects the valueof the sample exception to the population. Recordoverstatements (or understatements) as appropriate. (NOTE: Loans that are full exceptions will have themonetary interval as its population exception.)


Rank: Rank the population exceptions separately. Theexceptions should be ranked in descending order ofamounts because of the bias given to larger dollar loansamples in proportional sampling. When exceptions havethe same values, rank each separately. Loans exceedingthe cutoff are not ranked.

Precision: Enter the adjustment factors for overstatements (appendixC) or understatements (appendix D) based on thereliability of the sample. These factors are used toconvert the estimated exceptions in the population intoprecision-adjusted exceptions. No precision adjustmentfactor is required for cutoff loans.

Precision Enter the dollar value of each cutoff loan. For the otherAdjustment sample exceptions, multiply the population exception byExceptions: its precision adjustment factor to yield the precision-

adjusted exceptions. The overstatement calculations haveincreased (and the understatement calculations havedecreased) the precision limit based on the sampleexceptions so that the same reliability level can be usedto evaluate the sample.

The total of this column and the monetary precision (Tier1 capital and ALLL x precision percent) represents thepotential additional amount of classified loans in theportfolio. This figure is added to the known classifiedloan totals to estimate the upper limit of classifications.

A probability statement such as those in this booklet’ssection “Probability Statement” can then be made usingthis data.

Additional Evaluation:

Examiners analyze and evaluate the sample exceptions to determine theroot cause and decide whether the sample exceptions are a pattern orpractice.

Consider:C Inexperience of a loan officer.C An exception by type, division, or industry.C Differences by individuals assigning ratings in adhering to

internal classification definitions.C Timeliness of bank rating changes.


Sampling Methodologies Appendix B

NUMERICAL SAMPLE EVALUATION WORKSHEET

Sample DesignPrecision P Population size N Reliability factor (From appendix C, 2.3 for 90%, 3.0 for 95%) R

Sample size = R n P

Sampling interval = N I n

Sample Evaluation Number of errors found e Sum of the precision adjustment factors S

ERROR LOAN ADJUSTMENTNUMBER NUMBER FACTOR

PRECISION

(Fromappendix C)

TOTALS

Revised upper precision limit P' = I(R + S) N


Sampling Methodologies Appendix C

Statistical Sampling Overstatements (OCC classified, bank nonclassified)

Reliability Factors (R) 1.6 2.0 2.3 3.0

Reliability Levels 80% 86% 90% 95%

Rank of Errors1 Precision Adjustment Factors (P) For Evaluating Samples at Above Levels

1 1.39 1.51 1.59 1.75

2 1.28 1.38 1.44 1.56

3 1.24 1.31 1.36 1.46

4 1.21 1.27 1.32 1.40

5 1.19 1.25 1.29 1.36

6 1.17 1.23 1.26 1.33

7 1.16 1.21 1.24 1.31

8 1.15 1.20 1.23 1.29

9 1.14 1.19 1.22 1.28

10 1.14 1.18 1.21 1.26

11 1.13 1.17 1.20 1.25

12 1.13 1.16 1.19 1.24

13 1.12 1.16 1.18 1.23

14 1.12 1.15 1.18 1.22

15-19 1.11 1.15 1.17 1.22

20-24 1.10 1.13 1.15 1.19

25-29 1.09 1.01 1.13 1.17

30-39 1.08 1.00 1.12 1.15

40-49 1.07 1.09 1.10 1.13

50-74 1.06 1.08 1.09 1.12

75-99 1.05 1.07 1.08 1.10

100 and Over 1.04 1.06 1.07 1.09

1 Errors of overstatement should be ranked separately from those of understatement, and withineach group the ranking should be from the largest to the smallest amount of error.


Sampling Methodologies Appendix D

Statistical Sampling Understatements (OCC nonclassified, bank classified)

Reliability Factors (R) 1.6 2.0 2.3 3.0

Reliability Levels 80% 86% 90% 95%

Rank of Errors1 Precision Adjustment Factors (P) For Evaluating Samples at Above Levels

1 .22 .14 .10 .05

2 .60 .49 .42 .30

3 .71 .62 .57 .46

4 .76 .69 .64 .54

5 .79 .73 .68 .60

6 .81 .75 .71 .64

7 .83 .77 .74 .67

8 .84 .79 .76 .69

9 .85 .80 .77 .71

10 .86 .81 .78 .73

11 .86 .82 .79 .74

12 .87 .83 .80 .75

13 .87 .84 .81 .76

14 .88 .84 .82 .77

15-19 .88 .85 .83 .78

20-24 .90 .87 85 .81

25-29 .91 .89 .87 .83

30-39 .92 .90 .88 .85

40-49 .93 .91 .90 .87

50-74 .94 .92 .91 .88

75-99 .95 .93 .92 .90

100 and Over .96 .94 .93 .91

1 The distinction between errors of overstatement and understatement should be based on theireffect on the population from which the sample was drawn. If this population is reciprocal tothe one of primary interest, errors will have an opposite effect on the population of primaryinterest.


Sampling Methodologies Glossary

Adjusted precision level is a recalculated statistical assurance based on thenumber or monetary amount of tolerable errors noted. If errors are found in asample, precision must be revised to maintain the original degree ofreliability set by the sample plan.

Compliance exception or error is a deviation from established policies,procedures, or practices, i.e., internal control. Examples are noncompliancewith loan underwriting standards (such as exceeding approved officer lendinglimits or failure to obtain required documentation) and violations of laws orregulations (such as incomplete Truth-in-Lending or Truth-in-Savingsdisclosure statements).

Exception in sampling is an error or deviation in a feature of interest. Exceptions may be monetary (dollar amount) or compliance(nonconformance with policies, procedures, or practices or violations of lawor regulation). For example, in a proportional sample of a bank’s commercialloan portfolio, an exception is a loan that the bank does not classify internallybut the examiners do. In a numerical sample of an instalment loan portfolio,an exception could be a loan that is not in compliance with the bank'sunderwriting standards or extension, renewal, or override policies (or it couldbe a loan that is not underwritten in a safe and sound manner).

Feature of interest is the characteristic for which the sample is being taken,e.g., compliance with a regulation or adherence to a bank’s policy (such asinstalment loan overrides). Features of interest are defined by the objective ofthe review and commonly are represented as exceptions.

Monetary exception or error causes a dollar amount to be inaccurate. Forexample, a loan classified by examiners but passed by a bank is a monetaryerror because it affects the accuracy of the bank’s amount of classified assets.

Monetary interval is the dollar measurement used in a proportional sampleto select sample items. All items equal to or greater than the dollar amount ofthe monetary interval will be selected for review.

Nonstatistical or judgmental sampling is the selection of items based on thejudgment of an individual. Results of a judgmental sample cannot be used todraw statistically valid inferences about a population.


Numerical statistical sampling is the selection of sample items based on thenumber of items in a population. No item has a greater probability of beingselected than any other and the dollar amount of items is not relevant tosampling objectives (in contrast to proportional statistical sampling). Selection begins from a random starting point and proceeds in a constantmeasured interval between the selected items.

Poisson distribution expresses the mathematical probabilities that anexpected number of exceptions will occur in a selected sample of items froma population in relation to specific confidence intervals (precision andreliability). Appendixes C and D are quick references for parametersassociated with the distribution probabilities and precision and reliabilitylevels.

Population is a whole group of related items from which a sample is drawn. For example, all of a bank’s internally rated nonclassified notes in itscommercial loan portfolio may be the population from which an examinerdraws a sample.

Precision, precision limit, or precision level is the amount that sample resultscan deviate from the most likely results of a review of the entire populationand still be acceptable to the examiner. It can be viewed as the OCC’stolerance for the error inherent in not analyzing each item in the population. The OCC allows some imprecision in drawing sample conclusions to avoidanalyzing each item in the population. The lower the tolerance for error, thelarger the number of items that need to be selected in a sample.

Probability statement is a statement of statistical assurance. Using results ofa statistical sample, examiners can draw conclusions about the entireportfolio from which the sample was taken. The probability statementexpresses the examiner’s degree of confidence that the sample can stand forthe entire population.

Proportional statistical sampling is the selection, based on monetaryamounts, of items from a population. Larger items have a greater probabilityof being selected than smaller ones and the dollar amount of items is relevantto sampling objectives (in contrast to numerical statistical sampling). Anitem’s probability of being selected is proportional to its monetary amountrelative to an established monetary interval.

Reliability or reliability level is the probability that the value of the feature ofinterest in the sample is representative of the entire population, i.e., withinthe desired precision level. Reliability is a reflection of the degree ofconfidence an examiner has in the sample results and how much risk ofimprecision she or he is willing to accept.


Reliability factor is a mathematically calculated approximation of theprobability of finding at least one exception in a sample of selected items at agiven reliability level and error rate. Reliability factors are used to determinesample sizes and precision adjustment factors. They are common referencesfor audit sampling purposes. See appendixes C and D.

Sample, sample items or sample population is the group of items selected,using a sampling method, from a larger general population.

Sample interval is the constant measured interval between items selected fora sample. This interval can be every nth item, e.g., every 10th item. Intervalscan also be a set distance or position between items or the terminal digits ofpopulation items.

Sample plan is the process of setting objectives for the sample, selecting thepopulation to be sampled, designing and selecting the sample, and evaluatingand interpreting sample results.

Sampling risk is the risk that the sample is not representative of the entirepopulation. Sampling risk is determined by a formula — 1 minus thereliability level as a decimal. For example, with 90 percent reliability, thesampling risk is 10 percent (1 - .90 = .10). This means that 10 percent of thetime, or one time in ten, the results of the sample may not be indicative of theentire portfolio.

Statistical assurance, a product of precision and reliability, is the measure ofreliance an examiner places on inferences drawn using the sample. It iscommonly expressed in a probability statement.

Statistical or sample projection uses probability theory to apply sampleresults to the entire population sampled and depends on a random selectionprocess. To understand probability theory, examiners must understandreliability and precision, as well as their interrelationship.

Documents

Sampling Methodologies