W W W . C H I C A G O L A N D R I S K F O R U M . O R GW W W . C H I C A G O L A N D R I S K F O R U M . O R G

What’s New in Risk Assessment?

22

33

Risk Management Depends on Risk AssessmentThe simplest definition of Risk Management involves 3 steps:

55

Risk Assessment Has Many Moving Parts!

66

Strategic Risk Assessment: What is Important to Achieving Organizational Objectives andNot Under [Complete] Control?• Identifying threats and exposures without measurement only

generates lists -- that may or may not be applicable or important to the organization.

• Some ERM projects create spreadsheets full of “Critical Risks” that frustrate management and fail to provide a blueprint for action.

• Instead of identification run rampant, Strategic Risk Assessment starts with corporate objectives and considers what is at risk, identifies potential threats, and assesses the impact and the effectiveness of current controls to counter those threats – and points to controls where objectives are threatened.

77

Strategic Risk Assessment IssuesTo be effective, risk assessment cannot be merely checklists or a process that is disconnected from business strategy. • Risk Assessment must be integrated in a way that provides

timely and relevant risk information to management. • For risk management to be a strategic process, risk assessment

must be owned by the business units and be embedded within the business cycle, starting with strategic planning.

• And: Risk assessment begins and ends with the organization’s specific objectives.

8

Strategic Risk Assessment

Qualitative AnalysisRisk Register ◄

Risk Map ◄Risk Categorization ◄

Quantitative Analysis► Decision Tree Analysis► Scenario Analysis► FMEA► Simulation & Modeling

99

Risk Assessment Basics• It is a matter of widespread understanding that risks should be

assessed in terms of the likelihood (probability) that an uncontrolled event will occur and the consequences (impact) to achieving one or more organizational objectives.– Applicable to both qualitative and quantitative methods of

assessment.

• Strategic Risk Assessment requires pursuing a systematic, logical set of actions to identify the magnitude of hazards and exposures, assess threats, and implement controls to mitigate, eliminate or control high-risk conditions.

Risk Maps Are Primarily Qualitative Assessments

Qualitative Methods & Risk Maps Highlight Critical Threats

Data gathering & representation

Select appropriate techniques(s)

Risk analysis & modeling Expert judgment

RISK

But Quantitative Methods are Often Required to Identify Corrective Actions

1313

Risk Assessment Tools & Techniques Are Rapidly EvolvingRisk Assessment needs to move beyond Probability x Severity and Risk Maps to evaluate emerging issues, warning & detectability, and other key threats to strategic objectives.• Over the past decade, developments in economic and

financial theory -- plus computing and data advancements –are providing new methods for quantitative risk assessment, as well as improvements to existing techniques.

• Risk Managers should understand available risk assessment techniques and adopt a set of tools they can apply to their organization's unique Risk Management requirements.

1414

Three Basic Types of Quantitative Assessment Tools – In Order of Complexity

1.Comparative methods; 2.Temporal methods; and,3.Functional methods.

1515

Comparative Assessment MethodsA Comparative Analysis takes an explicit standard – eg., “Best Practices”– and compares a system, process and/or set of procedures to that standard, resulting in a “Gap Analysis”. • A “good standard” is prepared and maintained as “the distillation of

continually developing expert opinion and experience in the face of a continually changing environment”.

• One of the strengths of this approach is its simplicity. Comparative methods can be ideal for organizations as they begin to focus attention on specific systems, processes or threats.

• A weakness is that there is no explicit list of threats as there is in other approaches.

1616

Sample “Best Practices” Matrix –Claims Handling

O Managerial OversightP PrimaryS SecondaryC Consultative InputD Data Resource

Dire

ctor o

f Ins

uran

ce

Dire

ctor o

f Leg

al Su

ppor

t &

Claim

s

Exec

utive

Vice

Pres

ident

, Aon

Senio

r Vice

Pres

ident

, Clai

ms

Vice

Pres

ident

, Clai

ms

Assis

tant

VP,

Claim

s

Senio

r Con

sulta

nt, C

laim

s

Senio

r Clie

nt Sp

ecial

ist,

Claim

s (M

egan

)

Senio

r Clie

nt Sp

ecial

ist,

Claim

s (M

arth

a)

Claim

Ass

istan

t

CLAIM MANAGEMENT PROCEDURES

1)Establish formal claims service standards for TPA's, carriers and other vendors C P C P C C

2)Develop annual written service plan for TPA's and other vendors and monitor performance C P O P C C

3) Develop written Claims Procedures or Manual C O C P C C

4)Establish internal claims reporting and management procedures and monitor compliance C O C O P S

5)Develop claim reports, distribute and review with business units as necessary C O C O C P S S S D

6) Maintain listing of all insured claims O O O C P S S S D7) Maintain listing of all self-insured claims O O O C P S S S D8) Establish and monitor WC post-injury management program C O O O D P S9) Manage claims litigation process C O O P D S D10) Administer OCIP claims C O O O P S11) Administer non-litigated GL claims O O O P S S D D12) Administer auto claims O O O P S S D D13) Administer D&O, fidelity, fiduciary, EPL C P C14) Administer Litigated GL claims O O P D S D15) Administer Property claims O O O P D S D16) Pursue subrogation activities O O O P S S S D17) Review losses and identify trends C C C O C P S S S D18) Conduct/coordinate periodic claims audits D D19) Monitor large loss activity C O O P D D

20)

Review and adjust safety/loss control initiatives as needed to proactively treat risk and address trends observed in claims management activities

O C C C D C D

Management assures activ ity is addressed

Principally responsible for driv ing the activ ity

Responsible to perform or drive certain aspects of the activ ity , but is not the leader

Can prov ide guidance or feedback at a high level for activ ity

Prov ides data or information that is used in the activ ity

1717

Sample “Best Practices” Gap Analysis –RM Strategy

1818

Temporal Analysis MethodsA Temporal Assessment applies quantitative tests to a system, process or set of procedures. These “tests” involve analyzing the results of specific threats or attacks against actual protections and controls, subject to some constraints.• Since it is often impractical to test a system directly, a model of the

system is generally used instead. – However, a model introduces the question of fidelity: an

inaccurate model may not only confuse matters; it may provide a false sense of security that is even worse than confusion.

• A key weakness of a temporal method is that it is not possible to model all possible threats; it is not even possible to list them all.

1919

Temporal Method: Scenario AnalysisScenario analysis considers the questions ‘what might happen and what should/would we do?’ It can not only highlight risks and opportunities in the short and long term; but also test the effectiveness and efficiency of specific controls and plans.• The central idea is to consider a variety of possible futures that

include many of the important uncertainties in the system, rather than to focus on the accurate prediction of any particular outcome.

• A strength of scenario analysis is that it can consider “existential threats” that involve large swaths of the organization.

2020

Four Critical Components of Scenario Analysis1. Determining which factors the scenarios will be built around. In

general, analysts should focus on the two or three most critical factors.

2. Determining the number of scenarios to analyze for each factor. Depends upon how different the scenarios are, and how well the results of each scenario can be forecast.

3. Estimating results – e.g., asset cash flows, control failures, unexpected breakdowns, etc. -- under each scenario.

4. Assigning probabilities to each scenario. Note that this makes sense only if the scenarios cover the full spectrum of possibilities; otherwise, the probabilities will not add up to 100%

2121

Sample ScenarioA Scenario Analysis can be used to ensure effective and reliable insurance coverage. • It typically involves sitting down with brokers, underwriters, lawyers,

adjusters and managers to analyze and talk through how each insurance policy would respond to different circumstances.

• The results are compiled in systematic tables and charts that point out problem areas and suggest solutions.

• One of the strengths of Scenario Analysis is that it tests the system itself (or a model), clearing away misconceptions and uncovering specific elements or issues needing attention.

2222

Other Temporal Analysis MethodsThe most important Temporal Assessment methods use Predictive Analytics to not only determine What might happen, but How Much it could impact objectives.• Two useful tools are:

– Decision Tree Analysis; and,– Modeling & Simulation.

2323

Decision Tree AnalysisA Decision Tree is a structure in which each internal node represents a "test" on an attribute; each “branch” represents the outcome of the test; and each “leaf” represents a decision taken after computing all attributes. • The paths from root to leaf represent classification rules:

– A Root node represents the start of the decision tree, where a decision maker is faced with an uncertain outcome. The objective is to evaluate the overall net positive or negative outcomes at this node.

– Event nodes represent outcomes based upon the probable occurrence of various events.

– Decision branches represent choices that are made by the decision maker.

– End nodes represent final outcomes where a payoff value is identified.

2424

Sample Decision Tree: Jenny Lind• Jenny Lind is a writer of romance novels. A movie company

and a TV network both want exclusive rights to one of her more popular works.

• If she signs with the network, she will receive a single lump sum, but if she signs with the movie company, the amount she will receive depends on the market response to her movie.

• What should she do?

Jenny Lind Decision TreeSmall Box Office

Medium Box Office

Large Box Office

Small Box Office

Medium Box Office

Large Box Office

Sign with Movie Co.

Sign with TV Network

$200,000

$1,000,000

$3,000,000

$900,000

$900,000

$900,000

.3

.6

.1

.3

.6

.1

Estimated OutcomesEstimated

Likelihood

Root Node

Event Nodes

Jenny Lind Decision Tree - SolvedSmall Box Office

Medium Box Office

Large Box Office

Small Box Office

Medium Box Office

Large Box Office

Sign with Movie Co.

Sign with TV Network

$200,000

$1,000,000

$3,000,000

$900,000

$900,000

$900,000

.3

.6

.1

.3

.6

.1

Expected$900,000

Expected$960,000

Best Result$960,000

2727

Modeling & SimulationWhere Scenario Analysis and Decision Tree Analysis are techniques to assess discrete risk events, simulation methods measure continuous risk exposures and outcomes.• Simulations yield a distribution of outcomes rather than a single point

estimate. • One simulation tool is an “Exceedance Probability Curve” that

measures whether an outcome will exceed a specific estimate, based upon predetermined probabilities.

• Simulation has few limitations in terms of events, probabilities and outcomes – very robust models may be constructed, evaluated and displayed graphically.

2828

Simulation Example: Quantifying the Risk of Natural CatastrophesHow do companies prepare for the financial impact of natural catastrophes? How can they possibly have an idea of what the potential cost can be for events that haven't yet happened?Catastrophe Modeling provides the answers. A catastrophe model can be roughly divided into three modules:• The Hazard Module looks at the physical characteristics of potential

disasters and their frequency.• The Vulnerability Module assesses the vulnerability (or

“damageability”) of buildings and their contents.• The Damage Module determines the overall loss distribution for a

specific event by multiplying building values by potential damage.

Sample Catastrophe Model Results

3030

Functional Assessment MethodsA Functional Analysis focuses on specific threats and protections. • A threat model -- a list of system vulnerabilities, and the likelihood of

successful threats against those vulnerabilities -- is weighed against organizational objectives, assets, protections, and the likelihood of available protections successfully defending those assets against specified threats.

• Temporal Assessment methods, such as statistical modeling; and Comparative Assessment techniques, such as expert systems, are often employed jointly.

• The key strength of a Functional Assessment is its ability to consider a wide range of threats, vulnerabilities, assets and countermeasures.

3131

Failure Mode & Effects Analysis (FMEA)FMEA identifies where & how failures can occur within processes and measures the impact of those failures.• The FMEA Process has 4 basic steps:

1. Determine the failure modes of specific process elements; 2. Analyze the effects on other elements and the overall system; 3. Rank criticality; and,4. Identify existing and potential controls.

• FMEA is particularly useful for evaluating critical risks in very complex systems.

FMEA Thought Process

3333

Sample FMEA TemplateAction Results

Item / FunctionPotential Failure

Mode(s)

Potential Effect(s) of Failure

Sev

Potential Cause(s)/

Mechanism(s) of Failure

Prob

Current Design Controls

Det

RPN

Recommended Action(s)

Responsibility & Target

Completion DateActions Taken

New

Sev

New

Occ

New

Det

New

RP

N

Coolant containment. Hose connection. Coolant fill. M

Crack/break. Burst. Side wall flex. Bad seal. Poor hose rete

Leak 8 Over pressure 8 Burst, validation pressure cycle.

1 64 Test included in prototype and production validation testing.

J.P. Aguire 11/1/95 E. Eglin 8/1/96

Response Plans and Tracking

Risk Priority Number - The combined weighting of Severity, Likelihood, and Detectability.RPN = Sev X Occ X Det

Likelihood - Write down the potential cause(s), and on a scale of 1-10, rate the Likelihood of each failure (10= most likely). See

Severity - On a scale of 1-10, rate the Severity of each failure (10= most severe). See Severity

Detectability - Examine the current design, then, on a scale of 1-10, rate the Detectability of each failure(10 = least detectable). See Detectability sheet.

Write down each failure mode and potential consequence(s) of that

FMEA Path Model Example

3535

FMEA Technique: Fault Tree Analysis• A Fault Tree is a logical diagram that starts with an actual or

potential failure and works backward to identify all of the possible causes or origins of that failure.

• Made up of branches connected by AND nodes and ORnodes.– ALL of the branches below an AND node must occur for the

event above the node to occur.– Only ONE of the branches below an OR node needs to occur for

the event above the node to occur

3636

Fault Tree Example

Identified “Fault”

Both Required

Any of These

Required

3737

FMEA Technique: Event Tree Analysis• An Event Tree is a logical diagram that starts with an actual or

potential event and works forward to identify all of the possible corrective actions -- and failures that could result.

• Essentially the reverse of a Fault Tree; in an analysis, one Event Tree may lead to multiple Fault Trees and vice-versa.

• Although initially developed by engineers to determine vulnerabilities in nuclear power generators; it is applicable, and has been applied, to assess many complex processes.

.302

.034

.084

.180

.400

3939

Summary – Strategic Risk AssessmentVarious strategic risk assessment methods view the landscape from different heights, so to speak -- altitude is a tradeoff between scope and detail.• The more abstract the method, the greater the scope but the

coarser the detail; the more concrete the method, the smaller the scope and the finer the detail.

• Different objectives, systems, threats, perils, hazards, controls, etc. dictate the use of different assessment tools and methods.

• Identifying the appropriate technique should be the first – and most important – step in risk assessment.

4040

And, Don’t Forget – the Real Objective is to Manage Risk

• The techniques examined in this discussion should only be used when you need to identify exposures, risks, perils and/or hazards that can be eliminated, mitigated or otherwise managed.

• NO measurement is necessary when you KNOW what to DO – and everyone AGREES!

QUESTIONS?

Thank you very much for listening!

Backup

4444

Categorizing Risk Assessment Techniques• Three basic types of assessment tools are:

1. Temporal methods;2. Comparative methods; and,3. Functional methods.

• Assessment techniques and tools can be classified on three axes:1. by their level of formality on a continuum from abstract to

concrete; 2. the type of analysis performed; and 3. the threats they are attempting to find and address.

4545

Types of Temporal Assessment Methods

• An Engagement consists of experts looking for any way, within given bounds, to compromise assets.

• An Exercise links experts and owners together in order to test the protection on assets particular to a particular system.

• Compliance Testing includes methods that the owner can execute them himself without the aid of an expert.

4646

Types of Comparative Assessment Methods• A Principles Method type, like all of the Comparative types, is a list.

This type asks the user to apply the principles to their system. • A Best Practices list consists of directives: Do this, Don’t do that. This

method type asks the user to compare what they do—their current practice—with the best practice list: the list of differences represents the “Gaps” between actual practices and ideal.

• An Audit is based on an explicit standard, such as a Best Practice list or a Principles list. This type asks the user to evaluate the effectiveness of the controls in place in fulfilling each item in the standard.

4747

Types of Functional Assessment Methods• Sequence Methods are the epitome of abstract methods. A simple

sequence method asks the questions:1. What can happen? (i.e., What can go wrong?)2. How likely is [it] that that will happen?3. If it does happen, what are the consequences?

• An Assistant Method type keeps track of details; best instances of this type “walk” the user through the process, prompting for the input needed to populate and rank lists of threats, vulnerabilities and remedial actions.

• A Matrix Method asks the user to select ranges for n dimensions – assets, threats, vulnerabilities and protections. The information in the cells of the corresponding n-dimensional subspace is the result of analysis.

• An Expert System is one implementation that is representative of the functional approach.