Risk based decision making - Rev 4

Supplier RisksRisk Based Decision Making

Julie M. RodriguesGlobal Manager of Supplier

Development

Introduction What is Risk? Risk Assessment Methods Risk Matrix Development Integrated Risk Based Decision Making Questions

Agenda

What are we going to do today?

Risk in Supplier Quality systems Assessment methods Some ways to apply risk assessment to supplier

quality systems◦ Easy◦ Efficient◦ In alignment with organizational needs

Introduction

Quality System Standards◦ ISO 9001:2008◦ ISO 13485:2003◦ ISO 14971:2007

Regulatory standard◦ 21 CFR 820

Guidance Documents

Global Harmonization Task Force (GHTF) Documents◦ SG3/N17/2008, Quality Management System – Medical

Devices – Guidance on the Control of Products and Services Obtained from Suppliers

◦ SG3/N15R8/2005 , Implementation of Risk Management Principles and Activities within a QMS

◦ SG4/N30R20/2006, Guidelines for Regulatory Auditing of QMS of Medical Device Manufacturers, Part 2, Regulatory Auditing Strategy

Guidance Documents (cont.)


Agenda

Product or Service is an item, tangible or intangible, which is purchased or otherwise obtained by the manufacturer. - A Product is the result of a process. - Service is the result of at least one activity necessarily performed at the interface between the supplier and customer and is generally intangible (ISO 9000:2005, Clause 3.4.2)

A Supplier is anyone that is independent from the manufacturer’s quality management system from whom a Product or Service is purchased. (ISO 9000:2005, Clause 3.3.6)

Definitions (from GHTF)

Risk is the combination of the probability of occurrence of harm and the severity of that harm. (ISO/IEC Guide 51:1999, definition 3.2)

What can go wrong? How bad is it? How likely is it to happen?

What is Risk?

When identifying risk:

◦ Do it to the best of our ability! - don’t do this in isolation

◦ Removed from the emotional impact◦ Without being alarmist: Cassandra Syndrome◦ With a value meaningful to the organization

Identifying Risk

Risk Assessment is the overall process comprising a Risk Analysis [identification of hazards and estimate of risk] and a Risk Evaluation [judgment as to whether a risk is acceptable]. (ISO/IEC Guide 51:1999, definition 3.12)

Process that identifies:What is the risk level?Based on the risk level, what happens?

What is Risk Assessment?

Supplier Risk: ◦ Risk to the organization resulting from a supplier

component, service, or process ◦ Risk associated with Supplier Quality Processes.

This includes◦Selection◦Evaluation◦Control◦Re-Evaluation

What is Supplier Risk

Not hard at all! ◦We do it all the time, within and without our

QMS

◦Examples: Who should I select as my primary care

physician? What brand of tires should I use? Should my next car be foreign or domestic? What cereal should I buy? Should I buy organic fruits and vegetables?

How hard is this really?

Why don’t we do this already?◦ We do! Intuitively

So what is the problem?◦ Consistency◦ Objective evidence – auditor wants to know why◦ Documented activities◦ Buy in from the entire organization

Not Difficult!


Agenda

What are some desirable characteristics of a Risk Assessment?◦ Consistent◦ Easy to use and easy to understand◦ Provides objective evidence◦ Flexible◦ Subsequent activities can be based on results◦ Single assessment to satisfy all Supplier Quality

activities

Characteristics

Risk Assessment = Decision Making

We can use the same toolset

How?

There are lots of decision making tools.◦Build consensus through discussion◦Decision Tree◦Decision Table◦Decision Matrix

Each method has pros and cons

Decision Making Tools

Technique:- Discuss the options- Use good meeting practices to formulate options and achieve agreement

Method: Building Consensus

Pros• Easy• Familiar• No training required• Meeting minutes

serve as documentation

Cons• Lack of consistency• May take a while• Emotional

Technique:- Follow decision tree- Use good meeting practices to formulate options and achieve agreement

Method: Decision Tree

Pros• Single path• Minimal training• Sequential, some paths

eliminate options• Used frequently in

medical field

Cons• Can be complicated• May not be flexible,

limited to existing path• Decision process can be

emotional

Decision Tree, Example

From GHTF/SG3/N15R8

Technique:- Search for appropriate action in the table- Use good meeting practices to formulate options and achieve agreement

Method: Decision Table

Pros• More flexible• Non-linear• Visual

Cons• Training required,

complex• More options than needed• Decision process can be

inconsistent

Decision Table, Example

From GHTF/SG3/N15R8

Technique:- Combines decision tree and decision table methodologies- Answer questions, calculate weighted sums, look up result in table.

Method: Decision Matrix

Pros• Fact based decisions• Minimal training to use• Consistent results• Tool itself is documentation• Can be automated

Cons• Initial setup is complex• Calculations can be tedious

if not automated

Decision Matrix, Example

Material #:

Supplier Type Rati

ng

Complexity Rati

ng

1 - Fully Certified 1 - Simple, not critical (DIN)2 - Existing Supplier 2 - Nominal difficulty3 - New Supplier 3 - Complex/Critical

Similarity of parts Exposure1 - Modification of existing part 1 - Inside Device2 - Family part with characteristic differences 2 - Exposed to operating environment3 - No part similarities 3 - Patient contact/Critical Component

Delivery (last 12 weeks) Cosmetic1 - 98-100 1 - None/Internal2 - 90-97 2 - Process variable3 - <=89 3 - High visibility

Quality (last 12 weeks) Replacement Risk1 - 98-100 1 - Easily replaceable/Minimal rework2 - 90-97 2 - Reworkable3 - <=89 3 - Difficult or no rework/Must replace

Qualification Performance Detectable 1 - 5 of last 5 complete 1 - Visibly detectable2 - 4 of last 5 complete 2 - Detectable in assembly3 - <= 3 of last 5 complete 3 - Not detectable

Total 20 Total 27

4 - 11 = low, 12 - 24 = Nom, 25 - 36 = High Overall Risk Matrix

Overall Risk Assessment High

1 2

3 2

3 2

Risk Assessment123456

Supply Chain Risk Quality Risk

Nom

inal

Hig

h

2 3

3 3

Each process has pros/cons Selection will depend on the organization

KSE uses the Decision Matrix model

There is no “right” or “wrong” method


Agenda

Internationally recognized organization that trains management in rational decision making.

Originally part of the Rand Corporation Integral in the development of 8D process and

other problem solving tools

Kepner-Trego Decision Making

Steps:◦ Establish decision criteria◦ Give each criteria a weight (numerical)◦ Score each option against the criteria◦ Weighted sum calculation establishes “best”

solution

Decision Making

Speed( 5 )

(1-10)

Cost( 3 )

(1-10)

Effectiveness( 9 )

(1-10)

Weighted Sum

200% Inspection 9 6 1

(9*5)+(6*3)+(1*9)= 72

Simple EP 5 5 8(5*5)+(5*3)+(8*9)=112

Vision System 2 1 10

(2*5)+(1*3)+(10*9)=

103

Example:

9*5=45 6*3=18 1*9=45

5*5=25 5*3=15 8*9=72

2*5=10 1*3=3 10*9=90

45+18+45 =108

25+15+72 =112

10+ 3+90 =103

Leverage the KT decision making approach Team based development

◦ QA◦ Purchasing◦ Manufacturing Engineering◦ Manufacturing◦ R&D

Advantages:◦ Tough decisions are made up front◦ Buy-in and comprehension across the organization◦ Can be automated in Excel

Risk Assessment team

Select Risk Decision Scale◦Low (1), Nominal (2), High (3)

All decisions in the matrix made using this scale

Why? ◦Typical of decision processes already used in

the organization (green/yellow/red)

Weight/Scale selection

Split into 2 decision path, and combine the results◦ QA◦ Supply chain

Why?◦ Both organizations need something from the

system◦ The needs sometimes conflict, but are real

Decision Paths

Select Options

Purchasing – Supplier Capability◦ Low: Certified Supplier (1)◦ Nominal: Existing approved supplier (2)◦ High: New supplier (3)

Options

Quality – Part Complexity◦ Low: Off the shelf/DIN/ISO component (1)◦ Nominal: No particular difficulty, special process or

other complication (2)◦ High: Complex component or assembly, or a critical

component (3)

Options (cont.)

Select Decision Criteria (Low/Medium/High risk):

Similarity of Parts◦ Modification of an existing part made by the supplier◦ One of a family of similar parts◦ No part similarities

Delivery Score◦ 98-100◦ 90-97◦ <89

Purchasing Criteria

Quality Score◦ 98-100◦ 90-97◦ <89

Success in completing qualification◦ Completed 5 of 5 most recent qualifications◦ Completed 4 of 5 most recent qualifications◦ Completed 3 or fewer of most recent 5 qualifications

Purchasing Criteria (cont.)

Purchasing Risk Matrix

Supplier Type Rati

ng

Complexity1 - Fully Certified 1 - Simple, not critical (DIN)2 - Existing Supplier 2 - Nominal difficulty3 - New Supplier 3 - Complex/Critical


Delivery (last 12 weeks) Cosmetic1 - 98-100 1 - None/Internal2 - 90-97 2 - Process variable3 - <=89 3 - KST acceptable

Quality (last 12 weeks) Replacement Risk1 - 98-100 1 - Easily replaceable/Minimal rework2 - 90-97 2 - Reworkable3 - <=89 3 - No rework/Must replace

RIS Performance Detectable 1 - 5 of last 5 complete 1 - Visibly detectable2 - 4 of last 5 complete 2 - Detectable in assembly3 - <= 3 of last 5 complete 3 - Not detectable

Total 16 Total

1

3

3


Nom

inal

2

1

Options

Criteria

CalculatedRisk

Purchasing Risk4 - 11 = Low 12 - 24 = Nom 25 - 36 = High

Exposure◦ Internal component◦ Exposure to operating environment◦ Patient Contact/Critical component

Cosmetics◦ None/Internal◦ Process variable◦ High visibility

Quality Criteria

Replacement Risk◦ Easy to replace, standard practice◦ Re-workable◦ Difficult or no rework/must replace

Detectable◦ Visually detectable◦ Detectable in process◦ Not detectable

Quality Criteria (cont.)

Quality Risk Matrix

Options

Criteria

CalculatedRisk

Complexity Rati

ng

1 - Simple, not critical (DIN)2 - Nominal difficulty3 - Complex/Critical

Exposure1 - Inside Device2 - Exposed to operating environment3 - Patient contact/Critical Component

Cosmetic1 - None/Internal2 - Process variable3 - High Visibility

Replacement Risk1 - Easily replaceable/Minimal rework2 - Reworkable3 - No rework/Must replace

Detectable 1 - Visibly detectable2 - Detectable in assembly3 - Not detectable

Total 30

2

2

3

Quality Risk

Hig

h

3

3

Quality Risk4 - 11 = Low 12 - 24 = Nom 25 - 36 = High

Combine Risk

L N HH High High High

N Low Nominal High

L Low Low NominalPurc

hasi

ng R

isk

Quality RiskOverall Risk

Risk Matrix

Material #:

Supplier Type Rati

ng

Complexity Rati

ng

1 - Fully Certified 1 - Simple, not critical (DIN)2 - Existing Supplier 2 - Nominal difficulty3 - New Supplier 3 - Complex/Critical


Delivery (last 12 weeks) Cosmetic1 - 98-100 1 - None/Internal2 - 90-97 2 - Process variable3 - <=89 3 - High visibility

Quality (last 12 weeks) Replacement Risk1 - 98-100 1 - Easily replaceable/Minimal rework2 - 90-97 2 - Reworkable3 - <=89 3 - Difficult or no rework/Must replace

Qualification Performance Detectable 1 - 5 of last 5 complete 1 - Visibly detectable2 - 4 of last 5 complete 2 - Detectable in assembly3 - <= 3 of last 5 complete 3 - Not detectable

Total 20 Total 27

4 - 11 = low, 12 - 24 = Nom, 25 - 36 = High Overall Risk Matrix

Overall Risk Assessment High

1 2

3 2

3 2

Risk Assessment123456


Nom

inal

Hig

h

2 3

3 3

Do we have these characteristics?◦ Consistent?◦ Easy to use and easy to understand?◦ Provides objective evidence?◦ Flexible?

We don’t have these yet…◦ Subsequent activities can be based on results◦ Single assessment to satisfy all Supplier Quality

activities

Characteristics


Agenda

Is it possible to apply a single risk assessment model to all aspects of purchased product control?

Integration

Risk Assessment

Selection

Evaluation

Control

Re-Evaluation

Consider ◦ all components provided by the supplier or◦ all those components will be provided by the

supplier, Select the worst case for each option over all

components to get the overall supplier risk level.

i.e. If the supplier provides 5 components, but 4 are nominal, but 1 of them is a critical component, then the score for part complexity is High

Uses: Supplier Audit plans, improvement plans, critical supplier list

Evaluation – Supplier Risk

Risk assessment is completed for a particular component to get component risk level for a particular supplier

This can be done for ◦ New parts◦ Parts that have changed

Scale requirements based on risk level:◦ Part qualification requirements◦ Need for a supplier corrective action◦ Receiving inspection requirements (STS eligibility,

etc)

Control – Component Risk

Supplier Risk level can be used as part of supplier selection. ◦ Based on the score for each element of the

assessment, strengths and weakness become apparent

Re-Evaluation can be completed each time a new or modified component is assessed◦ New component (i.e. more complex or critical)◦ New data (i.e. on-time delivery)◦ This new assessment can be compared to the

existing one as part of Re-Evaluation

Selection and Re-Evaluation

Documented evidence of risk assessment may be an initialed copy of the completed Risk Matrix itself.

Subsequent review of the matrix clearly indicates the decision process

Results are easy to understand and auditable

Documentation

Can a single assessment method be used to satisfy multiple systems?

Yes! And subsequent activities can be scaled based on the result

As Part of the QMS

Risk Assessment

Selection

Evaluation

Control

Re-Evaluation

Questions?

KSE Supplier Quality Systems

Thank you