Diagnostic quality problem solving - BRAUDE quality problem solving Conceptual framework: Diagnosis is a search through a state-space. 9 A study of diagnostic strategies Quality engineering:

Diagnostic quality problem solving

… Strategies for identifying causes of problems

Jeroen de MastUniversity of Amsterdam

Karmiel, 2 June 2016 [email protected]

2

Personal introduction

Jeroen de Mast Prof. of Methods & Statistics for Operations Management Amsterdam Business School, University of Amsterdam Principal consultant, much experience in high-tech,

manufacturing, healthcare, services, logistics industries.

Research interests: Statistical evaluation of measurement and testing systems. Appointment scheduling Operations improvement in healthcare Lean, Six Sigma, problem-solving.

3

Electrical instabilities

High voltageelectrical device

Connector / cable

Power supply

Problem: “Electrical instabilities”

4

“Electrical instabilities”: From week 12 (2008) onwards.

Product and connector completely destroyed when connected to power supply for 1 out of 8 products (12%).

Only affects product type TA. Similar product TB not affected.

Diagnostic problem solving: What are the causes of the problem?

Diagnostic problem solving

5


Check for anomalies in the production process: Dimensional variation in the connectors Color differences Contamination of tools with chemicals.

Brainstorming: Main suspect is the connector. Various explanations and causes were suggested.

Result:> 45 potential causes, to be investigated in a statistically designed experiment

6


Sort 45 potential causes by multi-voting Design experiment for top 15

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0

10

20

30

40

50

60

70

80

90

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

¨¨¨¨¨¨¨

2 design: 32 runs

Geometry of connector

Tool coating

Operator handling of connector

Process tool setup

Connector depth

Process control

Amount of grease

Insulator geometry

Geometry of connector

Tool coating

Operator handling of connector

Process tool setup

Connector depth

Process control

Amount of grease

Insulator geometry

Three months later:Still 1 out of 8 products destroyed.

7

Case: conclusion

Brainstorming & DOE approach:- Extensive search space causes not enumerable or

multitudinous laborious- Extensive search space problem solver may get bogged

down in wrong part of the search space- Raised candidate causes are too unspecific to allow useful

experimentation

The Brainstorming & DOE approach is effective only after sufficient focus has been achieved.

Needed for diagnosis in complex situations: strategies for achieving focus(= narrowing down the search space).


Conceptual framework:Diagnosis is a search through a state-space

9

A study of diagnostic strategies

Quality engineering: Juran: “diagnostic” and “remedial journey” Six Sigma: rather incoherent and poorly structured collection of

tools (5×Why, C&E-diagram, exploratory data-analysis, …) Shainin, Kepner & Tregoe: branch-and-prune strategy

Scientific literature: Troubleshooting of devices Artificial intelligence Medical diagnosis

- Conceptual framework- Six strategies for efficient

diagnosis

10

Conceptual framework

Conceptual framework for the process of diagnosis: Based on concepts from Artificial Intelligence As a search through a state space until a goal state is reached.

State space

Domain knowledge:‐Physical structure‐Functional structure‐Operations context‐Normal behavior‐General knowledge

Fault theory:‐Fault dictionaries‐Pathologies‐Taxonomies‐Pattern recognition procedures

Observations:‐Observational data‐Tests and experiments‐Findings

Domain knowledge and observations

Hypothesisgeneration

Hypothesistesting

Hypothesisevaluation

Accepted:Under

consideration: Rejected:

H1H3

H4H2

H5

H6

H7 H8

11

Find a causal explanation for the observed problem.Potential explanations: hypotheses.

Conceptual framework: hypotheses

Many rejects

Cutting process malfunctions

H1

Product stretches when cut

Many rejects

Knife chipped

Dimensional variation

H2

Hypotheses can be global and vague,or specific and detailed.

12

Diagnosis: search through a state space

Hypotheses can be under consideration, accepted or rejected. These determine the states in the search process.

Goal-state: a sufficiently specific hypothesis is accepted.

State space

Accepted:Under


H1H3

H4H2

H5

H6

H7 H8

13

State space





Hypothesis generation

Accepted:Under


H1H2

H4H3

Hypothesis generation

Hypothesis testing

Hypothesis testing

Hypothesis evaluationHypothesis evaluation


14


The search progresses from state to state by three operators: Hypothesis generation: invent a new hypothesis

Status of new hypothesis: under consideration Driven by domain knowledge and observations

Hypothesis testing: collect new observations and knowledge for evaluating a hypothesis Guided by hypotheses under consideration

Hypothesis evaluation: determine whether a hypothesis is accepted or rejected Driven by domain knowledge and observations

15


State space






Hypothesistesting


Accepted:Under


H1H3

H4H2

H5

H6

H7 H8

Design of experimentsDesign of experiments

Statistical inferenceStatistical inference

Brain-stormingBrain-storming

16

Diagnostic strategies

Diagnostic strategy: makes the search process more efficient.

State space



Hypothesistesting


Accepted:Under


H1H3

H4H2

H5

H6

H7 H8

Diagnostic strategy

(how, when, in what order?)


Six diagnostic strategies:Search tactics that make diagnosis more efficient

18

Six strategies for diagnosis

Tools for diagnosing quality problems:C&E matrix, 6xWhy, Shainin System, brainstorming, DOE, pairwise comparisons, IS vs IS-NOT analysis, …

Strategy: the thinking patterns on which they are based:1. Blind trial & error2. Branch-and-prune tactics3. Known problem4. Proximate causes5. Syndrome-based search6. Funneling strategy

19

1. Blind trial & error2. Branch-and-prune tactics3. Known problem4. Proximate causes5. Syndrome-based search6. Funneling strategy

Six strategies for diagnosis

1. Blind trial & error search:- Randomly try out candidate causes

until you find the one that causes the problem.

- Effectively the least efficient search strategy possible.

- Finite number of causes expected trials.

20

2. Branch-and-prune

2. Branch-and-prune tactics:- Divide the space of all possible causes in high-level sub-classes

(“branching”) - On the basis of the structure of the product or process or based

on generic structures such as time and space.

- Next, by observation and testing try to rule out whole branches at once (“pruning”).

- Zoom-in on the retained branches, studying them in more detail.

- “Eliminate-and-zoom-in”

21

“Component swapping”Power supply and connector used for TA s adjusted such that they can be used for the TB s.

Result: power supply and connector give no problems when used for the TB s, but they do give problems when used for the TA s.Conclusion: problem is in the product itself, not in the connector or power supply

Cause of instabilities

ConnectorPower supply

Product & production history

2. Branch-and-prune

Product Connector

TB

TA Power supply

22

2. Branch-and-prune

4137332925211713951

0.20

0.15

0.10

0.05

0.00

Week

Pro

po

rtio

4137332925211713951

0.20

0.15

0.10

0.05

0.00

Week

Pro

por

tion

oF

Inst

abili

ties

_P=0.0425

UCL=0.1289

LCL=0

11

1

11

1

P Chart of Instabilities

Tests performed with unequal sample sizes

Branch-and-prune based on time

- Problems emerged suddenly in week 29.

- Conclusion: cause must be something that changed in wk 28-29.

Cause of instabilities

Events wks. 28,29

Events after wk. 29

Events before wk. 28

23

2. Branch-and-prune

Branch-and-prune based on product type- TA s are affected, TB s are not.- Conclusion: cause must be something that distinguishes the

production processes or product design of the TA s from those of the TB s.

- Zoom-in on differences main difference turns out to be the soldering process.

TA TB

Other techniques for branch-and-prune: Half-split technique (bisection) 4W2H questions (who, where, what, when, how, how much) Branch-and-prune based on functional structure

24

Looking on the internet whether a problem is known …

3. Known problem

Symptoms

Possible explanations

25

3. Known problem3. Known problemStudy whether the problem is known. Examine defective products closely, listing the observed

symptoms. What does the damage look like? Can you find abnormal measurements? Can you disassemble a product to examine what went wrong

(“autopsies”)?

Use the list of symptoms as a query in a search through a knowledge base, where earlier experiences with similar problems are stored: Consult experts or the expert literature. Do a search on internet. Discuss the observed symptoms with colleagues

(“brainstorming”). Study analogous processes / products / problems

26

3. Known problem

Literature search for known issues with similar electrical devices: Dust Contamination with salts Contamination with metal particles Enclosures of air bubbles

Surfaces of some TAs cleaned, and residue analyzed sodium chloride (“table salt”) present on the surfaces.

Salt

27

4. Proximate causes

4. Proximate causes strategy Try to reason backwards from the observed symptoms to their

immediate (“proximate”) causes. Thus, one moves the problem description one step upwards in the chain of cause-and-effect. This gives a more focused problem definition.

Technique: 5xWhy? (Ask “why?” five times).

Instabilities

Short circuit

Salt depositFrom observations, the immediate cause of the instabilities turns out to be: a short circuit caused by a salt residue on the product’s surface.

Problem definition recast from “What causes the instabilities?”, to: “Where does the salt residue come from?”

28

Combining cluesTA TB

Soldering process

Where does the salt come from?

Soldering flux- Used for the TAs but not for the TBs.- Introduced recently (week 28?).- Turns out to contain sodium chloride. Soldering flux discontinued

- 4 weeks later: 0 instabilities- Problem solved!

29

5. Syndrome-based search

1. Blind trial & error2. Branch-and-prune tactics3. Known problem4. Proximate causes5. Syndrome-based search6. Funneling strategy

Observe a number of occurrences of the problem, and compare them to normal behavior (BOB vs WOW comparison).

Try to find a pattern of concomitant symptoms and characteristics that occur together with the problem This pattern is the syndrome.

The pattern may reveal characteristics of the causal mechanism that may help in ruling out options.

30

Example: eccentricity of pins on cell phone components

Syndrome: eccentricity has a bi-modal distribution. cause of eccentricity must have two distinguishable states.

Injection molding: 2 molds, one of them worn-out.

0.180.150.120.090.060.03

20

15

10

5

0

Eccentricity

Freq

uenc

y

Histogram of Eccentricity

5. Syndrome-based search

31

6. Funneling strategy

6. Funneling strategy Generate a list of specific and detailed hypotheses by

brainstorming. Design an efficient testing strategy (for example: DOE).

Only efficient after the search space has been narrowed down (focus).


A general strategy for problem diagnosis

33

Electrical instabilities: search process

Brainstorming Focus on connector No convincing results

Swapping test Focus on product itself

Comparison of production processes of TA and TB Focus on soldering process

Literature search Found salt residue

Possible explanation Salt residue causes short circuits

Study soldering process in detail, looking for origin of salt residue Soldering flux identified as culprit

Funneling

Branch & prune

Branch & prune

Known problem

Proximate causes

Funneling

34

Think strategically …

Think strategically …Use pruning principles to focus the search.

… but do not follow a single strategy rigidlyAt each stage in the search, reassess the situation and its tactical consequences.

1. Known problem

2. Proximate causes

4. Syndrome-based strategy

5. Funneling

3. Branch-and-prune

Known problem?

Achieve focus on the relevant part of the problem space

Efficient testing of detailed hypotheses

35

Problem solving in quality engineering

Joseph Juran: Universal sequence for quality improvement Diagnostic journey from symptoms to causes. Remedial journey from causes to remedy.

Shainin and Kepner & Tregoe systems Well structured strategies for problem diagnosis Limited to pruning strategies Poor grounding in science

Six Sigma’s DMAIC model

Analyze (DMAIC): identification of causes that determine the CTQ’s behavior

Many tools and techniques for hypothesis generation: Brainstorming Cause & Effect diagram 5xWhy? Exploratory data analysis

Strong techniques for testing the effects of hypothesized causes

36

Six Sigma

No or only rudimentary strategic structure Examples: Pyzdek and Breyfogle:

Incoherent collection of tools for hypothesis generation and testing

No strategy or structure

Example: Pande et al. (“Root cause analysis circle”) Analyze data / process Develop causal hypotheses (one or more) Analyze data / process Refine or reject hypotheses After a few cycles: Confirm and select vital few causes

Limited to funelling strategy Examples: George et al, Gitlow et al.

Brainstorming and fishbone diagram: list of candidate causes

Regression analysis: narrow down candidate causes until the vital few remain

37

Conclusions

Lessons from AI, medical diagnosis, troubleshooting: Problem diagnosis =

search through a space of potential explanations (“hypotheses”) Search progresses towards goal state by

hypothesis generation, testing and evaluation. Diagnostic strategies structure theses tasks and make them

efficient.

Opportunity: Courses on Six Sigma’s DMAIC fall short Organize tools in strategic structure

38

Documents

Diagnostic quality problem solving - BRAUDE quality problem solving Conceptual framework: Diagnosis is a search through a state-space. 9 A study of diagnostic strategies Quality engineering: