How Can Control Charts Advance Your Work?

Melanie Rathgeber MERGE Consulting rathgeber.melanie@gmail.com

Andrew Wray BC Patient Safety & Quality Council awray@bcpsqc.ca

The Health Care Data Guide, Provost & Murray (2011)

This course is designed to demonstrate:

1. When to use a control chart

2. What you need to make a control chart

3. How to interpret a control chart using plain language

4. The value of control charts for making decisions –

including use on leadership dashboards

Review: Run Charts

1. Make performance visible 2. Is there improvement – probability based rules? 3. Is the improvement holding? 4. What is the impact of our PDSA cycles?

What do Run Charts tell us?

Control Charts – What features are different than a run chart?

Variation 101

Walter Shewhart

(1891 – 1967)

W. Edwards Deming

(1900 - 1993)

The Pioneers of Understanding Variation

Understanding Variation

• Intended variation is an important part of effective, patient-centered health care.

• Unintended variation is due to changes introduced into healthcare process that are not purposeful, planned or guided.

• Walter Shewhart focused his work on this unintended variation. He found that reducing unintended variation in a process usually resulted in improved outcomes and lower costs. (Berwick 1991)

Intended and Unintended Variation

Examples of Intended Variation?

Most improvement work is focused on unintended variation.

Examples?

Variation

Intended Variation

Unintended Variation

Common Cause

Variation

Special Cause Variation

Common Causes—those causes inherent in the system over time, affect everyone working in the system, and affect all outcomes of the system

– Common cause of variation – Chance cause – Stable process – Process in statistical control

Special Causes—those causes not part of the system all the time or do not affect everyone, but arise because of specific circumstances

– Special cause of variation – Assignable cause – Unstable process – Process not in statistical control

Health Care Data Guide, p. 108

Shewhart’s Theory of Variation

Common and Special Causes of variation

Write the letter “a” five times in a column

Stable Process – Implies that the variation is predictable within common bounds – only common cause variation.

Unstable Process – A process that is affected by both special cause variation and common cause variation. The variation from one time period to the next is unpredictable.

Shewhart Charts (aka control charts)

The Shewhart chart is a statistical tool used to distinguish between variation in a measure due to common causes and variation due to special causes

Health Care Data Guide, p. 113

UCL

LCL

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

ICUNov

ICUDec

ICU Jan ICUFeb

EmergNov

EmergDec

EmergJan

EmergFeb

SurgNov

SurgDec

SurgJan

SurgFeb

MedNov

MedDec

MedJan

MedFeb

Percent of Staff Vaccinated Percent

Control Charts – What features are different than a run chart?

3 lines on a control chart: – centre line (mean) – upper and lower limit (+ 3 sigma)

Tchebycheff theory of probability

3 sigma shown by Deming and Shewhart in practice to distinguish between special cause and common cause

Calculated by Statistical Process Control Software (special calculations for limits depending on the type of data)

Not the same of confidence intervals

Limits

1. You have a different sample size for each time period

2. You want to determine change (improvement). Evidence of special cause variation.

3. You want to know if the results are stable and predictable? If not, this system is not ready for improvement.

4. You want to predict what performance will be next month.

5. You want to understand the reasons for variation.

When to Use a Control Chart

UCL

LCL

0%

10%

20%

30%

40%

50%

60%

70%

1/1/

10

2/1/

10

3/1/

10

4/1/

10

5/1/

10

6/1/

10

7/1/

10

8/1/

10

9/1/

10

10/1

/10

11/1

/10

12/1

/10

1/1/

11

2/1/

11

3/1/

11

4/1/

11

5/1/

11

6/1/

11

7/1/

11

8/1/

11

9/1/

11

10/1

/11

11/1

/11

Percent of clients seen within 2 hours of arrival

1. Different Sample Size for Each Time Period

UCL

LCL

0

5

10

15

20

25

30

35

40

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45

Wait time

2. Evidence of Special Cause Variation (Improvement)

UCL

LCL

20%

30%

40%

50%

60%

70%

80%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

Percent of Med Orders With Transcription Errors

3. Is System Stable and Predictable?

UCL

LCL

20%

30%

40%

50%

60%

70%

80%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

Percent of Med Orders With Transcription Errors

4. What will result be next month?

UCL

LCL

20%

30%

40%

50%

60%

70%

80%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

Percent of Med Orders With Transcription Errors

Senior Leader Dashboards Instead of Traffic Lights

Do you want to show Run chart Control chart 1. whether measures are acceptable/meeting targets?

√

√

2. whether there is variation within the province or your HA in your measures?

√

3. what factors are responsible for variation and/or improvement?

√

4. whether changes/interventions are resulting in improvement at a HA level?

√

√ More sensitive to change

5. whether small-tests-of-change are resulting in improvement at a unit/improvement team level?

√

√

What do you need from your data?

UCL

LCL

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

ICUNov

ICUDec

ICU Jan ICUFeb

EmergNov

EmergDec

EmergJan

EmergFeb

SurgNov

SurgDec

SurgJan

SurgFeb

MedNov

MedDec

MedJan

MedFeb

Percent of Staff Vaccinated Percent

5. What are the Sources of Variation?

What you need - data and software Choosing the right Shewhart Chart Rules for analyzing Shewhart Charts

The Technical Stuff:

Case Study #1a

Case Study #1b

Percent of cases with urinary tract infection

Case Study #1c

Percent of cases with urinary tract infection Percent of cases with urinary tract infection

Case Study #1d

Percent of cases with urinary tract infection

Case Study #1e

Percent of cases with urinary tract infection

Case Study #1f

Percent of cases with urinary tract infection

Rat

e pe

r 100

ED

Pat

ient

sUnplanned Returns to Ed w/in 72 Hours

M41.78

17

A43.89

26

M39.86

13

J40.03

16

J38.01

24

A43.43

27

S39.21

19

O41.90

14

N41.78

33

D43.00

20

J39.66

17

F40.03

22

M48.21

29

A43.89

17

M39.86

36

J36.21

19

J41.78

22

A43.89

24

S31.45

22

MonthED/100Returns

u chart

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 190.0

0.2

0.4

0.6

0.8

1.0

1.2

UCL = 0.88

Mean = 0.54

LCL = 0.19

Rat

e pe

r 100

ED

Pat

ient

sUnplanned Returns to Ed w/in 72 Hours

M41.78

17

A43.89

26

M39.86

13

J40.03

16

J38.01

24

A43.43

27

S39.21

19

O41.90

14

N41.78

33

D43.00

20

J39.66

17

F40.03

22

M48.21

29

A43.89

17

M39.86

36

J36.21

19

J41.78

22

A43.89

24

S31.45

22

MonthED/100Returns

u chart

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 190.0

0.2

0.4

0.6

0.8

1.0

1.2

UCL = 0.88

Mean = 0.54

LCL = 0.19

Special cause: point outside the limits

%Percent Trauma Patients D/C to Home

M626.7

231

A658.0

241

M597.0

220

J600.0

227

J570.0

260

A651.0

233

S588.0

238

O628.0

250

N626.0

270

D645.0

240

J594.0

227

F600.0

228

M723.0

264

A658.0

278

M598.0

261

J543.0

208

J627.0

268

A658.0

293

S582.0

264

MonthTrauma Volume# D/C to Home

p chart

M A M J J A S O N D J F M A M J J A S25

30

35

40

45

50

55

UCL = 45.83

Mean = 39.93

LCL = 34.03

%Percent Trauma Patients D/C to Home

M626.7

231

A658.0

241

M597.0

220

J600.0

227

J570.0

260

A651.0

233

S588.0

238

O628.0

250

N626.0

270

D645.0

240

J594.0

227

F600.0

228

M723.0

264

A658.0

278

M598.0

261

J543.0

208

J627.0

268

A658.0

293

S582.0

264

MonthTrauma Volume# D/C to Home

p chart

M A M J J A S O N D J F M A M J J A S25

30

35

40

45

50

55

UCL = 45.83

Mean = 39.93

LCL = 34.03

Special cause 2 out of 3 consecutive points in outer third of limits or beyond

#

of

Ne

ed

les

tic

ks

Employee Needlesticksc c ha r t

UCL = 12.60

Mean = 5.54

New Needles Test

1-05 3-05 5-05 7-05 9-05 11-05 1-06 3-06 5-06 7-06 9-06 11-06 1-00

5

10

15

20

#

of

Ne

ed

les

tic

ks

Employee Needlesticksc c ha r t

UCL = 12.60

Mean = 5.54

New Needles Test

1-05 3-05 5-05 7-05 9-05 11-05 1-06 3-06 5-06 7-06 9-06 11-06 1-00

5

10

15

20

Con

tam

inat

ions

/100

0Blood Culture Contaminations Org 1: last 2 years

u chart

Jan-0

3

Mar-0

3

May-0

3Ju

l-03

Sept-0

3

Nov-03

Janu

ary-0

4

March

-04

May-0

4

July-

04

Septem

ber-0

4

Novem

ber-0

4

Decem

ber-0

4

20

25

30

35

40

45UCL

Mean

LCL

Con

tam

inat

ions

/100

0Blood Culture Contaminations Org 1: last 2 years

u chart

Jan-0

3

Mar-0

3

May-0

3Ju

l-03

Sept-0

3

Nov-03

Janu

ary-0

4

March

-04

May-0

4

July-

04

Septem

ber-0

4

Novem

ber-0

4

Decem

ber-0

4

20

25

30

35

40

45UCL

Mean

LCL

Common Cause

1. It might be evidence of improvement

2. It might be evidence that things are getting worse

3. It might be evidence of an unintended consequence

4. If you haven’t introduced improvements, it may indicate an unstable system – therefore, you can’t predict future performance

Is Special Cause Variation Good or Bad?

Using Control Charts to Make Decisions

If we have Special Cause Variation… What can we do to learn about it and remove it from the system? If we have only Common Cause Variation… Are we happy with our level of performance? How can we improve the system?

ACTION

No Special Cause, Only Common Cause

Special Cause

Take action on individual points

Mistake1:

costs money and time

Good approach

Try to change the whole system to improve it

Good approach

Mistake2: costs money and time

Reacting to Common and Special Cause Variation

Source: The Data Guide. Provost and Murray 2010

INDICATOR GOAL TARGET 2007 2008 2009 Q1 2009 Q2 2009 Q3

Percent of patients waiting less than 2 hours ↑ 95.0 46.0 74.1 88.0 91.7 88.7

How do Leaders Make Decisions?

Based on relation to goal, but not on variation Doesn’t show whether things are actually getting better or worse No focus on prediction

The Health Care Data Guide, p. 355

Currently meeting goal Near goal (e.g. within 75% of goal Not near goal/heading in the wrong direction

What is the problem with Traffic Lights?

%

UCL = 60.90

CTL = 46.06

LCL = 31.22

UCL = 87.30

CTL = 74.24

LCL = 61.18

Good

J 07 M M J S N J 08 M M J S N J 09 M M J S N D0

20

40

60

80

Source: The Data Guide. Provost and Murray 2010

INDICATOR GOAL TARGET 2007 2008 2009 Q1 2009 Q2 2009 Q3

Percent of patients waiting less than 2 hours ↑ 95.0 46.0 74.1 88.0 91.7 88.7

The Health Care Data Guide, p. 356

INDICATOR GOAL TARGET 2007 2008 2009 Q1 2009 Q2 2009 Q3

Percent of staff vaccinated ↑ 65.0 53.5 51.2 54.3 61.2 65.1

%

UCL = 75.18

CTL = 52.33

LCL = 29.48

J 07 M M J S N J 08 M M J S N J 09 M M J S N D0

20

40

60

80

The Health Care Data Guide, p. 356

INDICATOR GOAL TARGET 2007 2008 2009 Q1 2009 Q2 2009 Q3

Percent of staff vaccinated ↑ 65.0 53.5 51.2 54.3 61.2 65.1

UCL

LCL 0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

9.00

10.00

Jan

Feb

Mar

ch

Apr

il

May

June July

Aug

ust

Sep

tem

ber

Oct

ober

Nov

embe

r

Dec

embe

r

Janu

ary

Febr

uary

Mar

ch

Apr

il

May

June July

Aug

ust

Sep

tem

ber

Oct

ober

Nov

embe

r

Dec

embe

r

Janu

ary

Febr

uary

Mar

ch

Apr

il

May

June July

Aug

ust

Sep

tem

ber

Surgical Infection Rate

Jan Feb Mar April May June July Aug Sept Oct Nov Dec Jan Feb Mar April May June July Aug Sept Oct Nov Dec Jan Feb Mar April May June July Aug Sept

1.4 3.5 1.0 2.3 4.4 6.4 5.4 2.1 4.7 3.3 3.5 3.1 4.0 2.5 2.4 4.7 5.9 6.7 2.2 4.2 6.8 2.3 6.7 4.4 3.2 4.4 5.4 2.0 4.5 1.6 2.3 1.1 2.6

UCL

LCL 0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

9.00

10.00

Jan

Feb

Mar

ch

Apr

il

May

June July

Aug

ust

Sep

tem

ber

Oct

ober

Nov

embe

r

Dec

embe

r

Janu

ary

Febr

uary

Mar

ch

Apr

il

May

June July

Aug

ust

Sep

tem

ber

Oct

ober

Nov

embe

r

Dec

embe

r

Janu

ary

Febr

uary

Mar

ch

Apr

il

May

June July

Aug

ust

Sep

tem

ber

Surgical Infection Rate

Jan Feb Mar April May June July Aug Sept Oct Nov Dec Jan Feb Mar April May June July Aug Sept Oct Nov Dec Jan Feb Mar April May June July Aug Sept

1.4 3.5 1.0 2.3 4.4 6.4 5.4 2.1 4.7 3.3 3.5 3.1 4.0 2.5 2.4 4.7 5.9 6.7 2.2 4.2 6.8 2.3 6.7 4.4 3.2 4.4 5.4 2.0 4.5 1.6 2.3 1.1 2.6

• Deciding a sampling strategy will depend on what you want to do with the data.

• We’ll talk about 2 situations:

• For the measures you are trying to plot for a typical local improvement project

• For measure that will be reported to the Board/Ministry/public and is used for improvement and accountability

Sampling Strategy for Shewhart Charts

P-chart

– One of the most common charts used in healthcare – Used to plot attribute data

– Yes/no – Good/bad – Compliant/non-compliant

Subgroup size for improvement measures at a unit/team/department level

Guidelines for Selecting Subgroup Size for an Effective P chart (adapted from The Health Care Data Guide)

What do you expect

your result to be?

Minimum Subgroup Size

(n) Required to Have < 25% zero for p's

Minimum Subgroup Size Guideline* – based on

formula: 300 / pbar

Minimum Subgroup

Size Required to Have LCL > 0

2% 70 150 450 3% 47 100 300 4% 35 75 220 5 % 28 60 175 6% 24 50 130 8% 17 38 104 10% 14 30 81 12% 12 25 66 15% 9 20 51 20% 7 15 36 25% 5 12 28 30% 4 10 22 40% 3 8 14 50% 2 6 10

*traditional SPC guideline to get reasonably symmetric distribution of P’s

Minimum to be useful: Start here: minimum to have a limit to quickly detect change:

- Can use “Operating Characteristic Curves” approach to identifying sample size

- What do you need to know:

1. Current performance 2. The size of the change you want to detect 3. The amount of time you want to detect it in

What about ‘big’ measures?

Operating characteristics curve

Monthly sample size

Percent chance of detecting 15% change in one month (x)

Percent Chance of Detecting 15% change in three months (1 – (1- x)3)

35 2.5 7.3 50 11.2 30.0 70 28.7 63.8 90 58.8 93.0 100 70.3 97.4

References

BCPSQC Measurement Report http://www.bcpsqc.ca/pdf/MeasurementStrategies.pdf Langley GJ, Moen R, Nolan KM, Nolan TW, Norman CL, Provost LP (2009) The Improvement Guide (2nd ed). Provost L, Murray S (2011) The Health Care Data Guide. Berwick, Donald M, Controlling Variation in Health Care: A Consultation with Walter Shewhart, Medical Care, December, 1991, Vol. 29, No 12, page 1212-1225. Perla R, Provost L, Murray S (2010) The run chart: a simple analytical tool for learning from variation in healthcare processes, BMJ Qual Saf 2011 20: 46-51. Associates in Process Improvement website www.apiweb.org Perla R, Provost L, Murray S (2013) Sampling Considerations for Health Care Improvement, Q Manage Health Care 22;1: 36–47