C6 Melanie Rathgeber and Heidi Johns - Building a Measurement Plan for QI Projects

Building a Measurement Plan for QI Projects

Quality Forum 2013

Melanie Rathgeber

MERGE Consulting

rathgeber.melanie@gmail.com

Heidi Johns

BC Patient Safety & Quality Council

hjohns@bcpsqc.bc.ca

This course if for you if ….

Objectives

This course is designed to demonstrate:

1. Guidelines for choosing project indicators

2. Tips for collecting data

3. The use of run charts to display data

http://www.ihi.org/knowledge/Pages/Tools/RunChart.aspx

Langley GJ, Moen R, Nolan KM, Nolan TW, Norman CL, Provost LP (2009) The Improvement Guide (2nd ed).

Provost L, Murray S (2011) The Health Care Data Guide.

Perla R, Provost L, Murray S (2013) Sampling Considerations for Health Care Improvement, Q Manage Health Care 22;1: 36–47

Perla R, Provost L, Murray S (2010) The run chart: a simple analytical tool for learning from variation in healthcare processes, BMJ Qual Saf 2011 20: 46-51.

Perla R, Provost L, (2012). Judgment sampling: A health care improvement perspective., Q Manage Health Care 21;3: 169-175

Resources

On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project?

1 4Not at all confident Extremely confident

QI projects

“Systematic, data guided, activities designed to bring about immediate improvement in a health care setting”

Lynn et al. The ethics of using Quality Improvement methods in Health Care, Annals of Internal Medicine. 2007; 146: 666-73

QI projects

Trying to improve something, by changing or introducing a process

- Need to measure something- Need to measure the new process

Purpose of Data in QI Projects

Need to know:- where we started (baseline)- how we change over time (e.g. each week)- when we have reached our target

- Not for judgment (doesn’t go on dashboards or to external agencies)

- Not for research

Run Charts

Measurement Worksheet

Measure Operational Definition

Outcome, Process or Balancing

Data Collection Strategy

Frequency of Data Collection

How will measure be displayed

Baseline result

Target result

Source: Langley et al. (2009). The Improvement Guide. 2nd edition

Lean/Six Sigma

Define

Measure

Analyze

Improve

Control

What are we trying to accomplish?

How will we know that a change is animprovement?

What changes can we make that will result in improvement?

Act Plan

Study Do

Model for Improvement

What are we trying to accomplish?

How will we know that a change is animprovement?

What changes can we make that will result in improvement?

Act Plan

Study Do

Aim Statement

A measurement plan starts with an Aim statement

An Aim statements specifies

What will improve?When will it improve?How much will it improve?For whom will it improve?

Example: The percent of diabetes patients seen by their own GP at Canada Way Clinic will increase from 40% to 95% by May 2013.

Dissecting the Aim statement

“Some is not a number; soon is not a time” Donald Berwick, Former CEO of IHI

What will improve? Percent of patients seeing their own GP

When will it improve? By May 2013

How much will it improve? From 15% to 90%

For whom will it improve? Diabetes patients at Canada Way Clinic

Examples

What will improve?

When will it improve?

How much will it improve? (numerical goal)

For whom will it improve?

What are we trying to accomplish?

How will we know that a change is animprovement?

What changes can we make that will result in improvement?

Act Plan

Study Do

Family of Measures

Family of Measures

Outcome measures Based on your Aim statement What are we trying to accomplish? What is ultimately better? Voice of the patient/customer

Process measures What are you changing – is it really happening? Voice of the system – what is being done differently? Change more quickly than outcomes

Balancing measures What unintended consequences might occur?

Example

Aim Statement: The percent of diabetes patients seen by their own GP at Canada Way Clinic will increase from 40% to 95% by May 2013.

Outcome Measure:

Percent of diabetes patients seen by their own GP

Process Measure(s): Go back to Model – Question 3

Balancing Measure(s):

What are we trying to accomplish?

How will we know that a change is animprovement?

What changes can we make that will result in improvement?

Act Plan

Study Do

Identifying, testing, and

implementing changes

Example: What changes can we make?

Changes tested and ready to implement:

- Patients are booked for a follow-up before they leave the office- GP’s set aside Wed morning and Friday afternoon for diabetes

group visits

Process Measure(s)

1. Percent of diabetes patients that leave the clinic with their next appointment booked

2. Number of non-diabetes patients each week who were ‘fit in’ on Wed morning or Friday afternoon

Example: Unintended Consequences

Staff are concerned that other patients will have to wait longer for an appointment, if they are not able to be seen on Wednesday morning and Friday afternoon

Balancing Measure

1. Average wait time for non-diabetes patients between calling for an appointment and being seen

On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project?

1 4Not at all confident Extremely confident

Family of Measures in Action – An Improvement Project

- What were the outcome/process/balancing measures?

- How were they chosen?

- How was the data useful in driving improvement?

- What was the data showing us?

Where do I start?• I have a hunch

• I need to determine

a target

• How am I going to get

the information

• What do I actually want to

accomplish?

I really needed to develop my AIM

• What was I going to DO • by WHEN • by HOW MUCH

• By September 2011 the wait time between referral and being seen will decrease from X to X

• Change idea: look at referrals coming in to see what the problem is. Maybe need to standardize the process. Hunch that process wasn’t consistent.

• I needed to gather the data to see:• What the actual wait time was• What the referrals looked like

Gathering the data

• Here is what I did………………….

April May June July Aug0%

5%

10%

15%

20%

25%

30%

9%

12%11%

10%

16%

Percent of Forms That Were Illegible

April May June July Aug0

5

10

15

20

25

30

35

40

24

15

17

12

19

Number of Different Forms Seen

Tracking Key Process Measure over Time

Percent of referral forms fully complete

3235

2926 24

80 80

90

0

10

20

30

40

50

60

70

80

90

100

April '11 May June July August Jan '12 Feb April

* Calculations to be confirmed

Starting to Track Time Between Referral and Date Patient Seen for First Appointment

* Calculations to be confirmed

Patient1 Patient2 Patient3 Patient4 Patient5 Patient60

5

10

15

20

25

30

35

40

45

29

1815

30

15 15

Days Between Referral and Appointment Date

Day

s

Family of Measures in Action – An Improvement Project

- What were the outcome/process/balancing measures?

- How were they chosen?

- How was the data useful in driving improvement?

- What was the data showing us?

On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project?

1 4Not at all confident Extremely confident

Making a Run Chart:

- time along the bottom (X axis)

- results along the side (Y axis)

- a centre line which is the median of all data points on the chart

- each dot on a run chart is the result for:

one case or

one day or

one week or

one month

Making a Run Chart In Excel

Monday Tuesday WednesdayThursday Friday Saturday Sunday Monday Tuesday Wednesday0%

20%

40%

60%

80%

100%

Result

Collecting the data

“Measurement should be used to speed things up, not to slow them down”

- IHI Breakthrough Series Guide

Some tips for getting started

1. Get Started Right Away – Real Time Data on a Run Chart

2. Sampling – Small samples are okay. Sample size increases over time.

3. Seek Usefulness, Not Perfection - Discuss an Operational Definition With Your Team

1. Get Started Right Away – Real Time Data on a Run Chart

Pre = 8 days wait timePost = 3 days wait time

Why not pre and post?

Adapted from Health Care Data Guide, p. 16 Figure 1.5 and 1.6 , Provost and Murray, 2011. San Francisco: Jossey Boss

Adapted from Health Care Data Guide, p. 16 Figure 1.5 and 1.6 , Provost and Murray, 2011. San Francisco: Jossey Boss

Wee

k 1

Wee

k 2

Wee

k 3

Wee

k 4

Wee

k 5

Wee

k 6

Wee

k 7

Wee

k 8

Wee

k 9

Wee

k 10

Wee

k 11

Wee

k 12

Wee

k 13

Wee

k 14

0

1

2

3

4

5

6

7

8

9

10

Scenario 1 when we measure the same thing over time.

Adapted from Health Care Data Guide, p. 16 Figure 1.5 and 1.6 , Provost and Murray, 2011. San Francisco: Jossey Boss

Wee

k 1

Wee

k 2

Wee

k 3

Wee

k 4

Wee

k 5

Wee

k 6

Wee

k 7

Wee

k 8

Wee

k 9

Wee

k 10

Wee

k 11

Wee

k 12

Wee

k 13

Wee

k 14

0

1

2

3

4

5

6

7

8

9

10

change made between week 7 and 8

Scenario 2 when we measure the same thing over time.

Adapted from Health Care Data Guide, p. 16 Figure 1.5 and 1.6 , Provost and Murray, 2011. San Francisco: Jossey Boss

Wee

k 1

Wee

k 2

Wee

k 3

Wee

k 4

Wee

k 5

Wee

k 6

Wee

k 7

Wee

k 8

Wee

k 9

Wee

k 10

Wee

k 11

Wee

k 12

Wee

k 13

Wee

k 14

0

1

2

3

4

5

6

7

8

9

10

change made between week 7 and 8

Scenario 3 when we measure the same thing over time.

Showing improvement:

No improvement. Random fluctuation.

Improvement. Trend going up.

- There are simple rules, based on probability, that are used to determine evidence of improvement in our projects

- Interpretation: the rules tell us if there is a non-random pattern in our data.

- If we have implemented a change, and we see a non-random pattern (going in the right direction), it is evidence of improvement

Analyzing Run Charts

A Shift: 6 or more

An astronomical data point

Too many or too few runs

A Trend5 or more

Evidence of a non-random signal if one or more of the circumstances depicted by these four rules are on the run chart. The first three rules are violations of random patterns and are based on a probability of less than 5% chance of occurring just by chance with no change.

The Health Care Data Guide, p 78

Some tips for getting started

2. Sampling – Small samples are okay. Sample size increases over time.

Small samples per day or week are okay

Sample size builds over time

How much data to satisfy team that it is representative?

Simple strategies:

- every 5th patient

- all patients on Thursday morning

If you are reporting externally, or if you want to publish results of QI – may need different strategy

See papers by Perla, Provost, and Murray

Some tips for getting started

3. Seek Usefulness, Not Perfection - Discuss an Operational Definition With Your Team

Operational Definitions

Deciding on an operational definition should be done with your QI team

What time frame? Which patients? What criteria? What diagnosis? What constitutes “met the guideline?” What about patients that wanted something different? etcetera, etcetera, etcetera ……………………..

Operational Definition Example

Basic definition:

Patient satisfaction ratings from patient survey

Operational Definition Example

Basic definition:

Patient satisfaction ratings from patient survey

Operational definition:

Percent of surgical patients discharged this week that rated their experience with the discharge process as good or excellent, based on the surgical patient survey

Operational Definition Example

Basic definition:

Patient satisfaction ratings from patient survey

Operational definition:

Percent of surgical patients discharged this week that rated their experience with the discharge process as good or excellent, based on the surgical patient survey

“The Data Are Wrong”

“The Data Are Wrong”

Not a matter of right versus wrong

What is your operational definition?

Involve others from the start in this decision.

Are we good at using data to drive decisions? Do we have a plan of action?

• What if we are not seeing evidence of improvement?

• What if we see improvement but not on target?

• How long do we collect the data?

Data Display Principles

April May June July Aug Sept Oct Nov Dec Jan0%

5%

10%

15%

20%Percent of Forms That Were Illegible

April May June July Aug Sept Oct Nov Dec Jan0%

5%

10%

15%

20%Percent of Forms That Were Illegible

April May June July Aug Sept Oct Nov Dec Jan0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%Percent of Forms That Were Illegible

April May June July Aug Sept Oct Nov Dec Jan0%

5%

10%

15%

20%Percent of Forms That Were Illegible

*hypothetical data – illustrative purposes only

1 2 3 4 5 6 7 8 9 10 11 12 13 140

5

10

15

20

25

30

35

40

Large Teaching HospitalsLarge Community HosptialsMedium Community HospitalsSmall Community Hospitals

SMALL MULTIPLES – all info on one page

0

10

20

30

40

1/1/

09

2/1/

09

3/1/

09

4/1/

09

5/1/

09

6/1/

09

7/1/

09

8/1/

09

9/1/

09

10/1

/09

11/1

/09

12/1

/09

1/1/

10

2/1/

10

ProvincialReadmission Rate

0

10

20

30

40Large Teaching

Hospitals

010203040

Large Community Hospitals

010203040

Medium Community Hospitals

010203040 Small Community

Hospitals

*hypothetical data – illustrative purposes only

On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project?

1 4Not at all confident Extremely confident