Measurement for Improvement Curriculum - hse.ie · The Measurement for Improvement Curriculum ... MFI is the analysis and presentation of quantitative and qualitative data to

Measurement for Improvement Curriculum A reference document to support consistent Measurement for Improvement training in Irish healthcare

Developed by the Quality Improvement Division, Measurement for Improvement, Health Service Executive Dr. Gemma Moore PhD Dr. Michael Carton PhD Ms. Gráinne Cosgrove Dr. Jennifer Martin Version 1, August 2017

2 | P a g e

Acknowledgements

The Measurement for Improvement Curriculum has been developed in consultation with national and

international experts in Measurement for Improvement and Quality Improvement. The Quality Improvement

Division (QID), Measurement for Improvement would like to express our gratitude to the following who took

the time to review and offer feedback and suggestions:

Dr Mary Browne; QID, HSE.

Dr Philip Crowley; National Director, QID, HSE.

Dr John Fitzsimons; Clinical Director for Quality Improvement, QID, HSE.

Caralyn Horne; Quality Standards and Compliance Officer, National Social Care Division - Quality &

Safety, HSE.

Dr. Peter Lachman; CEO, The International Society for Quality in Health Care.

Lorraine Murphy; Quality Improvement Facilitator, QID, HSE.

Prof. Lloyd Provost; Associates in Process Improvement.

© QID Measurement for Improvement, Health Service Executive, 2017.

Dr. Gemma Moore PhD

Dr. Michael Carton PhD

Ms. Gráinne Cosgrove

Dr. Jennifer Martin

Address for Correspondence:

QID Measurement for Improvement,

Stewarts Hospital,

Mill Lane,

Palmerstown,

Dublin 20,

Ireland

Telephone: +353 (0) 766 956 927

Email: [email protected]

Web: https://www.hse.ie/eng/about/Who/QID/MeasurementQuality/measurementimprovement/

Twitter: @QIMeasurement

mailto:[email protected]

https://www.hse.ie/eng/about/Who/QID/MeasurementQuality/measurementimprovement/

https://twitter.com/QIMeasurement?lang=en

3 | P a g e

Contents

Introduction ............................................................................................................................................ 4

Measurement for Improvement (MFI) Definition ........................................................................................4

Purpose of Measurement for Improvement Curriculum .............................................................................4

Who is the Measurement for Improvement Curriculum Version 1 for? ......................................................5

Who will receive training on Measurement for Improvement? ..................................................................5

How the MFI Curriculum compliments the Framework for Improving Quality ...........................................5

How the MFI Curriculum compliments the Improvement Knowledge and Skills Guide ..............................6

Measurement for Improvement Levels of Expertise .................................................................................. 7

The Measurement for Improvement Curriculum ....................................................................................... 8

How to use the Curriculum ...........................................................................................................................8

Curriculum Diagram: The Context and the Seven Steps to Effective Measurement for Improvement ......9

MFI Level of Expertise 1: Tasks, Knowledge and Skill Areas ..................................................................... 10




Summary ............................................................................................................................................... 26

Bibliography .......................................................................................................................................... 27

Glossary ................................................................................................................................................. 28

Appendix I: Pre-requisites for Measurement for Improvement ................................................................ 31

Appendix II: Task listed by level .............................................................................................................. 32

Appendix III: Complete Curriculum ......................................................................................................... 36

Context for Measurement for Improvement ............................................................................................ 36

Step 1: Is there an Opportunity to Improve? ............................................................................................ 37

Step 2: Choose Measures .......................................................................................................................... 38

Step 3: Design Measurement Plan ............................................................................................................ 40

Step 4: Collect Data ................................................................................................................................... 46

Step 5: Analyse and Display Data .............................................................................................................. 47

Step 6: Interpret and Present Information ................................................................................................ 51

Step 7: Act On Findings .............................................................................................................................. 53

4 | P a g e

Introduction

Measurement for Improvement (MFI) Definition

MFI is the analysis and presentation of quantitative and qualitative data to identify opportunities for

improvement and to demonstrate if a change has resulted in an improvement. Its purpose is to drive better

decision making and support sustainable improvements in the quality of care. MFI is one of 6 drivers/ key

success factors identified in the ‘Framework for Improving Quality in our Health Service’ (HSE, 2016).1 The

key principles of MFI are outlined in the following Table:

Purpose of Measurement for Improvement Curriculum

The purpose of this curriculum is to identify the essential components that should be included in MFI

training in Ireland. It is intended as a reference document for all those designing and delivering MFI training.

These essential components should ensure that high quality and comprehensive MFI training is available and

provided consistently in Ireland.

1 Quality Improvement Division, Health Service Executive (2016) Framework for Improving Quality in our Health Service.

Part 1: Introducing the Framework http://www.hse.ie/eng/about/Who/QID/Framework-for-Quality-Improvement/

Key Principles of Measurement for Quality Improvement

•Measuring only what matters: defining and developing a limited number of qualitative and

quantitative measures that are robust and useful in demonstrating and driving improvement

•Measuring patient experience and outcomes as well as clinical outcomes

•Being smart in how we measure: use available data; measure once use often; look at families of

measures (e.g. infection rates, hand hygiene and hospital length of stay); measure variability; trends

over time; and benchmark with peers

•Seeking transparency in the measuring, sharing and reporting of information

•Building capability for extraction and sharing of information from data to provide assurance and

support improvement

•Building good data collection practices into routine work and record keeping

http://www.hse.ie/eng/about/Who/QID/Framework-for-Quality-Improvement/

5 | P a g e

Who is the Measurement for Improvement Curriculum Version 1 for?

Version one of the curriculum is intended primarily for use by QID Measurement for Improvement. This will

allow us to test and refine the curriculum with the aim of publishing Version Two in 2018. Version Two will

inform others in the HSE who provide MFI training and will form the basis of commissioning external

providers to deliver training in MFI on behalf of the HSE. However, we encourage anyone interested in

providing training in MFI to use Version One and we would greatly value your feedback on this version to

help up improve Version Two.

Who will receive training on Measurement for Improvement?

Our vision is that everyone working in health and social care services will receive some training in MFI,

depending on their requirements and the needs of the organisation. Four levels of MFI expertise are

envisaged, ranging from level one which provides a basic appreciation of the value of MFI (which it is

intended will be delivered through a 1-2 hour online module), progressing through the levels up to level four

which represents a comprehensive knowledge and skill set in MFI. This vision will take time and effort to

achieve; this curriculum is a foundation step in achieving the vision.

How the MFI Curriculum compliments the Framework for Improving Quality

The developments and learning in Quality Improvement over the last five years (HSE and internationally)

have been captured and presented in the form of the Framework for Improving Quality in our Health

Services, Part 1: Introducing the Framework. The aim of this Framework is to influence and guide the

thinking, planning and delivery of care in our services in order to achieve a culture of person centred quality

care that continuously improves. Measurement represents one of the six drivers or key elements of the

Framework alongside leadership, person and family engagement, staff engagement, use of improvement

methods, and governance (see figure 1 below).

Figure 1: Drivers for Framework for

Improving Quality

6 | P a g e

How the MFI Curriculum compliments the Improvement Knowledge and Skills Guide

The Improvement Knowledge and Skills Guide (HSE, 2017,draft) provides a list of knowledge and skills on

how to drive quality improvement across the six drivers of the Framework for Improving Quality in our

Health Services (HSE, 2016) including MFI. The Improvement Knowledge and Skills Guide assists staff in

assessing their current knowledge and skills in quality improvement, assists organisations in building staff

quality improvement capability and capacity so that they can participate in quality improvement initiatives

to improve the quality of care, and assists health sector trainers, third level colleges and institutions

developing quality improvement curriculums, education and training courses.

The Measurement for Improvement Curriculum compliments the Knowledge and Skills Guide by identifying

the essential training components required to achieve competence in the knowledge and skills required by

‘everyone’, ‘improvement team member’, ‘improvement team lead’ and ‘improvement advisor’ (as per the

diagram below). There is no ‘Improvement Champion/ Sponsor’ in the Curriculum as this role requires a mix

of the knowledge and skills seen in the other roles.

Measurement for Improvement Curriculum 'MFI Levels of Expertise'

Level 1

Level 2

Level 3

Level 4

The Improvement Knowledge and Skills

Guide 'Roles'

Everyone

Improvement Team Member

Improvement Team Measurement Lead

Improvement Advisor

7 | P a g e

Measurement for Improvement Levels of Expertise

In developing the curriculum, four MFI levels of expertise were identified, with each level presupposing the

knowledge and skills of the previous level.2

2 For a list of pre-requisites for Measurement for Improvement see Appendix I

•Values both measurement and measurement for improvement

•Has information on the basic principles of measurement for QI

•Understands when and why measurement for improvement should be considered

•Has an awareness of the three contexts of measurement for improvement (PDSA, QI project and system level)

•Can collect quantitative data for measurement for improvement

Level 1

•Understands how measurement for improvement fits with the 5 other drivers of the Framework for Improving

Quality

•Has an awareness of different types of data (quantitative and qualitative)

•Can interpret common charts used for measurement for improvement (run, SPC, funnel, bar, Pareto)

•Can create run charts

•Can act on findings

Level 2

•Can design a measurement plan

•Can design and analyse surveys

•Can design and collect qualitative data

•Can conduct basic thematic qualitative data analysis

•Can select and produce the common SPC chart (P, C, U, I) using either a recognised software package or an Excel

template

Level 3

•Can select and produce advanced SPC charts (T, G, P', U' and Cusum) using either a recognised

software package or an Excel template

•Can provide advice to other practitioners on the appropriate type of SPC chart to use

•Understands when qualitative methods should be used to measure the non-numerical aspects of

quality improvement and has the knowledge and skills to collect, analyse and interpret qualitative

data

Level 4

The Measurement for Improvement Curriculum

How to use the Curriculum

The curriculum describes The Context and 7 Steps to Effective Measurement for Improvement and details the tasks, knowledge and skills those engaging in MFI

work need to perform.3 Outlined under each step are the tasks (the specific tasks that must be performed), the knowledge and skills (the knowledge and skills

required to competently perform the associated task), and knowledge gradients (reflecting the increasing complexity progressing through the four MFI levels of

expertise) required for each of the four progressive MFI levels of expertise. Where a task is introduced at more than one MFI level of expertise, a knowledge

gradient is applied whereby the task is covered in increasing detail as the learner progresses through the relevant levels.4

The curriculum is displayed in a variety of user-friendly ways designed to facilitate its use:

Title Description Page

Curriculum Diagram: The Context and the Seven Steps to

Effective Measurement for Improvement’

Diagram displaying the key steps and associated tasks 9

MFI Level of Expertise 1: Tasks, Knowledge and Skill Areas The tasks and associated knowledge and skills associated with level 1 10 - 12




Complete Measurement for Improvement Curriculum Appendix III offers the full curriculum displaying each step, its

associated tasks, knowledge and skill area, and knowledge gradient

where it applies across the 4 MFI levels of expertise.

36 - 53

3 A glossary of terms is available on pages 28 – 30.

4 See Appendix II for further detail on how the knowledge gradient is applied across the four levels of expertise.

MFI Curriculum Diagram: The Context and the Seven Steps to Effective Measurement for Improvement

Context of Measurement

for Improvement

Definition of MFI

Different Applications of

MFI

Basic Principles of MFI

Defining the 7 Steps towards

Effective measurement

Step 1: Is there an

Opportunity to Improve?

Using Available Data and

Information to know if an

Improvement is Required

Subject Matter Expert

knowledge of where

Improvement is Required

Step 2: Choose

Measures

Select What to Measure

Identify Most Appropriate Data Source

Consider Inclusion of Qualitative Measures

Step 3: Design

Measurement Plan

Define Operational Definitions

Choose Data Collection

Method for Each Measure

Choose Population

Determine Frequency and

Duration of Data Collection

Establish Baseline Where Possible

Decide Which Analytical Tool to

Use

Plan Measurement Reporting

Ensure Plan is Ethically Sound

Prepare Topic Guide/ Questions for Interviews or

Focus Groups

Establish Objectives of Observation

Step 4: Collect Data

Collect Data According to

Measurement and Collection

Plans

Record Data Accurately

Record Details of PDSA Cycles

Step 5: Analyse and Display Data

Theory of SPC

Understanding Theory

Construct Control Chart

Construct Pareto Charts

Construct Run Charts

Construct Frequency Plots

Construct Scatter Plots

Analyse Survey Responses

Analyse Qualitative

Data

Step 6: Interpret and

Present Information

Apply the Five Rules to Control

Charts

Review Data with Subject

Matter Expert

Combine a Number of

Measures to Give Overall

Picture of the Aspect of

Quality of Care with Reference

to the Aim

Ensure Patient Voice is

Recognised and Maintained in the Family of

Measures

Action to address Special versus

Common Causes

Interpret Qualitative

Findings

Step 7: Act On Findings

Review your Measurement Results with

Respect to your Aim

Act appropriately

to Special Cause and

common cause variation

Use Measurement

Results to Identify Further

Areas for Improvement

Review and Share

Measurement Results with Stakeholders

Plan for Continued

Measurement

MFI Level of Expertise 1: Tasks, Knowledge and Skill Areas

Step Task Knowledge & Skill Area

Context for Measurement

for Improvement

C (a) Definition of Measurement for Improvement C (a) Understand what constitutes Measurement for

Improvement

C (b) Different Applications of Measurement for

Improvement

C (b) Understand the difference between Measurement for

Improvement for PDSA cycles, quality improvement projects

and at organisation level

C (c) Basic Principles of Measurement for Improvement C (c) Understand the basic principles of Measurement for

Improvement

C (d) Defining the 7 steps towards effective measurement

for Improvement

C (d) Understand the basic elements of the '7 steps Towards

Effective Measurement for Improvement'

C (e) Promote Measurement to Support Better Decisions C (e) Valuing measurement for improvement

Step 1:

Is there an Opportunity

to Improve?

1.1 Determine if there is an opportunity to Improve 1.1 (i) Making full use of available information.

Focuses on using both quantitative and qualitative

information already available to know if there is an

opportunity to improve

1.1 Determine if there is an opportunity to Improve 1.1 (ii) Making full use of Subject Matter Expert Knowledge.

Focuses on the understanding that Subject Matter Experts

have of where there are potential opportunities to improve

Step 2:

Choose Measures

2.1 Select What to Measure 2.1 (i) Measure only what matters.5

Focuses on measuring things that will have an impact on

your aim or your project, or that have the potential for

improvement.

5 Knowledge Gradient: Level 1 introduces the concept of measuring only what matters.

11 | P a g e

2.2 Identify Most Appropriate Data Source 2.2 (ii) Judgement, research and improvement measures.6

This knowledge area describes the differences between

Measurement for Judgement, for Improvement and for

research.

2.3 Consider Inclusion of Qualitative Measures 2.3 (i) Recognises the importance of including the voice of

patient

2.3 Consider Inclusion of Qualitative Measures 2.3 (ii) Understand the value of qualitative data in

Measurement for Improvement

Step 3:

Design Measurement Plan

3.1 Define Operational Definitions 3.1 (iv) Counts, rates, percentages, numerator and

denominator.7

3.3 Choose Population 3.3 (iv) Just enough data.

The principle of only collecting enough data for the purpose

at hand

3.4 Determine Frequency and Duration of Data Collection 3.4 (iii) Advantages of looking at data over time

3.6 Decide Which Analytical Tool to Use 3.6 (ii) Visual display.8

Awareness that there are different charts for different types

of data

3.8 Ensure Plan is Ethically Sound 3.8 (ii) Patient & staff confidentiality.

Ensure patients and staff are assured full confidentiality and

anonymity

6 Knowledge Gradient: Level 1 introduces the basic principles of the 3 types of Measurement

7 Knowledge Gradient: Those at level 1 and 2 understand counts, rates, percentages, numerator and denominator, which are defined with basic examples

8 Knowledge Gradient: Levels 1 & 2 discuss examples of basic charts that are commonly used

12 | P a g e

Step 4:

Collect Data

4.1 Collect Data According to Measurement and

Collection Plans

4.1 (i) Ensure fidelity and consistency of measurement plan

4.2 Record Data Accurately 4.2 (i) Principles of good data collection

4.3 Record Details of PDSA Cycles 4.3 (i) The knowledge of documenting PDSA cycles

Step 5:

Analyse & Display Data

5.1 Theory of SPC (SPC includes Run charts, Control Charts

and Funnel plots)

5.1 (i) Introduce the theory of SPC including its basis in

statistics

5.2 Understanding Theory 5.2 (i) Variation & Distribution9

5.2 Understanding Theory 5.2 (iv) The importance of trends and patterns in data10

5.3 Construct Control Chart S 5.3 (i) Mean & median

Step 6:

Interpret & Present

Information

6.3 Combine a Number of Measures to Give Overall Picture of

the Aspect of Quality of Care with Reference to the Aim

6.3 (iii) Awareness of HSE QI Framework for Improving

Quality, 2016

6.3 Combine a Number of Measures to Give Overall

Picture of the Aspect of Quality of Care with Reference to

the Aim

S 6.3 (iv) Awareness of National Standards for Safer Better

Health Care

Step 7:

Act on Findings

7.4 Review and Share Measurement Results with

Stakeholders

7.4 (ii) The importance of sharing learning and transparent

reporting

9 Knowledge Gradient: For level 1 variation at a basic level is introduced

10 Knowledge Gradient: At level 1 increasing vs. decreasing trends in data are discussed

13 | P a g e




for Improvement C (e) Promote Measurement to Support Better Decisions C (e)(ii) Measurement for improvement to support better

decisions11

Step 2:

Choose Measures 2.1 Select What to Measure 2.1 (ii) Measure only what matters.


your aim or that have the potential for improvement12

2.1 Select What to Measure 2.1 (iv) Structure, process, balancing and outcome

measures13

2.2 Identify Most Appropriate Data Source 2.2 (iii) Judgement, research and improvement 14measures.

This knowledge area describes the differences between

Measurement for Judgement and Measurement for

Improvement.

2.2 Identify Most Appropriate Data Source 2.2 (iv) Knowledge of existing data sources.

Include subject matter experts to identify appropriate

sources

11 Knowledge Gradient: At Level 2 measurement for improvement is compared to reporting using red, amber, green.

12 Knowledge Gradient: Level 2 understand the importance of involving the correct subject matter experts.

13 Knowledge Gradient: Those at level 2 understand the differences between structure, process, balancing and outcome measures and how these relate to the driver diagram.

14 Knowledge Gradient: Levels 2-4 provides more detail on the 3 faces of Measurement, using examples of measures used for judgement and improvement

14 | P a g e

Step 3:

Design Measurement Plan 3.1 Define Operational Definitions 3.1 (i) Understands the importance of having a

measurement plan

3.5 Establish Baseline Where Possible 3.5 (i) What is a baseline

3.5 Establish Baseline Where Possible 3.5 (ii) The value of a baseline15

3.7 Plan Measurement Reporting 3.7 (ii) Rational sub-grouping and stratification16

3.7 Plan Measurement Reporting 3.7 (vii) Identify stakeholders for reporting

3.8 Ensure Plan is Ethically Sound 3.8 (iii) Understand how to gain informed consent.

Ensure patients and staff are fully aware of all aspects of any

Measurement for Improvement work they are participating

in and provide their consent to participate

3.8 Ensure Plan is Ethically Sound 3.8 (iv) Use of data.

Plan for the secure storage and destruction of data to ensure

confidentiality

Step 5:

Analyse & Display Data 5.2 Understanding Theory 5.2 (ii) Variation & Distribution17

5.2 Understanding Theory 5.2 (v) The importance of trends and patterns in data18

15 Knowledge Gradient: Level 2 addresses the importance of knowing a baseline to ascertain whether changes have resulted in an improvement

16 Knowledge Gradient: Common examples of rational sub- grouping and stratification are used at level 2.

17 Knowledge Gradient: Levels 2, 3 & 4 looks at how to define special cause variation

18 Knowledge Gradient: At levels 2 & 3, trends and patterns which are unlikely to occur by chance are addressed

15 | P a g e

5.2 Understanding Theory 5.2 (vii) The role of targets for accountability19

5.2 Understanding Theory 5.2 (ix) Understanding the limitations of SPC.

Special cause variation can be due to issues with data

collection or differences in case mix

5.3 Construct Control Chart 5.3 (ii) Anatomy of a control chart

Knowledge of the elements of a control chart and the

advantages and disadvantages of including a target line.

5.3 Construct Control Chart 5.3 (iii) The five rules for identifying special cause variation

5.3 Construct Control Chart 5.3 (xi) Understanding the rationale for setting control

limits20

5.5 Construct Run Charts 5.5 (i) Know how to apply the four rules

5.5 Construct Run Charts 5.5 (ii) Median = Centre Line21

5.5 Construct Run Charts 5.5 (iv) Understand the rationale for run chart rules and have

a basic knowledge of how run chart rules are determined

5.5 Construct Run Charts 5.5 (v) Construct a run chart22

19 Knowledge Gradient: At level 2, the distinction between meeting a target and continuous measurement for Improvement is introduced

20 Knowledge Gradient: A basic description of how control limits are set is covered at level 2

21 Knowledge Gradient: At levels 2 Median = Centre Line

22 Knowledge Gradient: At levels 2, 3 & 4 how to construct run charts using median as the centre line is addressed

16 | P a g e

Step 6:

Interpret & Present

Information

6.1 Apply the Five Rules to Control Charts 6.1 (i) How to apply the 5 rules23

6.1 Apply the Five Rules to Control Charts 6.1 (iv) Examples of incorrect interpretation

In interpreting charts, there are a number of common

mistakes that should be avoided

6.2 Review Data with Subject Matter Expert S 6.2 (i) The role of subject matter experts.

Understanding the importance of subject matter expertise in

bringing context to the variation observed

S 6.2 Review Data with Subject Matter Expert S 6.2 (ii) Importance of annotating charts after making

changes and when you observe special cause variation

Step 7:

Act on Findings 7.1 Review your Measurement Results with Respect to

your Aim

7.1 (i) Check measurement has helped identify whether a

change resulted in an improvement

7.2 Distinguishing Signal from Noise 7.2 (i) Overreacting or failing to react. Focuses on

distinguishing between common cause and special cause

variation

23 Knowledge Gradient: At level 2 practical examples of data that breaches each of the 5 rules in C, U, P & I Charts and funnel plots are introduced (only one rule applies to funnel

plots)

17 | P a g e




for Improvement C (e) Promote Measurement to Support Better Decisions C (e) (iii) Measurement for improvement to support better

decisions24

Step 2:

Choose Measures 2.1 Select What to Measure 2.1 (iii) Measure only what matters.


your aim or that have the potential for improvement.25

2.1 Select What to Measure 2.1 (v) Structure, process, balancing and outcome

measures26

2.2 Identify Most Appropriate Data Source 2.2 (i) Identify when quantitative and qualitative measures

are appropriate

Step 3:

Design Measurement Plan 3.1 Define Operational Definitions 3.1 (ii) Define measures

3.1 Define Operational Definitions 3.1 (iii) Ensure consistency of measurement

3.1 Define Operational Definitions 3.1 (v) Counts, rates, percentages, numerator and

denominator27

24 Knowledge Gradient: At levels 3 & 4 specific examples of how measurement can be used to support better decisions are discussed

25 Knowledge Gradient: At levels 3 & 4 practical examples will be introduced, including measures from the HSE service plan.

26 Knowledge Gradient: Those at levels 3 & 4 can identify appropriate structure, process, balancing and outcome measures.

27 Knowledge Gradient: Levels 3 & 4 understand how to discriminate between the suitability of different types of SPC Charts based on the available denominator data

18 | P a g e

3.2 Choose Data Collection Method for Each Measure 3.2 (i) Can design survey questions

3.2 Choose Data Collection Method for Each Measure 3.2 (ii) Can design observation objectives

3.2 Choose Data Collection Method for Each Measure 3.2 (iii) Can design interview questions or topic guide

3.2 Choose Data Collection Method for Each Measure 3.2 (iv) Can design focus group topics and questions

3.2 Choose Data Collection Method for Each Measure 3.2 (v) Knowledge of qualitative data extraction tools if

available

3.2 Choose Data Collection Method for Each Measure 3.2 (vi) Knowledge of methods of manual data entry

3.3 Choose Population 3.3 (i) Completeness vs. sample

3.3 Choose Population 3.3 (ii) Random sampling

3.3 Choose Population 3.3 (iii) Targeted Sampling

3.3 Choose Population 3.3 (v) Knowledge of when to exclude portions of the

population in consultation with subject matter experts

3.4 Determine Frequency and Duration of Data Collection 3.4 (i) Dealing with low numbers28

3.4 Determine Frequency and Duration of Data Collection 3.4 (iv) Sustainability of data collection

3.4 Determine Frequency and Duration of Data Collection 3.4 (v) Balancing frequency with subgroup size

3.5 Establish Baseline Where Possible 3.5 (iii) The value of a baseline29

28 Knowledge Gradient: Level 3 introduces the concept of dealing with low numbers.

19 | P a g e

3.6 Decide Which Analytical Tool to Use 3.6 (i) Types of data.

Definition of continuous categorical, and qualitative data

3.6 Decide Which Analytical Tool to Use 3.6 (iii) Visual display.

Knowledge of how to choose appropriate charts when

displaying information30

3.6 Decide Which Analytical Tool to Use 3.6 (iv) Recommended chart types (SPC, Run & funnel plots).

Which SPC chart to use and why31

3.7 Plan Measurement Reporting 3.7 (i) The Impact of frequency of collection on time to know

if a change has occurred

3.7 Plan Measurement Reporting 3.7 (iii) Rational sub-grouping and stratification32

3.7 Plan Measurement Reporting 3.7 (v) Effective reporting.

Use ISBAR or other structured communication tool to

present information

3.7 Plan Measurement Reporting 3.7 (vi) Frequency of reporting.

Decide on the frequency of reporting, appropriate to the

receiver of the information

3.8 Ensure Plan is Ethically Sound 3.8 (i) Understand when ethical approval may be required.

Seek ethical approval from relevant Ethics Committee if

29 Knowledge Gradient: Levels 3& 4 examines what makes a good baseline and what to do in the absence of a baseline

30 Knowledge Gradient: Levels 3 & 4 discuss examples of SPC charts, bar charts, scatter plots, histogram, box plot & pareto charts

31Knowledge Gradient: Level 3 discusses examples of when you would use I, P, C, U, T & G.

32 Knowledge Gradient: How to subgroup and stratify are covered at levels 3 & 4

20 | P a g e

required

3.9 Prepare Topic Guide/ Questions for Interviews or

Focus Groups

3.9 (i) Types of questions for interviews and focus groups33

3.10 Establish Objectives of Observation 3.10 (i) Principles of observation34

3.11 Compose Survey Questions 3.11 (i) Types of questions for surveys35

3.11 Compose Survey Questions 3.11 (i) Types of scales

Step 5:

Analyse & Display Data 5.2 Understanding Theory 5.2 (viii) The role of targets for accountability36

5.2 Understanding Theory 5.2 (x) Understand causality and correlation (including using

a scatter plot)

5.3 Construct Control Chart 5.3 (iv) I, P, C, U, T, X-bar/S Charts

Knowledge of the criteria to use the relevant chart

5.3 Construct Control Chart 5.3 (v) How to construct a P, C, U, I or X-bar/S Chart

Knowledge of how control limits are calculated

5.3 Construct Control Chart 5.3 (viii) Issues with large denominators and over

dispersion37

33 Knowledge Gradient: Level 3 introduces different types of questions (open and closed)

34 Knowledge Gradient: How to conduct observation and record field notes are covered in level 3

35 Knowledge Gradient: Level 3 introduces different types of questions (open and closed)

36 Knowledge Gradient: At level 3 & 4, data viewed in relation to achieving a target is compared with data viewed with the intention of understanding variation

21 | P a g e

5.3 Construct Control Chart 5.3 (x) Deviation (Sigma).

The type of data dictates the type of control chart and in

turn, dictates how sigma is calculated

5.3 Construct Control Chart 5.3 (xii) Understanding the rationale for setting control

limits38

5.4 Construct Pareto Charts 5.4 (i) Construct a Pareto chart

5.5 Construct Run Charts 5.5 (iii) Median = Centre Line39

5.6 Construct Frequency Plots 5.6 (i) How to construct a Bar Chart

5.6 Construct Frequency Plots 5.6 (ii) How to construct a Histogram

5.6 Construct Frequency Plots 5.6 (iii) How to construct a Pareto Chart

5.7 Construct Scatter Plots 5.7 (i) How to construct a scatter plot

5.8 Analyse Survey Responses 5.8 (i) Knowledge of quantitative data analysis.

Knowledge of and access to quantitative software tools such

as survey monkey and statistical software packages.

5.9 Analyse Qualitative Data 5.9 (i) Ability to analyse qualitative data.

Knowledge of thematic analysis using comparison,

identifying similarities and differences across data.

37 Knowledge Gradient: How to deal with large denominators and over dispersion are covered at level 3: Basic examples of P’ and U’. Include I chart as an alternative to C chart

where the count is large. 38

Knowledge Gradient: At levels 3 & 4 the rationale for setting control limits is explained in more depth 39

Knowledge Gradient: At levels 3 & 4 instances where the mean may be used are addressed

22 | P a g e

Knowledge of and access to qualitative data analysis

software tools such as NVivo.

Step 6:

Interpret & Present

Information

6.1 Apply the Five Rules to Control Charts 6.1 (ii) How to apply the rules40

6.1 Apply the Five Rules to Control Charts 6.1 (iii) Recalculating the centre line and control limits41



the Aim

6.3 (i) Balancing measures. Understanding and Appreciation

of a System.

When optimising one process, it is important to use

measurement to see what other parts of the system are

affected and how they are affected.



the Aim

6.3 (ii) Triangulation verses validation

6.4 Ensure Patient Voice is Recognised and Maintained in

the Family of Measures

6.4 (i) Capturing the patient voice. Through the use of

Qualitative Methods

6.5 Action to address Special verses Common Causes 6.5 (i) Two distinct kinds of variation should lead to distinct

kinds of action.

Where normal variation is observed system wide

improvement is necessary, where special cause variation is

observed it is necessary to identify why and act on that

40 Knowledge Gradient: At levels 3 & 4, T and G Charts are added

41 Knowledge Gradient: Levels 3 & 4 addresses the issue of recalculation having applied the rules and emphasizes the role of subject matter experts in making decisions about

where and when to recalculate the centre line and control limits

23 | P a g e

specifically

6.6 Interpret Qualitative Findings 6.6 (i) Uses and limitations of qualitative data

Step 7:

Act on Findings 7.3 Use Measurement Results to Identify Further Areas

for Improvement

7.3 (i) Learning from PDSA cycles

7.3 Use Measurement Results to Identify Further Areas

for Improvement

7.3 (ii) Understanding the opportunity for improvement that

special cause variation represents

7.4 Review and Share Measurement Results with

Stakeholders

7.4 (i) Appropriate measurement reporting for different

audiences.

Addresses the importance of the right information going to

the right people

7.5 Plan for Continued Measurement 7.5 (i) Sustaining measurement for improvement.

The importance of ensuring that, where appropriate,

measurement is continued after the project is completed.

24 | P a g e



Step 3:

Design Measurement Plan 3.4 Determine Frequency and Duration of Data Collection 3.4 (ii) Dealing with low numbers42

3.6 Decide Which Analytical Tool to Use 3.6 (v) Recommended chart types (SPC, & funnel plots).

Which SPC chart to use and why43

3.7 Plan Measurement Reporting 3.7 (iv) Rational sub-grouping and stratification44

3.9 Prepare Topic Guide/ Questions for Interviews or

Focus Groups

3.9 (ii) Types of questions for interviews and focus groups45

3.10 Establish Objectives of Observation 3.10 (ii) Principles of observation46

3.11 Compose Survey Questions 3.11 (ii) Types of questions for surveys47

Step 5:

Analyse & Display Data 5.2 Understanding Theory 5.2 (iii) Variation & Distribution48

5.2 Understanding Theory 5.2 (vi) The importance of trends and patterns in data49

42 Knowledge Gradient: Level 4 addresses the importance of ensuring that a link between actual data and transformed data is maintained

43 Knowledge Gradient: Level 4 discusses P’, U’ and Cusum charts

44Knowledge Gradient: Level 4 has the skills to apply different ways of sub- grouping and stratification including criteria for deciding if rational subgrouping or stratification is

appropriate 45

Knowledge Gradient: Level 4 addresses question wording, flow, context 46

Knowledge Gradient: How to design observation aims and objectives are covered in level 4 47

Knowledge Gradient: Level 4 addresses question wording, flow, context 48

Knowledge Gradient: At level 4 how SPC deals with data that is not distributed normally

25 | P a g e

5.3 Construct Control Chart 5.3 (vi) How to construct a CUSUM chart. Looks at examples

of CUSUM & examines how control limits are calculated for

CUSUM

5.3 Construct Control Chart 5.3 (vii) How to construct a 'G Chart' or a 'T chart'.

Knowledge of the criteria to use the relevant chart

S 5.3 Construct Control Chart 5.3 (ix) Issues with large denominators and over dispersion50

S 5.5 Construct Run Charts 5.5 (vi) Construct a run chart.

Understand the limitations of run charts in healthcare

applications51

49 Knowledge Gradient: Level 4 addresses seasonality including who makes the decision to adjust for seasonality

50 Knowledge Gradient: Level 4 includes how the control limits are calculated for P’ and U’ and includes specific examples of where prime charts are used

51 Knowledge Gradient: At level 4 the issues with using run chart rules as described in the 2011 paper published in BMJ Quality and Safety (Perla et al.) are discussed

Summary

The Measurement for Improvement Curriculum has been developed by QID Measurement for Improvement,

and identifies the essential components that should be included in Measurement for Improvement (MFI)

training in Ireland. This document - version one of curriculum - provides a guide of the key content areas,

tasks, knowledge and skills, and knowledge gradients required across four levels of expertise in MFI.

Version One will be applied, tested and refined through providing training to frontline and other HSE staff by

QID Measurement for Improvement throughout 2017. On-line training for level one MFI level of expertise is

currently in development. Version 2 of the curriculum will incorporate the learning gained from the

application of version one, along with feedback from those receiving training. Version 2 is intended for use

by others within the HSE who provide MFI training. Furthermore, it will form the basis of commissioning

external providers to deliver training in MFI on behalf of the HSE.

27 | P a g e

Bibliography

Corbin, J. and Strauss, A. (2008) Basics of Qualitative Research. London: Sage

Davidge, M., Holmes, M., Shaw, A., Shouls S. and Tite, M. (2017) Guide to Measurement for Improvement.

NHS Elect. Available at:

https://www.nhselect.nhs.uk/uploads/files/1/Resource/Service%20Transformation%202016/NHS%20Elect-

Measurement%20for%20Improvement-Feb17.pdf

Matthews, B. and Ross, L. (2010) Research Methods. A Practical Guide for the Social Sciences. Harlow:

Longman.

Provost, L. and Murray S. (2011) The Healthcare Data Guide: learning from data for improvement. San

Francisco: Jossey-Bass

Quality Improvement Division, Health Service Executive (2016) Framework for Improving Quality in our

Health Service. Part 1: Introducing the Framework. Available at


Silverman, D. (2013) Doing Qualitative Research. London: Sage

https://www.nhselect.nhs.uk/uploads/files/1/Resource/Service%20Transformation%202016/NHS%20Elect-Measurement%20for%20Improvement-Feb17.pdf

https://www.nhselect.nhs.uk/uploads/files/1/Resource/Service%20Transformation%202016/NHS%20Elect-Measurement%20for%20Improvement-Feb17.pdf


28 | P a g e

Glossary

Term Operational Definition Baseline Measurement of a system prior to the introduction of a change or intervention

Categorical Data Data that can be organised into categories

Continuous Data Data that can take any value within a range

Counts Query if we need any of these

Driver Diagram A tool that helps to break down an aim statement into manageable parts.

Family of Measures A group of outcome, process and balancing measures that together can facilitate better understanding of the impact of changes.

Focus Groups A qualitative data collection method that brings together a group of people to take part in a facilitated discussion on a specific topic or set of open questions.

Funnel Plot A funnel plot is a type of SPC chart in which values are plotted by size (smallest to largest which results in a funnel shape) to show the variation among different organisations (e.g. hospitals).

Interview Questions Interview questions are a pre-prepared list of open questions used in qualitative interviews or focus groups which the interviewer or researcher asks the participants.

ISBAR A communication tool developed for clinical handover that comprises specific elements based on Identification, Situation, Background, Assessment, and Recommendation.


The analysis and presentation of qualitative and quantitative data in a format that allows us to:

Identify opportunities for improvement Demonstrate when a change has resulted in an improvement

Measurement Plan A tool that is used to plan, record and agree measurement activities.

Observation A qualitative method whereby participants are observed, watched or shadowed in a natural setting (in their work setting for example) by the researcher who notes events and interactions as they occur.

Operational Definitions A clear and detailed description of a measure with the intention of ensuring consistency of data collection and analysis.

Pareto Chart A tool consisting of a bar chart and a line chart for visualising the frequency with which events occur in order to focus on areas of improvement with the greatest impact.

PDSA Cycles The Plan, Do, Study, Act Cycle is a framework for an efficient trial-and- learning methodology used as part of the Model for Improvement. The cycle begins with a plan and ends with action taken based on the learning gained from each phase of the cycle. The four steps consist of planning the details of the test and making predictions about the outcomes (Plan), conducting the plan and collecting data (do), comparing predictions to the data collected (Study), and taking action based on the new knowledge (Act).

Population Population refers to the total number of cases that that can be included as research subjects

Quality Improvement Quality improvement (QI) is the combined and unceasing efforts of everyone - healthcare professionals, patients and their families, researchers, commissioners, providers and educators - to make the changes that will

29 | P a g e

lead to ● better patient outcomes ● better experience of care ● continued development and supporting of staff in delivering quality care

Qualitative Data Qualitative data is non-numerical information that can be captured through a variety of qualitative methods including interviews, focus groups, observations and written documents.

Qualitative Interviews A qualitative data collection method where there is direct communication between an interviewer/researcher and a participant. This can occur face-to-face, on the telephone or through internet video services. Interviews can be structured (whereby each participant is asked the same list of questions), semi-structured (the interviewer/researcher has flexibility to reword the question or topics and to pursue new issues as they emerge) or unstructured (no pre-prepared topic guide or structured questions)

Qualitative Methods Qualitative methods are used to collect qualitative information or data. Qualitative methods include structured or unstructured in-depth interviews, focus groups, participant observation, documentary analysis and visual methods

Quantitative Data Quantitative data is data that is structured and can be represented numerically

Random Sampling A sample selected from a population where every case has an equal chance of being included in the sample and the composition of the sample cannot be predicted

Run Chart A run chart is a graphical display of data plotted in time order. It usually includes a centre line based on the median of the values.

Stakeholders Stakeholders are a person, group, organisation, or system who affects or can be affected by an organisation’s actions. Health service provider’s stakeholders, for example, include its patients, employees, medical staff, government, insurers, industry and the community.

Statistical Process Control (SPC) Chart

An SPC chart consists of values plotted in order, usually over time (weeks, months etc). It includes a centre line based on the average of the values. It also includes upper and lower control limits based on statistical calculations (3 sigma deviations from the average). SPC charts are used as a tool to distinguish between special and common causes of variation.

Stratification Stratification is a method of organising a population in order to improve the representativeness of a sample

Subject Matter Experts In the context of healthcare subject matter experts include staff and service users with knowledge of a specific healthcare system or service.

Survey A set of questions with a set range of answers in a format that enables standardised, relatively structured data to be gathered about each of a (usually) large number of cases which can be represented numerically. Some surveys include open questions which allow the respondent to answer the question in their own words.

Survey Monkey Survey Monkey is an online survey platform which allows you to customise your survey questions and to monitor and analyse your responses.

Targeted Sampling Target or purposeful sampling are a sample of selected cases that will best enable the researcher to explore the research questions in depth

30 | P a g e

Thematic Analysis Thematic analysis is used in qualitative research as a process of identifying and interpreting key themes or ideas in raw data

Time Series Data that are displayed in time series are ordered chronologically, e.g. by day, month, year

Topic Guide A topic guide or interview guide is a pre-prepared list of topics or open questions, key points or prompts used in qualitative interviews and focus groups helping the interviewer or researcher to remember the issues and questions to introduce, reminds them to probe and follow up on the participant’s responses. Topic guides can be used across a series of interviews or focus groups.

Triangulation Triangulation is a measure of research quality; where different types of data are collected to address the same question or aim, each data set can be used to check the findings of the other.

Voice of the Patient Patients’ (and their families’) perspectives, opinions and views

31 | P a g e

Appendix I: Pre-requisites for Measurement for Improvement

Although knowledge areas and tools related to more general aspects of Quality Improvement are not

specifically covered in the measurement for improvement curriculum, three such broader knowledge areas

are referred to in this document. It is suggested that prior knowledge of these areas would be beneficial

when undertaking MFI training at the relevant level. The following table lists the three Quality Improvement

knowledge areas and the corresponding MFI curriculum levels for which they are relevant. It is envisaged

that learners will receive some training in broader Quality Improvement to compliment their training in

measurement for improvement.

QI Knowledge Area Relevance to the Measurement for

Improvement Curriculum

Definition of an Aim Every Quality Improvement Initiative should have an aim. The aim is a tool to answer the first question of the Model for Improvement.

At level 1 of the Measurement for Improvement curriculum, the aim is discussed in relation to the Model for Improvement.

Definition of a SMART Aim Building on the definition of an aim, the concept of a SMART Aim is introduced.

At level 2 of the Measurement for Improvement curriculum, the concept of a SMART Aim is introduced and the importance of measurement as a way of knowing if the aim has been achieved.

Develop a Driver Diagram Driver Diagrams are introduced as a tool for building and testing theories. Skills to complete a driver diagram are demonstrated.

Levels 3 & 4 of the Measurement for Improvement curriculum address the importance of how the Aim relates to the Driver Diagram and the measurement strategy.

32 | P a g e

Appendix II: Task listed by level

Where a task is listed at more than one MFI level of expertise, a knowledge gradient is applied whereby the

task is covered in increasing detail as the learner progresses through the relevant levels of expertise. The

information in this Appendix is described in more detail in Appendix III.

Context for Measurement for Improvement

C 1.1 Definition of Measurement for Improvement

Level 1

●

Level 2

●

Level 3

●

Level 4

●

C 1.2 Different Applications of Measurement for Improvement

Level 1

●

Level 2

●

Level 3

●

Level 4

●

C 1.3 Basic Principles of Measurement for Improvement

Level 1

●

Level 2

●

Level 3

●

Level 4

●

C 1.4 Defining the 7 steps towards effective measurement for Improvement

Level 1

●

Level 2

●

Level 3

●

Level 4

●

C 1.5 Promote Measurement to Support Better Decisions

Level 1

●

Level 2

●

Level 3

●

Level 4

●

Step 1 of the Seven Steps to Effective Measurement for Improvement: Is there an opportunity to improve?

S 1.1 Determine if there is an opportunity to Improve

Level 1

●

Level 2

●

Level 3

●

Level 4

●

Step 2 of the Seven Steps to Effective Measurement for Improvement: Choose Measures

S 2.1 Select What to Measure

Level 1

●

S 2.2 Identify Most Appropriate Data Source Level

3

●

Level 4

●

S 2.3 Consider Inclusion of Qualitative Measures

Level 1

●

Level 2

●

Level 3

●

Level 4

●

33 | P a g e

Step 3 of the 7 Steps to Effective Measurement for Improvement: Design Measurement Plan

S 3.1 Define Operational Definitions Level

2

●

Level 3

●

Level 4

●

S 3.2 Choose Data Collection Method for Each Measure Level

3

●

Level 4

●

S 3.3 Choose Population Level

3

●

Level 4

●

S 3.4 Determine Frequency and Duration of Data Collection

Level 3

●

S 3.5 Establish Baseline Where Possible Level

2

●

Level 3

●

Level 4

●

S 3.6 Decide Which Analytical Tool to Use Level

3

●

Level 4

●

S 3.7 Plan Measurement Reporting Level

3

●

Level 4

●

S 3.8 Ensure Plan is Ethically Sound Level

3

●

Level 4

●

S 3.9 Prepare Topic Guide/ Questions for Interviews or Focus Groups

Level 3

●

S 3.10 Establish Objectives of Observation Level

3

●

S 3.11 Compose Survey Questions Level

3

●

34 | P a g e

Step 4 of the 7 Steps to Effective Measurement for Improvement: Collect Data

S 4.1 Collect Data According to Measurement and Collection Plans

Level 1

●

Level 2

●

Level 3

●

Level 4

●

S 4.2 Record Data Accurately

Level 1

●

Level 2

●

Level 3

●

Level 4

●

S 4.3 Record Details of PDSA Cycles

Level 1

●

Level 2

●

Level 3

●

Level 4

●

Step 5 of the 7 Steps to Effective Measurement for Improvement: Analyse and Display Data

S 5.1 Theory of SPC (SPC includes Run charts, Control Charts and Funnel plots)

Level 1

●

Level 2

●

Level 3

●

Level 4

●

S 5.2 Understanding Theory

Level 1

●

S 5.3 Construct Control Chart

Level 1

●

Level 2

●

Level 3

●

Level 4

●

S 5.4 Construct Pareto Charts Level

3

●

Level 4

●

S 5.5 Construct Run Charts Level

2

●

Level 3

●

Level 4

●

S 5.6 Construct Frequency Plots Level

3

●

Level 4

●

S 5.7 Construct Scatter Plots Level

3

●

Level 4

●

S 5.8 Analyse Survey Responses Level

3

●

Level 4

●

S 5.9 Analyse Qualitative Data Level

3

●

Level 4

●

35 | P a g e

Step 6 of the 7 Steps to Effective Measurement for Improvement: Interpret and Present Information

S 6.1 Apply the Five Rules to Control Charts Level

2

●

S 6.2 Review Data with Subject Matter Expert Level

2

●

Level 3

●

Level 4

●

S 6.3 Combine a Number of Measures to Give Overall Picture of the Aspect of Quality of Care with Reference to the Aim

Level

3

●

Level 4

●

S 6.4 Ensure Patient Voice is Recognised and Maintained in the Family of Measures

Level 3

●

Level 4

●

S 6.5 Action to address Special verses Common Causes Level

3

●

Level 4

●

S 6.6 Interpret Qualitative Findings Level

3

●

Level 4

●

Step 7 of the 7 Steps to Effective Measurement for Improvement: Act On Findings

S 7.1 Review your Measurement Results with Respect to your Aim

Level 2

●

Level 3

●

Level 4

●

S 7.2 Distinguishing Signal from Noise Level

2

●

Level 3

●

Level 4

●

S 7.3 Use Measurement Results to Identify Further Areas for Improvement

Level 3

●

Level 4

●

S 7.4 Review and Share Measurement Results with Stakeholders

Level 3

●

Level 4

●

S 7.5 Plan for Continued Measurement Level

3

●

Level 4

●

Appendix III: Complete Curriculum

Context for Measurement for Improvement

Task Knowledge and Skill Area Knowledge Gradient Applies to the following MFI Levels of Expertise:

C (a) Definition of Measurement for

Improvement

C (a) Understand what constitutes Measurement for

Improvement

No gradient Level 1

●

Level 2

●

Level 3

●

Level 4

●

C (b) Different Applications of


C (b) Understand the difference between

Measurement for Improvement for PDSA cycles,

quality improvement projects and at organisation

level

No gradient Level 1

●

Level 2

●

Level 3

●

Level 4

●

C (c) Basic Principles of Measurement

for Improvement

C (c) Understand the basic principles of


No gradient Level 1

●

Level 2

●

Level 3

●

Level 4

●

C (d) Defining the 7 steps towards

effective measurement for

Improvement

C (d) Understand the basic elements of the '7 steps

Towards Effective Measurement for Improvement'

No gradient Level 1

●

Level 2

●

Level 3

●

Level 4

●

C (e) Promote Measurement to

Support Better Decisions

C (e) (i) Valuing measurement for improvement No gradient Level 1

●

Level 2

●

Level 3

●

Level 4

●



C (e) (ii) Measurement for improvement to support

better decisions

At Level 2 measurement

for improvement is

compared to reporting

using red, amber, green

Level 2

●



C (e) (iii) Measurement for improvement to support

better decisions

At levels 3 & 4 specific

examples of how

measurement can be used

to support better decisions

are discussed

Level 3

●

Level 4

●

37 | P a g e

Step 1: Is there an Opportunity to Improve?


1.1 Determine if there is an

opportunity to Improve

1.1 (i) Making full use of available information.

Focuses on using both quantitative and qualitative

information already available to know if there is an

opportunity to improve

No gradient Level 1

●

Level 2

●

Level 3

●

Level 4

●

1.1 Determine if there is an

opportunity to Improve

1.1 (ii) Making full use of Subject Matter Expert

knowledge.

Focuses on the understanding that Subject Matter

Experts have of where there are potential

opportunities to improve

No gradient Level 1

●

Level 2

●

Level 3

●

Level 4

●

38 | P a g e

Step 2: Choose Measures


2.1 Select What to Measure 2.1 (i) Measure only what matters.

Focuses on measuring things that will have an

impact on your aim or your project, or that have the

potential for improvement.

Level 1 introduces the

concept of measuring only

what matters.

Level 1

●

2.1 Select What to Measure 2.1 (ii) Measure only what matters.


impact on your aim or that have the potential for

improvement.

Level 2 understand the

importance of involving the

correct subject matter

experts.

Level 2

●

2.1 Select What to Measure 2.1 (iii) Measure only what matters.


impact on your aim or that have the potential for

improvement.

At levels 3 & 4 practical

examples will be

introduced, including

measures from the HSE

service plan.

Level 3

●

Level 4

●

2.1 Select What to Measure 2.1 (iv) Structure, process, balancing and outcome

measures

Those at level 2 understand

the differences between

structure, process,

balancing and outcome

measures and how these

relate to the driver

diagram.

Level 2

●

2.1 Select What to Measure 2.1 (v) Structure, process, balancing and outcome

measures

At levels 3 & 4 can identify

appropriate structure,

process, balancing and

outcome measures.

Level 3

●

Level 4

●

39 | P a g e

2.2 Identify Most Appropriate Data

Source

2.2 (i) Identify when quantitative and qualitative

measures are appropriate

No gradient Level 3

●

Level 4

●


Source

2.2 (ii) Judgement, research and improvement

measures.

This knowledge area describes the differences

between Measurement for Judgement, for

Improvement and for research.

Level 1 introduces the basic

principles of the 3 types of

Measurement)

Level 1

●


Source

2.2 (iii) Judgement, research and improvement

measures.

This knowledge area describes the differences

between Measurement for Judgement and

Measurement for Improvement.

Levels 2-4 provides more

detail on the 3 faces of

Measurement, using

examples of measures used

for judgement and

improvement

Level 2

●

Level 3

●

Level 4

●


Source

2.2 (iv) Knowledge of existing data sources.

Include subject matter experts to identify

appropriate sources

No gradient Level 2

●

Level 3

●

Level 4

●

2.3 Consider Inclusion of Qualitative

Measures

2.3 (i) Recognises the importance of including the

voice of patient

No gradient Level 1

●

Level 2

●

Level 3

●

Level 4

●

2.3 Consider Inclusion of Qualitative

Measures

2.3 (ii) Understand the value of qualitative data in


No gradient Level 1

●

Level 2

●

Level 3

●

Level 4

●

40 | P a g e

Step 3: Design Measurement Plan


3.1 Define Operational Definitions 3.1 (i) Understands the importance of having a measurement plan

No gradient Level 2 ●

Level 3 ●

Level 4 ●

3.1 Define Operational Definitions 3.1 (ii) Define measures No gradient Level 3 ●

Level 4 ●

3.1 Define Operational Definitions 3.1 (iii) Ensure consistency of measurement No gradient Level 3 ●

Level 4 ●

3.1 Define Operational Definitions 3.1 (iv) Counts, rates, percentages, numerator and denominator

Those at level 1 and 2 understand counts, rates, percentages, numerator and denominator, which are defined with basic examples

Level 1 ●

Level 2 ●

3.1 Define Operational Definitions 3.1 (v) Counts, rates, percentages, numerator and denominator

Levels 3 & 4 understand how to discriminate between the suitability of different types of SPC Charts based on the available denominator data

Level 3 ●

Level 4 ●

3.2 Choose Data Collection Method for Each Measure

3.2 (i) Can design survey questions No gradient Level 3 ●

Level 4 ●


3.2 (ii) Can design observation objectives No gradient Level 3 ●

Level 4 ●


3.2 (iii) Can design interview questions or topic guide No gradient Level 3 ●

Level 4 ●

41 | P a g e


3.2 (iv) Can design focus group topics and questions No gradient Level 3 ●

Level 4 ●


3.2 (v) Knowledge of qualitative data extraction tools if available


Level 4 ●


3.2 (vi) Knowledge of methods of manual data entry No gradient Level 3 ●

Level 4 ●

3.3 Choose Population 3.3 (i) Completeness vs. sample No gradient Level 3 ●

Level 4 ●

3.3 Choose Population 3.3 (ii) Random sampling No gradient Level 3 ●

Level 4 ●

3.3 Choose Population 3.3 (iii) Targeted Sampling No gradient Level 3 ●

Level 4 ●

3.3 Choose Population 3.3 (iv) Just enough data. The principle of only collecting enough data for the purpose at hand


Level 2 ●

Level 3 ●

Level 4 ●

3.3 Choose Population 3.3 (v) Knowledge of when to exclude portions of the population in consultation with subject matter experts


Level 4 ●

3.4 Determine Frequency and Duration of Data Collection

3.4 (i) Dealing with low numbers Level 3 introduces the concept of dealing with low numbers.

Level 3 ●


3.4 (ii) Dealing with low numbers Level 4 addresses the importance of ensuring that a link between actual data and transformed data is maintained

Level 4 ●

42 | P a g e


3.4 (iii) Advantages of looking at data over time No gradient Level 1 ●

Level 2 ●

Level 3 ●

Level 4 ●


3.4 (iv) Sustainability of data collection No gradient Level 3 ●

Level 4 ●


3.4 (v) Balancing frequency with subgroup size No gradient Level 3 ●

Level 4 ●

3.5 Establish Baseline Where Possible 3.5 (i) What is a baseline No gradient Level 2 ●

Level 3 ●

Level 4 ●

3.5 Establish Baseline Where Possible 3.5 (ii) The value of a baseline Level 2 addresses the importance of knowing a baseline to ascertain whether changes have resulted in an improvement

Level 2 ●

3.5 Establish Baseline Where Possible 3.5 (iii) The value of a baseline Levels 3& 4 examines what makes a good baseline and what to do in the absence of a baseline

Level 3 ●

Level 4 ●

3.6 Decide Which Analytical Tool to Use

3.6 (i) Types of data. Definition of continuous categorical, and qualitative data


Level 4 ●


3.6 (ii) Visual display. Awareness that there are different charts for different types of data

Levels 1 & 2 discuss examples of basic charts that are commonly used

Level 1 ●

Level 2 ●

43 | P a g e


3.6 (iii) Visual display. knowledge of how to choose appropriate charts when displaying information

Levels 3 & 4 discuss examples of SPC charts, bar charts, scatter plots, histogram, box plot & pareto charts

Level 3 ●

Level 4 ●


3.6 (iv) Recommended chart types (SPC, Run & funnel plots). Which SPC chart to use and why

Level 3 discusses examples of when you would use I, P, C, U, T & G.

Level 3 ●


3.6 (v) Recommended chart types (SPC, & funnel plots). Which SPC chart to use and why

Level 4 discusses P’, U’ and Cusum charts

Level 4 ●

3.7 Plan Measurement Reporting 3.7 (i) The Impact of frequency of collection on time to know if a change has occurred


Level 4 ●

3.7 Plan Measurement Reporting 3.7 (ii) Rational sub-grouping and stratification Common examples of rational sub- grouping and stratification are used at level 2.

Level 2 ●

3.7 Plan Measurement Reporting 3.7 (iii) Rational sub-grouping and stratification How to subgroup and stratify are covered at levels 3 & 4

Level 3 ●

Level 4 ●

3.7 Plan Measurement Reporting 3.7 (iv) Rational sub-grouping and stratification Level 4 has the skills to apply different ways of sub- grouping and stratification including criteria for deciding if rational subgrouping or stratification is appropriate

Level 4 ●

44 | P a g e

3.7 Plan Measurement Reporting 3.7(v) Effective reporting. Use ISBAR or other structured communication tool to present information


Level 4 ●

3.7 Plan Measurement Reporting 3.7 (vi) Frequency of reporting. Decide on the frequency of reporting, appropriate to the receiver of the information


Level 4 ●

3.7 Plan Measurement Reporting 3.7 (vii) Identify stakeholders for reporting No gradient Level 2 ●

Level 3 ●

Level 4 ●

3.8 Ensure Plan is Ethically Sound 3.8 (i) Understand when ethical approval may be required. Seek ethical approval from relevant Ethics Committee if required


Level 4 ●

3.8 Ensure Plan is Ethically Sound 3.8 (ii) Patient & staff confidentiality. Ensure patients and staff are assured full confidentiality and anonymity


Level 2 ●

Level 3 ●

Level 4 ●

3.8 Ensure Plan is Ethically Sound 3.8 (iii) Understand how to gain informed consent. Ensure patients and staff are fully aware of all aspects of any Measurement for Improvement work they are participating in and provide their consent to participate


Level 3 ●

Level 4 ●

3.8 Ensure Plan is Ethically Sound 3.8 (iv) Use of data. Plan for the secure storage and destruction of data to ensure confidentiality


Level 3 ●

Level 4 ●

3.9 Prepare Topic Guide/ Questions for Interviews or Focus Groups

3.9 (i) Types of questions for interviews and focus groups

Level 3 introduces different types of questions (open and closed)

Level 3 ●

45 | P a g e

3.9 Prepare Topic Guide/ Questions for Interviews or Focus Groups

3.9 (ii) Types of questions for interviews and focus groups

Level 4 addresses question wording, flow, context

Level 4 ●

3.10 Establish Objectives of Observation

3.10 (i) Principles of observation How to conduct observation and record field notes are covered in level 3

Level 3 ●

3.10 Establish Objectives of Observation

3.10 (ii) Principles of observation How to design observation aims and objectives are covered in level 4

Level 4 ●

3.11 Compose Survey Questions 3.11 (i) Types of questions for surveys Level 3 introduces different types of questions (open and closed)

Level 3 ●

3.11 Compose Survey Questions 3.11 (ii) Types of questions for surveys Level 4 addresses question wording, flow, context

Level 4 ●

3.11 Compose Survey Questions 3.11 (i) Types of scales No gradient Level 3 ●

Level 4 ●

46 | P a g e

Step 4: Collect Data


4.1 Collect Data According to Measurement and Collection Plans

4.1 (i) Ensure fidelity and consistency of measurement plan

No gradient Level 1

● Level 2

● Level 3

● Level 4

●

4.2 Record Data Accurately 4.2 (i) Principles of good data collection No gradient Level 1

● Level 2

● Level 3

● Level 4

●

4.3 Record Details of PDSA Cycles 4.3 (i) The knowledge of documenting PDSA cycles No gradient Level 1

● Level 2

● Level 3

● Level 4

●

47 | P a g e

Step 5: Analyse and Display Data


5.1 Theory of SPC (SPC includes Run charts, Control Charts and Funnel plots)

5.1 (i) Introduce the theory of SPC including its basis in statistics


Level 2 ●

Level 3 ●

Level 4 ●

5.2 Understanding Theory 5.2 (i) Variation & Distribution For level 1 variation at a basic level is introduced

Level 1 ●

5.2 Understanding Theory 5.2 (ii) Variation & Distribution Levels 2, 3 & 4 looks at how to define special cause variation

Level 2 ●

Level 3 ●

Level 4 ●

5.2 Understanding Theory 5.2 (iii) Variation & Distribution At level 4 how SPC deals with data that is not distributed normally

Level 4 ●

5.2 Understanding Theory 5.2 (iv) The importance of trends and patterns in data

At level 1 increasing vs. decreasing trends in data are discussed

Level 1 ●

5.2 Understanding Theory 5.2 (v) The importance of trends and patterns in data At levels 2 & 3, trends and patterns which are unlikely to occur by chance are addressed

Level 2 ●

Level 3 ●

5.2 Understanding Theory 5.2 (vi) The importance of trends and patterns in data

Level 4 addresses seasonality including who makes the decision to adjust for seasonality

Level 4 ●

5.2 Understanding Theory 5.2 (vii) The role of targets for accountability At level 2, the distinction between meeting a target and continuous measurement for Improvement is introduced

Level 2 ●

48 | P a g e

5.2 Understanding Theory 5.2 (viii) The role of targets for accountability At level 3 & 4, data viewed in relation to achieving a target is compared with data viewed with the intention of understanding variation

Level 3 ●

Level 4 ●

5.2 Understanding Theory 5.2 (ix) Understanding the limitations of SPC. Special cause variation can be due to issues with data collection or differences in case mix


Level 3 ●

Level 4 ●

5.2 Understanding Theory 5.2 (x) Understand causality and correlation (including using a scatter plot)


Level 4 ●

5.3 Construct Control Chart 5.3 (i) Mean & median No gradient Level 1 ●

Level 2 ●

Level 3 ●

Level 4 ●

5.3 Construct Control Chart 5.3 (ii) Anatomy of a control chart Knowledge of the elements of a control chart and the advantages and disadvantages of including a target line.


Level 3 ●

Level 4 ●

5.3 Construct Control Chart 5.3 (iii) The five rules for identifying special cause variation


Level 3 ●

Level 4 ●

5.3 Construct Control Chart 5.3 (iv) I, P, C, U, T, X-bar/S Charts Knowledge of the criteria to use the relevant chart


Level 4 ●

5.3 Construct Control Chart 5.3 (v) How to construct a P, C, U, I or X-bar/S Chart Knowledge of how control limits are calculated


Level 4 ●

5.3 Construct Control Chart 5.3 (vi) How to construct a CUSUM chart. Looks at examples of CUSUM & examines how control limits are calculated for CUSUM


5.3 Construct Control Chart 5.3 (vii) How to construct a 'G Chart' or a 'T chart'. Knowledge of the criteria to use the relevant chart


49 | P a g e

5.3 Construct Control Chart 5.3 (viii) Issues with large denominators and over dispersion

How to deal with large denominators and over dispersion are covered at level 3: Basic examples of P’ and U’. Include I chart as an alternative to C chart where the count is large.

Level 3 ●

5.3 Construct Control Chart 5.3 (ix) Issues with large denominators and over dispersion

Level 4 includes how the control limits are calculated for P’ and U’ and includes specific examples of where prime charts are used

Level 4 ●

5.3 Construct Control Chart 5.3 (x) Deviation (Sigma). The type of data dictates the type of control chart and in turn, dictates how sigma is calculated


Level 4 ●

5.3 Construct Control Chart 5.3 (xi) Understanding the rationale for setting control limits

A basic description of how control limits are set is covered at level 2

Level 2 ●

5.3 Construct Control Chart 5.3 (xii) Understanding the rationale for setting control limits

At levels 3 & 4 the rationale for setting control limits is explained in more depth

Level 3 ●

Level 4 ●

5.4 Construct Pareto Charts 5.4 (i) Construct a Pareto chart No gradient Level 3 ●

Level 4 ●

5.5 Construct Run Charts 5.5 (i) Know how to apply the four rules No gradient Level 2 ●

Level 3 ●

Level 4 ●

5.5 Construct Run Charts 5.5 (ii) Median = Centre Line At levels 2 Median = Centre Line

Level 2 ●

5.5 Construct Run Charts 5.5 (iii) Median = Centre Line At levels 3 & 4 instances where the mean may be used are addressed

Level 3 ●

Level 4 ●

50 | P a g e

5.5 Construct Run Charts 5.5 (iv) Understand the rationale for run chart rules and have a basic knowledge of how run chart rules are determined


Level 3 ●

Level 4 ●

5.5 Construct Run Charts 5.5 (v) Construct a run chart At levels 2, 3 & 4 how to construct run charts using median as the centre line is addressed

Level 2 ●

Level 3 ●

Level 4 ●

5.5 Construct Run Charts 5.5 (vi) Construct a run chart. Understand the limitations of run charts in healthcare applications

At level 4 the issues with using run chart rules as described in the 2011 paper published in BMJ Quality and Safety (Perla et al) are discussed

Level 4 ●

5.6 Construct Frequency Plots 5.6 (i) How to construct a Bar Chart No gradient Level 3 ●

Level 4 ●

5.6 Construct Frequency Plots 5.6 (ii) How to construct a Histogram No gradient Level 3 ●

Level 4 ●

5.6 Construct Frequency Plots 5.6 (iii) How to construct a Pareto Chart No gradient Level 3 ●

Level 4 ●

5.7 Construct Scatter Plots 5.7 (i) How to construct a scatter plot No gradient Level 3 ●

Level 4 ●

5.8 Analyse Survey Responses 5.8 (i) Knowledge of quantitative data analysis. Knowledge of and access to quantitative software tools such as survey monkey and statistical software packages.


Level 4 ●

5.9 Analyse Qualitative Data 5.9 (i) Ability to analyse qualitative data. Knowledge of thematic analysis using comparison, identifying similarities and differences across data. Knowledge of and access to qualitative data analysis software tools such as NVivo.


Level 4 ●

51 | P a g e

Step 6: Interpret and Present Information


6.1 Apply the Five Rules to Control Charts

6.1 (i) How to apply the 5 rules At level 2 practical examples of data that breaches each of the 5 rules in C, U, P & I Charts and funnel plots are introduced (only one rule applies to funnel plots)

Level 2 ●


6.1 (ii) How to apply the rules At levels 3 & 4, T and G Charts are added

Level 3 ●

Level 4 ●


6.1 (iii) Recalculating the centre line and control limits

Levels 3 & 4 addresses the issue of recalculation having applied the rules and emphasizes the role of subject matter experts in making decisions about where and when to recalculate the centre line and control limits

Level 3 ●

Level 4 ●


6.1 (iv) Examples of incorrect interpretation In interpreting charts, there are a number of common mistakes that should be avoided


Level 3 ●

Level 4 ●

6.2 Review Data with Subject Matter Expert

6.2 (i) The role of subject matter experts. Understanding the importance of subject matter expertise in bringing context to the variation observed


Level 3 ●

Level 4 ●

6.2 Review Data with Subject Matter Expert

6.2 (ii) Importance of annotating charts after making changes and when you observe special cause variation


Level 3 ●

Level 4 ●

52 | P a g e

6.3 Combine a Number of Measures to Give Overall Picture of the Aspect of Quality of Care with Reference to the Aim

6.3 (i) Balancing measures. Understanding and Appreciation of a System When optimising one process, it is important to use measurement to see what other parts of the system are affected and how they are affected.


Level 4 ●


6.3 (ii) Triangulation verses validation No gradient Level 3 ●

Level 4 ●

3 Combine a Number of Measures to Give Overall Picture of the Aspect of Quality of Care with Reference to the Aim

6.3 (iii) Awareness of HSE QI Framework for Improving Quality, 2016


Level 2 ●

Level 3 ●

Level 4 ●


6.3 (iv) Awareness of National Standards for Safer Better Health Care


Level 2 ●

Level 3 ●

Level 4 ●

6.4 Ensure Patient Voice is Recognised and Maintained in the Family of Measures

6.4 (i) Capturing the patient voice. Through the use of Qualitative Methods


Level 4 ●

6.5 Action to address Special verses Common Causes

6.5 (i) Two distinct kinds of variation should lead to distinct kinds of action. Where normal variation is observed system wide improvement is necessary, where special cause variation is observed it is necessary to identify why and act on that specifically


Level 4 ●

6.6 Interpret Qualitative Findings 6.6 (i) Uses and limitations of qualitative data No gradient Level 3 ●

Level 4 ●

53 | P a g e

Step 7: Act On Findings


7.1 Review your Measurement Results with Respect to your Aim

7.1 (i) Check measurement has helped identify whether a change resulted in an improvement


Level 3 ●

Level 4 ●

7.2 Distinguishing Signal from Noise 7.2 (i) Overreacting or failing to react. Focuses on distinguishing between common cause and special cause variation


Level 3 ●

Level 4 ●

7.3 Use Measurement Results to Identify Further Areas for Improvement

7.3 (i) Learning from PDSA cycles No gradient Level 3 ●

Level 4 ●

7.3 Use Measurement Results to Identify Further Areas for Improvement

7.3 (ii) Understanding the opportunity for improvement that special cause variation represents


Level 4 ●

7.4 Review and Share Measurement Results with Stakeholders

7.4 (i) Appropriate measurement reporting for different audiences. Addresses the importance of the right information going to the right people


Level 4 ●

7.4 Review and Share Measurement Results with Stakeholders

7.4 (ii) The importance of sharing learning and transparent reporting


Level 2 ●

Level 3 ●

Level 4 ●

7.5 Plan for Continued Measurement 7.5 (i) Sustaining measurement for improvement. The importance of ensuring that, where appropriate, measurement is continued after the project is completed.


Level 4 ●

Documents

Measurement for Improvement Curriculum - hse.ie · The Measurement for Improvement Curriculum ... MFI is the analysis and presentation of quantitative and qualitative data to