11

Best Practices in Selecting, Using and Assessing Indicators for Research, Evaluation

and Performance

© Fraser Health Authority, 2009

The Fraser Health Authority (“FH”) authorizes the use, reproduction and/or modification of this publication for purposes other than commercial redistribution. In consideration for this authorization, the user agrees that any unmodified reproduction of this publication shall retain all copyright and proprietary notices. If the user modifies the content of this publication, all FH copyright notices shall be removed, however FH shall be acknowledged as the author of the source publication.

Reproduction or storage of this publication in any form by any means for the purpose of commercial redistribution is strictly prohibited.

This publication is intended to provide general information only, and should not be relied on as providing specific healthcare, legal or other professional advice. The Fraser Health Authority, and every person involved in the creation of this publication, disclaims any warranty, express or implied, as to its accuracy, completeness or currency, and disclaims all liability in respect of any actions, including the results of any actions, taken or not taken in reliance on the information contained herein.

Outline

Objectives Background Anatomy Good Indicators Variation

Objectives

Understand and appreciate the purpose of indicators

Identify criteria for designing, appraising or choosing indicators

Apply indicators for research, evaluation or quality improvement.

Background

What is an indicator? Succinct measures that describe as much as possible

about a system Why use indicators?

Understanding – how a system works and how it might be improved (research)

Performance monitoring – to assess if the system is performing to at an expected level, compare and improve a system (improvement)

Accountability – to inform assessment of effectiveness, efficiency and of responsibility (evaluation)

Pop Quiz

Considerations Indicators indicate:

Cannot capture and reflect the complexity of a system program or service

Must be considered within context Encourage explicitness:

Help to clarify our understanding of a system Facilitate communication of expectations

Numerically based: Rates, ratios etc.

Not specifically designed to detect fault: Can be used for interpreting both strengths and

weaknesses

Improvement

Indicators primarily used to measure systems and outcomes in health care. Goal - improvement Understand how things work leads to

understanding of how things can be done better

Measurement alone does not lead to improvement.

Anatomy

Basic construction How to deconstruct and assess

Metadata (indicator) Title, rationale, and how indicator is

defined/constructed Data

Information that is entered into the indicator

Metadata

Metadata will help asses if an indicator is: Important and relevant Able to be populated with reliable data Likely to have desired effect when

communicated well

Metadata

What is being measured? Why is it being measured? How is this indicator defined? Who/what does it measure? When does it measure? Absolute or proportion? What is the source of data? What is the accuracy/completeness of the data? Any considerations, warnings, caveats? Are specific tests needed to test the meaning of the

data?

Statistics Canada – Catalogue no. 82-221-X

Statistics Canada – Health Indicators

Definition: Wait time for hip fracture surgery (same/next day) Proportion with surgery same or next day: risk-adjusted

proportion of hip fracture patients aged 65 and older who underwent hip fracture surgery on the day of admission or the next day.

Wait time for surgery following hip fracture provides a measure of the access to care. While some hip fracture patients need medical treatment to stabilize their condition before surgery, research suggests patients typically benefit from timely surgery in terms of reduced morbidity, mortality, pain, length of stay in hospital, as well as improved rehabilitation.

Rates for Quebec are not available due to differences in data collection.

Source(s): Canadian Institute for Health Information, Discharge Abstract

Database.

Group Activity

Deconstruct the Wait Time for Hip Fracture Surgery indicator using the Metadata Assessment Questions in the handout.

Identify information that you would need to fully understand the composition of this indicator.

Report back.

Data

Is the indicator populated with the best available data? Reliable Valid Bias Completeness Error Convenience

Acronyms

Good Indicators

SMART

S SpecificM MeasureableA AchievableR RelevantT Time-bound

DUMB

D DoableU UseableM MeasurableB Believable

Criteria for Good Indicators

1. Importance and relevance

2. Validity

3. Feasibility

4. Meaning

5. Implications

Adapted from: The Good Indicators Guide: Understanding how to use and choose indicators. NHS Institute for Innovation and Improvement, 2008http://www.apho.org.uk/resource/item.aspx?RID=44584

1. Importance and Relevance

Does the indicator measure what is relevant in the system?

Indicators must relate to objectives of the system Key parts of process and/or outcome should have

associated indicators Is there balance?

All important system components are covered No over or under-represented components

Will the indicators promote consensus? Well formed indicators can be helpful in developing

consensus on the objectives of the system

2. Validity Does the indicator actually measure what it is

intended to measure? Accuracy and degree to which the indicator actually

measures the construct May require validation if existing valid indicators are

not available or appropriate• Translation validity

Face validity Content validity

• Criterion-related validity Predictive validity Concurrent validity Convergent validity Discriminant validity

http://www.socialresearchmethods.net/kb/measval.php

3. Feasibility

Are reliable data available to populate the indicator?

Are data available at appropriate times? Are comparator data similarly available? Can the cost/resource allocation be

justified?

4. Meaning

Will the indicator be sensitive to detect and demonstrate meaningful changes in the system? (it should not identify random variations)

Can high and low indicator values provide appropriate signals to take action?

Can the indicator be deconstructed to provide insight about specific results or patterns?

Will others accept the indicator as having meaning?

5. Implications

What will you do if an indicator suggests further action?

Will indicator results induce perverse incentives and unintended uses?

Is there sufficient lag time in measurement so as to allow interventions to have an effect?

MacNamara Fallacy

The MacNamara Fallacy is named after the Robert McNamara, the US Secretary of Defence in the 1960s who was obsessed with quantifying the Vietnam War in a way that tended to ignore what was truly going on.

MacNamara argued that the ratio of Viet Cong losses to US losses was an important measure of effectiveness

MacNamara Fallacy

The first step is to measure whatever can be easily measured. This is OK as far as it goes.

The second step is to disregard that which can’t be easily measured or to give it an arbitrary quantitative value. This is artificial and misleading.

The third step is to presume that what can’t be measured easily really isn’t important. This is blindness.

The fourth step is to say that what can’t be easily measured really doesn’t exist. This is suicide.

Charles Handy, ‘The Empty Raincoat’, page 219.

Qualitative Indicators

Qualitative indicators such as perceptions are important.

Qualitative indicators may enable or hamper improvement/change.

It is important to balance quantitative with qualitative indicators to provide context

SPICED IndicatorsSubjective: Informants have a special position or experience that gives

them unique insights which may yield a very high return on the investigators time.

Participatory: Indicators should be developed together with those best placed to assess them.

Interpreted and Communicable: Locally defined indicators may not mean much to other stakeholders, so they often need to be explained.

Cross-Checked and Compared: The validity of assessment needs to be cross-checked, by comparing different indicators and progress, and by using different informants, methods, and researchers.

Empowering: The process of setting and assessing indicators should be empowering in itself and allow groups and individuals to reflect critically on their changing situation.

Diverse and Disaggregated: There should be a deliberate effort to seek out different indicators from a range of groups. This information needs to be recorded in such a way that differences can be assessed over time.

Roche (2002)

Program Theory

A statement of goals accompanied by the

underlying assumptions that guide a

system/program/service delivery strategy

and are believed to be critical to producing

the desired outcomes.

Program Theory Considerations

Who are you serving? What are you striving to accomplish for the

population that you are serving? What strategies do you believe will help

you successfully accomplish your goals?

Logic Model

A program logic model links outcomes with program activities … and the theoretical principles of the program” (Kellogg, 2001)

Thus, logic models set up both formative and summative evaluation questions

Evaluative answers are “useful” when they reduce the risks of making the wrong decision

Types of Evaluation

Formative “Improve” Periodic and timely Focus on program

activities and outputs Leads to early

recommendations for program improvement

Summative “Prove” Were resources

committed worthwhile Focus on outcomes

and impact Measures value of

program based on impact

* Kellogg logic model development guide

ResourcesDental Clinic Coordinator

Community Health Director

Staff dentist

Staff pediatrician

Medical providers

Money for supplies

ActivitiesTraining•Develop curriculum•Two one-hour didactic trainings to medical providers in oral health assessment•One-on-one training to medical providers on oral health

Outreach•Order dental supplies for packets•Make up packets•Distribute to parents at end of each visit

Outputs

Training# of two-hour trainings held# of one-on-one trainings held# of medical providers trained

Outreach# of parents/children receiving packets

Outcomes

Medical providers demonstrate accurate oral health assessment, education and prevention activities

More children receive high-quality oral health assessment, education and prevention activities during well-child visits

Parents/children are more knowledgeable about oral health and caring for children’s teeth

Reduced incidence of caries in children at the community health center

Program Goal: To improve the oral health of low-income children who receive primary care in a community health center

Example Logic Model

Example – Logic Model Linked with Indicators

Framework

Indicators Linked with Program Framework

Group Activity

Using the DoctorDad logic model: Develop one indicator for an anticipated output Develop one indicator for an anticipated outcome

Create a definition and identify sources of data Consider your indicator with respect to Indicator

Quality and Planning checklist. Report back.

Variation

indicators indicate – variation informs action

Objectives: Understanding variation. Know when to investigate. How to respond to variation.

Statistical Process Control

Common Cause Variation Everyday and inevitable variation which tends

to be small, with observed values close to the average.

Special Cause Variation Variation outside the historical experience

base. Signal of some important change in the

system.

Statistical Process Control

Statistical Process Control (SPC) is an effective method of monitoring a process through the use of control charts.

Control charts enable the use of objective criteria for distinguishing background variation from events of significance based on statistical techniques.

Statistical Process Control

Allows you to determine: That the system is working with an acceptable level of

performance and there are no outliers• No action needed

That the system is working with an acceptable level of performance and there are outliers

• Address outliers That the system’s average level of performance is not

acceptable• Address entire system

SPC Run Chart

Time ordered presentation of observations

A centre line (average or median) of all the observations plotted.

SPC Control Chart

three basic components:

a centre line, usually the average of all the samples plotted.

upper and lower statistical control limits that define the constraints of common cause variations.

performance data plotted over time.

SPC Run Chart Example

Steps to create a Run Chart Ideally, there should be a minimum of 15 data points. Draw a horizontal line (the x-axis), and label it with the

unit of time. Draw a vertical line (the y-axis), and scale it to cover the

current data, plus sufficient room to accommodate future data points. Label it with the outcome.

Plot the data on the graph in time order and join adjacent points with a solid line.

Calculate the mean or median of the data (the centre line) and draw this on the graph.

Run Chart Run chart of the number of red beads drawn across 25

draws

Run Chart – What to look for:

Useful Observations – number of observations that do not fall directly on centre line.

Run – sequence of one or more consecutive useful observations on the same side of the centre line.

Trend – sequence of successive increases or decreases in useful observations

Run Chart RulesIdentifying Special Cause

VariationNumber of Runs If there are too few or too many runs in the process.

Run Chart RulesIdentifying Special Cause

VariationNumber of Runs If there are too few or too many runs in the process.

Run Chart RulesIdentifying Special Cause

VariationShift If the number of successive useful observations falling

on either side of the centre line is greater than 7.

Run Chart RulesIdentifying Special Cause

VariationTrend If the number of successive useful observations increasing or

decreasing is greater than 7.

Run Chart RulesIdentifying Special Cause

VariationZig-Zag If the number of useful observations increasing or decreasing

alternately (zig-zag pattern) is greater than 14.

Control Chart RulesIdentifying Special Cause

VariationControl Limits – 2SD above or below centre line If there is one or more observations beyond the control limits

Warning Limits – 3SD above or below centre line If there are two successive observations beyond the control limits

Control Limit2SD

Warning Limit3SD

Example – uniform system underperforming

Control Chart Resource

http://www.indicators.scot.nhs.uk/SPC/SPC.html

Addressing Special Cause Variation

When you detect a special cause: Control any damage or problems with an immediate, short-term fix. Once a quick fix is in place, search for the cause. Ask people in the process what was

different that time. What was out of the ordinary? It might not have been much – an unexpected emergency, a change in schedules, or new employees.

Once you have discovered the special cause, you can develop a longer-term remedy. Most special causes have a negative impact on the output of the process and need to be removed.

Occasionally, a special cause can have a positive impact depending on the nature of the process. If this is the case, finds ways to capture and integrate it into the system.

Avoid these mistakes: Be careful not to view short-term fixes as a permanent solution or the process will

never be improved. Changing the process to accommodate the special cause. This usually adds cost and

bureaucracy. Blaming individuals. Warn employees to do better. People can only do as well as the system allows.

Addressing Common Cause Variation

Common Cause Variation – ideal situation for quality improvement

If a process indicator is stable, or in statistical control, does not mean that its results are satisfactory.

An indicator may be very consistent, but not meeting an expected outcome

Variation can be systematically reduced, even in stable processes, enabling a gradual tightening of control limits, and an overall increase in quality.