Public Service Productivity Measurement:Macro or Micro?

Aileen Simkins, Department of HealthCo-Director of the Atkinson Review

What I will Talk About

• Some concepts

• Macro productivity measurement: the Atkinson Review (and Gershon)

• Micro productivity measurement: some NHS examples

• Challenges for the future

Productivity: What do we mean?

Getting more for the same, or the same for less

Technical efficiencyallocative efficiency

value for moneyeffectiveness

economycuts

Increase in output divided by

increase in volume of input

Improving services while money is tight

Measuring Productivity: Who By, What for?

Do you get what you pay for?

Accountability to the tax payer

Just try harder

Performance management

Can’t we get another one in? - Local drivers to expand/ improve

The XX industry is inefficient by international standards

Evidence of use of resources

Best of breed have 3 times more throughput than bottom quartile

Benchmarking, competitive pressures

Some key questions

• Can Government measures of public service productivity be valid?

• Is measuring productivity the same as measuring performance?

• Improving productivity matters – is measurement a side alley?

• Does measurement of efficiency create distortions?

The Atkinson Review

How to use National Accounts

to measure public service productivity

and

How to do it better

How National Accounts Measure Public Services

• Traditionally, output = input• From 1997, System for National Accounts

changed: measure outputs via activities• UK early implementer – health (cost weighted

activity index), schools (pupil days and quality adjustment), social services, social security admin

• ONS began publishing productivity articles

ONS Output Measure, 2003

Better Measures for the National Accounts

• Queries on basis of public service measures

• National Statistician set up Review Dec 2003, chaired Sir Tony Atkinson

• ‘Measurement of Government Output and Productivity for the National Accounts’

Atkinson Report Jan 2005

• 9 principles ‘output should be measured in a way that is

adjusted for quality, taking account of the attributable incremental contribution of the service to the outcome’

• Specific recommendations for improvement to measures used in NA

• Encouraged ONS productivity articles• UK Centre for Measurement of

Government Activity

ONS Health Productivity Oct 2004

80

85

90

95

100

105

110

1995 1996 1997 1998 1999 2000 2001 2002 2003

Output w ithout quality; inputs: drugs deflated by cost of all items; capitalservices; missing years estimated as average of last 3 years

Output w ithout quality; inputs: drugs deflated by Paasche Price Index;capital consumption; missing years estimated as previous year

DH Press Release Oct 04

John Reid (Secretary of State for Health) says

“ it is absurd to measure NHS output without taking account of quality”

Quality as part of NHS Output• How many domains of quality?

– Health gain– Patient experience

• What can we measure?• How can we link quality measures to the NHS

output index?• How should we weight different aspects of

quality?• How valid is a partial story?

Accounting for Quality Change

Average over last 5 years:• Value of health 1.5%• Value weight for statins 0.81%• York/NIESR adjustment 0.17%• Patient experience* 0.07%• Blood pressure control * 0.05%• Heart attack survival 0.01%Total ** 2.68% Quality adjusted output growth 6.29%

ONS Health Productivity 2006

80

85

90

95

100

105

110

1995 1996 1997 1998 1999 2000 2001 2002 2003 2004

Output w ith quality and value of health; inputs: drugs deflated by cost of allitems; capital consumption, direct labour method

Output w ith quality and value of health; inputs: drugs deflated by PaaschePrice Index; capital services, indirect labour method

Education and Other Areas• DfES action following Atkinson – new use of

pupil attainment tests and GCSE grades• DfES article, ONS education productivity

article• Controversial issues – grade drift, ‘real

earnings’ effect• Also productivity articles for adult social

care, social security administration

Value and Validity of Atkinson-based Productivity Measures• Designed to compare total outputs with total

inputs• Focus on attributable impact on outcomes

and quality change• Data incomplete; biased towards areas of

attention / improvement; analysis by Depts• Major developmental issues on techniques• ONS consultation document Sept 2006

Performance = Productivity?• A perfect Atkinson measure captures all

activities, all contributions to outcomes

• Is this the same as measuring organisation performance?

• If not – choices about key priorities, value judgements

• Should those values inform the aggregate productivity measure?

Another macro measure: Gershon

• Gershon includes savings from better procurement, Atkinson doesn’t

• Specific changes in use of inputs valued as £m

• Better use of productive time – evidence from outputs and outcomes?

Gershon and Atkinson

• Gershon efficiency improvements are managed, not just measured

• Strong delivery support for Gershon, changes evident

• Measured to deliver target? – NAO scrutiny• Will measured productivity gains match? –

HMT, OGC and UKCeMGA should be able to explain

Micro Measures: NHS

• NHS Institute for Innovation and Improvement, with DH

• Developing and issuing key metrics for health authorities, PCTs, Trusts, FTs

• Systematic focus on key areas for improvement – volume, costs, variation

• Clinical engagement, benchmarking

First Metrics: 22 Sept 2006

• Pdfs and web based tool• Finance, workforce, procurement • Clinical productivity:

– Potential bed days saved through reduced length of stay

– Day case rate– Reduce pre-operative bed days

Reduce wasted bed days

Percentage of patients admitted on the day of operation

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Acute hospital trusts

390,000 bed days£100m

Benchmarking

• Evidence based assessment of performance• Addressing variations and helping outliers• Providing a tool kit• Opportunity to test performance against

peers• Stimulus to learn how to improve practice• Recognition of success and challenge to do

better

XXX NHS Trust

Value and Power of Micro Productivity Indicators

• Practical engagement in what to change• Service based, clinical focus• Small steps on long journey• Risks that change happens, savings don’t• Check for unintended perverse incentives• Tipping points for cutting overheads

Impact of Micro Changes on Macro Measure

NHS III says:1 – Reduce Avoidable Emergency Admissions 2 – Reduce Unnecessary Outpatient Appointments,

Follows-up, and DNAs 3 – Avoid Unnecessary Procedures 4 – Improve Day Case Performance 5 – Reduce Wasted Bed Days6 – Accurate Clinical Coding7 – Reduce Variation in Length of Stay8 – Improve Staff Productivity9 – Actively Manage Staff Costs

If NHS follows NHS III advice..• Lower growth in some hospital activities• Shifts in unit costs• Shifts in NHS spending to primary care,

community, public health (harder to count)• Probably, slower growth in output• Measurable change in quality, outcomes?• Productivity measure?

Measures, not Targets

• Macro and micro productivity measures both need careful interpretation

• Micro are more useful for managers, front line, actionability

• Macro should help with ‘do you get what you pay for’ – but may mislead

• Use of any efficiency measures changes behaviours – intended, maybe also unintended

Lessons of History

‘For the first decade of its existence (1982-92) the HCHS efficiency index received little adverse comment either from the NHS or policy analysts. However, once it was converted from a retrospective analytical exercise into a management tool it attracted a barrage of criticism….attaching rewards to counts of particular activities led to reporting drift, reclassification and at worst statistical dishonesty’

Clive Smee ‘Speaking Truth to Power’ 2005

Dilemma: Measuring with Right Impact

• Productivity is Govt business: accountability • Many drivers in public service to improve

productivity• Accountability needs metrics• Metrics are almost always partial, may

mislead or have unintended incentives • Choice of metrics affects what is delivered• Moral: use mix of macro and micro

indicators, with intelligence & transparency