© Nuffield Trust28 July 2017

Challenges of Evaluating the Impacts

of Ambulatory Emergency Care

Paul Smith

Martin Bardsley

Nuffield Trust

© Nuffield Trust

• Promote independent analysis

and informed debate on

healthcare policy across the UK

• Charitable organization founded

in 1940

• Formerly a grant-giving

organization

• Since 2008 we have been

conducting in-house research

and policy analysis

• Significant interest in uses of

data linkage and predictive risk

techniques

The Nuffield Trust

William Morris

1st Viscount Nuffield

(1877 -1963)

© Nuffield Trust

Concern about trends in emergency admissions

Source: A&E Annual activity statistics, NHS and independent sector organisations in England

© Nuffield Trust

Trends differ by condition…

© Nuffield Trust

Nuffield work includes studies of….

Telehealth and Telecare – Whole System Demonstrator in 3 areas

National Integrated Care Pilots

Partnerships for Older People*

Birmingham Own Health

Virtual Wards in 4 sites*

Marie Curie Nursing Service*

NW London Integrated Care Pilot

British Red Cross ..Care in the Home

Not clear what works see Purdy et al (2012) Interventions to Reduce Unplanned Hospital

Admission: A series of systematic reviews. Bristol University Final Report)

© Nuffield Trust

We would like to find things that reduce emergency

admission

© Nuffield Trust

But we tend to find this…….

We

© Nuffield Trust

Cautions in evaluation……

1. Timing….Recognise that planning and implementing large scale

service changes take time

2. Fidelity….Define the service intervention clearly – and be clear when

the model is changed

3. Sample sizes…If you want to demonstrate statistically significant

change, size and time matter

4. Range of outcomes….Hospital use and costs are not the only impact

measures

5. Evaluation methods….Carefully consider the best models for

evaluation – prospective/retrospective; formative/summative;

quant./qualitative

6. Monitoring change in process

© Nuffield Trust

Specific potential problems with evaluating AEC

• No standard AEC dataset therefore data collection inconsistent

• AEC is changing over time (number of pathways, referral routes etc.)

• Different models and interpretations of AEC and large variation as to the

range of conditions managed (DVT most common)

• Sites vary in the scale of implementation (e.g. hours service available, size

of designated ambulatory care areas)

• Majority of sites around country provided some form of ambulatory care in

2009/10 (McCallum et al, 2010) so control groups problematic

• Identifying conditions consistently over time in routine data also complicated

by coding changes (mainly HRGs)

• The year service introduced and the extent to which AEC developed is

highly variable (at certain trusts services began before 2002)

• The field of emergency and urgent care is changing rapidly so may be

difficult to isolate an AEC effect

© Nuffield Trust

Evaluation goals and analysis decisions

Two main research questions

1. Do we see an overall AEC effect?

2. Do we see network-specific effects?

• HES inpatient data from 2009/10 to 2014/15 used as majority of sites should be

using admitted patient data

• Focus mainly on 18 of the clinical scenarios found in the Directory of Ambulatory

Care that are either included in the PbR Best Practice Tariffs or were reported as

the conditions most commonly managed by sites

• Reduction in bed days for the 18 clinical scenario chosen as our main measure

as this would be expected to decrease if AEC effective irrespective of dataset

used

• Limited to patients aged 18 and over (children excluded from BPTs) and

admissions occurring Monday to Friday

• Comparator groups selected were based on membership of AEC network

© Nuffield Trust

Clinical scenarios selected

© Nuffield Trust

Analysis cohorts

We used four main groups for comparison based on when the trusts joined the

AEC network :

• Cohort 1 - joined the network in September 2011, n = 10

• Cohort 2 – joined the network in September 2012, n = 12

• Cohort 3 – joined the network in April 2013, n = 10

• Remaining providers were used as a “control” (although not strictly true as

many will have AEC services), n = 110 (and 29 of these subsequently joined

the network). Specialist providers excluded from this cohort

• In total we examined 20,516,751 emergency admissions to English hospitals

between April 2009 and March 2014, encompassing 118,532,710 emergency

bed days

© Nuffield Trust

Results – Change in bed days by analysis cohort

© Nuffield Trust

Change in bed days for selected clinical scenarios by cohort

© Nuffield Trust

Change in emergency admissions for selected clinical

scenarios by cohort

© Nuffield Trust

Change in proportion of same-day discharges for selected

conditions by cohort

© Nuffield Trust

Reduction in emergency bed days over 6 years by condition

© Nuffield Trust

Variation in 0-day stay rates by condition (using latest

2014/15 data)

© Nuffield Trust

Summary

• Most sites do appear to be recording AEC activity using the admitted patient

dataset though certainly not the case for all

• Bed days appear to be the most clear-cut way of examining the impact of AEC

• There is evidence that AEC has helped reduce acute bed days for certain

conditions (with caveats around controls)

• There is also evidence of AEC network membership having an effect on bed day

reduction for specific conditions (again more thorough control groups may need

to be established though)

• There is variation between conditions in both the extent to which bed days have

been reduced, and the proportion of same-day discharges

© Nuffield Trust

Improvements and potential future work

• More rigorous definition of control groups – either drawing on more detailed

local knowledge to derive trust comparator groups or draw controls from a

different time period (say pre-AEC)

• More sophisticated modelling e.g. propensity matched controls and tracking

specific cohorts of patients who used the AEC service

• Fuller/multi-method evaluation capturing patient and staff views/experience,

locally defined metrics, and cost-benefit analysis

• More fully explore the potential for AEC – large variation between conditions

and how much of this based on casemix and how much due to the maturity of

the specific pathway?

© Nuffield Trust28 July 2017

www.nuffieldtrust.org.uk

Sign-up for our newsletterwww.nuffieldtrust.org.uk/newsletter

Follow us on Twitter:Twitter.com/NuffieldTrust

© Nuffield Trust

paul.smith@nuffieldtrust.org.uk