Evaluation of integrated care initiatives: challenges, approaches and the evaluation of Integrated Care and Support Pioneers in England

Nicholas MaysProfessor of Health PolicyDirector, Policy Innovation Research UnitDepartment of Health Services Research & Policy

HARC Forum, ‘Evaluation of integrated care initiatives’, 16 September 2015

Funding acknowledgement and disclaimer

Part of this presentation summarises independent research commissioned and funded by the Department of Health Policy Research Programme (Policy Research Unit in Policy Innovation Research, 102/0001). The views expressed are those of the authors and not necessarily those of the Department of Health.

Outline

1. Challenges of evaluating integrated care programmes

2. Approaches to evaluation of integrated care programmes

3. NHS and social care context of better ‘integrated care’

4. Integrated Care and Support Pioneer Programme

5. Design of longer term evaluation of integrated care and support Pioneers in England

Challenges of evaluating integrated care

Some challenges of integrated care evaluation

• ‘Integrated care’ covers a wide range of different service changes– No single definition

– IC has to reflect the system requirements and context

– Difficult to cumulate knowledge to define what the next evaluation should cover

• Everyone aspires to more integrated care– Highly likely to be other developments happening in parallel making

comparisons difficult to identify and interpret

– ‘noise to signal ratio’ likely to be adverse

• IC programmes are typically multi-level, multi-faceted, multi-site– Potentially many different interventions & outcomes of interest

– Hard, if not impossible, to assess cost-effectiveness

Some challenges of integrated care evaluation

• Programmes rarely designed to facilitate evaluation– Researchers may have little or no input to design of programmes

– Even ‘pilots’ may not have clear objectives and comparisons

– Programme start date may not be clear cut

– Mismatch between evaluation ‘imperative’ and reality of programmes especially if programmes evolve flexibly at local level

• Impacts not likely to become visible in less than 3-5 years– Yet priorities will change over five years so evaluations need flexibility

Relevant questions for evaluating major system change: much more than ‘does it work?’

• Researchers tend to focus on effectiveness questions like:– What are the likely threats to validity (secular trends, biased outcome

ascertainment, contamination)?

• But ‘does it work?’ is often a meaningless question

• Likely much more relevant are questions from different stakeholders’ perspectives are:

– What works, for whom, in what circumstances, in what respects, how and why?

– How can the programme be adapted to help it work better?

– How should the context (wider system) be modified to help?

Approaches to evaluation of integrated care

How to approach evaluation of integrated care initiatives

• Likely to be strong policy impetus with implications for design (e.g. experiments)

• Effects will be affected by context and vice versa– Hence attraction of a realist approach to evaluation testing

the effects of interventions and their components in different contexts

• Likely to require different strands of data collection and analysis – More a synthesis than a single design– There may be elements looking at clearly defined

interventions with experimental designs but this is unlikely to suffice

An example of a high quality multi-method evaluation of large scale system change

Morris, S. et al. (2014). Impact of centralising acute stroke services in English metropolitan areas on mortality and length of hospital stay: difference-in-differences analysis. BMJ, 349, g4757

An example of a high quality multi-method evaluation of large scale system change

Fulop N, Morris S. Evaluating major system change in acute stroke services. HSRN Nottingham, 1-2 July 2015

Features of the stroke service centralisation evaluation

1. Seizes a ‘natural experiment’ over time

2. Wide range of research questions

3. Mixed qualitative (‘how and why’ of change) and quantitative (‘what impact, at what cost’?) methods– Interviews, observations, documentary analysis

– Controlled before and after (difference in difference)

4. Administrative and original data

5. Includes process of implementation (e.g. why different models were chosen, how each model was implemented), changes in services delivered and used, clinical outcomes (mortality & LOS) and cost-effectiveness

• Sophisticated study but arguably a far simpler initiative to evaluate than many integrated care programmes

Fulop N, Morris S. Evaluating major system change in acute stroke services. HSRN Symposium 2015. Nottingham, 1-2 July 2015

Value of combining quantitative and qualitative methods in stroke service centralisation evaluation

Fulop N, Morris S. Evaluating major system change in acute stroke services. HSRN Nottingham, 1-2 July 2015

The NHS and social care context of better ‘integrated care’

Health and social care (HSC) system(s)

NHS

• GP, community health, acute hospital & mental health services delivered through different organisations

• Mostly public , some private & Third Sector providers

• Universal, largely free at the point of use

• Different provider payment systems for different services

Social care

• Elected local government responsibility

• Home care, residential care

• Younger disabled & older people

• Focus on disability & frailty

• Mostly private providers

• Mix of self-payers & those receiving means-tested public subsidy

Government spending on English NHS as share of GDP is planned to fall so there is an imperative to raise productivity to a degree never seen in any system

Source: OBR Economic and Fiscal Outlook, March 2014, Table 1.3

Accompanied by an epidemic of pilots, trailblazers, demonstrators, pioneers, vanguards on related themes ……

Vanguards: Integrated Primary and Acute Care Systems –joining up GP, hospital, community and mental health service

The Integrated Care and Support Pioneer Programme

Pioneer programme definition of integrated care

My care is planned with people who work together to understand me and my carer(s), put me in control, co-ordinate and deliver services to achieve my best outcomes.” (National Voices 2013)

• A user experience-focused definition that does not prescribe howthis result is to be achieved at local level, hence scope for considerable variation in interpretation

The Pioneer programme

• DH called for the “most ambitious and visionary” local areas to become [health and social care] integration Pioneers to drive change “at scale and pace, from which the rest of the country can benefit” (DH, May 2013)

• 14 successful out of >100, November 2013

• 11 successful, January 2015

• Supported but no extra service funding

• Evaluation (1+5 years)

Specific problems that the Pioneers were to address

• Lack of coordination between NHS and social care, and between parts of NHS (hospital, CHS, general practice)

• Separate funding and payment systems

• Separate governance and accountability

• Experience of fragmentation, duplication, overlap, gaps in service at user/patient level

• (Threats to financial sustainability of system)

Characteristics of the Pioneers

• Ambitious whole system transformation

• Emphasis on health and social care horizontal integration

• Strong user/patient experience focus

• Existing organisations remain separate

• No governance/management template

• Variation on how vision is to be realized though many similarities in target populations and service changes/initiatives

What are they doing?

• Primary prevention and alternatives to statutory services, e.g. developing community assets and fostering self-care

• Getting professionals to work together, e.g. MDTs, often based around GPs

• Improving patient experience, e.g. single point of contact, care ‘navigators’

• Moving from reactive to proactive care, e.g. risk stratifying patients at risk of admission and providing a care plan

• Moving provision from acute sector to primary care and community services in an effort to reduce avoidable hospital admissions

Pioneers vary in scale and related complexity

‘Large, complex’ – NW London2.2 million population (targeting specific neighbourhoods)31 organisations – 8 clinical commissioning groups (CCGs), 7 Local Authorities, NHS England and community health, mental health & acute NHS providersOlder people with complex needs to be empowered to direct their care & support at home/outside hospitalGPs at the centre of organising & coordinating careSupportive systems to enable thisFocus on older people with complex needs

‘Small, simple’ – South Tyneside148,000 population4 organisations – S Tyneside Local Authority, S Tyneside CCG, S Tyneside NHS foundation trust, Tyne and Wear NHS FTFocus on strengthening self-care

Design of longer term evaluation of integrated care

and support Pioneers in England

Aims of the longer term evaluation

• Assess extent to which the 25 Pioneers, in the context of new funding arrangements, are successful in providing ‘person-centred coordinated care’, including improved outcomes and quality of care, in a cost-effective way in different contexts

• Builds on 18-month ‘early’ evaluation

Structure of longer-term evaluation

Three inter-related work packages (WPs):

• WP1: Pioneer level process evaluation and (limited) impact evaluation

– Analysis of routine data (indicators), panel survey, interviews

• WP2: Scheme/initiative level impact and cost-effectiveness

– 3-4 cost-effectiveness studies of initiatives across Pioneers including qualitative research to explain any differences

• WP3: Working with Pioneers, national policy makers and partners, patient/user organisations and experts to derive and spread learning

PIRU (2014) Integrated Care Pioneers: indicators for measuring integrated carehttp://www.piru.ac.uk/assets/files/IC%20and%20support%20Pioneers-Indicators.pdf

How the design responds to the evaluation challenges

• Summative, process and formative elements

• Multi-level with intention to link between levels

• Attempt to combine breadth with depth given scale of the Pioneers

• Comparison of cost-effectiveness of initiatives across local system contexts

• Mixed methods

• Focus on impacts allowed by length of evaluation

Challenges remaining including

• How to select ‘initiatives’ at different levels within and between Pioneers

• Many possible criteria including:

– ‘Level’ of intervention

– Characteristics of the initiative (i.e. patient/client groups; who leads; who provides; mechanism of change)

– Sufficient time to study outcomes

– Potential to improve (cost) effectiveness locally

– Potential to be scaled up and ‘rolled out’

– Potential to measure outcomes validly

– Implemented in more than one Pioneer

• Practical criteria may trump all others:

– Potential availability of outcome data and/or the ability to collect such data

– Possibility of an adequate sample size

– Willingness to cooperate with a detailed C-E evaluation

Further reading on Pioneers national evaluation

NHS England (2015) Integrated Care and Support Pioneer Programme Annual Report 2014

http://www.local.gov.uk/documents/10180/6927502/Integrated+Care+Pioneer+Programme+Annual+Report+2014/76d562c3-4f7d-4169-91bc-69f7a9be481c

PIRU (2014) Integrated Care Pioneers: indicators for measuring integrated care

http://www.piru.ac.uk/assets/files/IC%20and%20support%20Pioneers-Indicators.pdf

Erens B, et al. (2015) Early evaluation of the Integrated Care and Support Pioneers Programme: interim report

http://www.piru.ac.uk/assets/files/Early%20evaluation%20of%20IC%20Pioneers,%20interim%20report.pdf