Roundtable on Realism and Resources:Reflections on using realist methodology to explore

and explain costs and cost-effectiveness Rob Anderson

Joanna Charles

Bruno Marchal

Rebecca Hardwick*

28th October 2014, CARES Conference, LiverpoolThis systematic review was funded by the NIHR CLAHRC for South West Peninsula.

The views expressed in this presentation are those of the authors and not necessarily those of the National Health Service, NIHR or the Department of Health.

Plan of Round table

• Introduction 5 minutes• 2 case example presentations 20 mins each

– Jo Charles– Rob Anderson

• Reflections – Bruno Marchal ~20 minutes• Open discussion ~25 minutes

Carol Weiss, 1995,• “social programmes are based on implicit or explicit or

explicit theories about how the programme will work. The evaluation should surface those theories and lay them out in as fine detail as possible, identifying all the assumptions and sub-assumptions built into the program.

• The evaluators then construct methods for data collection and analysis to track the unfolding of the assumptions … and examine the extent to which [alternative] programme theories hold.”

Theory-driven methods – driven by questions of effectiveness?

Realist evaluation and review methods – still driven with explaining effectiveness?

The core question of realist enquiry:

What works,

for whom

in what circumstances

in what respects

and how?

Whither resources and costs in all this?

1. Pawson and Tilley’s conception of mechanisms as “reasoning and resources”

2. ‘Every programme is some version of the theory that: if we provide these resources [organised in this way, to these people] we should expect these outcomes.’

Pawson 2003. Nothing as practical as good theory. Evaluation

3. More specifically, any hypothesised:• Contexts• Mechanisms• Outcomes

May each have different resource impacts or resource requirements

And, we are not alone!

• “… the real contribution of a systematic review of economic evaluations may not be to produce a single authoritative result, but to help decision makers understand the structure of the resource allocation problem they are addressing …

Thus, the emphasis in such a review is likely to be less on producing a summarised estimate of the cost-effectiveness ratio, and more on demonstrating by how much this varies from setting to setting, and why it varies.”

Mike Drummond, 2002 p.151

Thus, the emphasis in such a review is likely to be less on producing a summarised estimate of the cost-effectiveness ratio, and more on demonstrating by how much this varies from setting to setting, and why it varies.”

Mike Drummond, 2002 p.151

Realism and resources:Example: realist review of the cost-effectiveness of shared care for chronic conditionsRob Anderson, Rebecca Hardwick, Mark Pearson, Richard Byng

28th October 2014, CARES Conference, LiverpoolThis systematic review was funded by the NIHR CLAHRC for South West Peninsula.

The views expressed in this presentation are those of the authors and not necessarily those of the National Health Service, NIHR or the Department of Health.

Plan of talk

• Background & definitions• Review objectives• Method• Findings• Conclusions & wider challenges

Background

• Definition:

Shared care is a way to manage the care of people

in relation to a particular health condition when the nature of their condition means that

optimal care requires the planned involvement of both primary care and specialist care professionals and services.

Typically …:Agree shared care protocol:

Roles and responsibilitiesFrequency and nature of follow-upRules for referral

Enhanced information exchangeCare coordination (systems/person)Education &/or training (e.g. for GPs, o patients

Background

• Definition:• Why shared care?

– Brisbane circa 2001

Background

• Definition:• Why shared care?

– Brisbane circa 2001– ‘classic’ complex intervention– ‘Economically motivated’ health service changes

• Realist review methods & cost-effectiveness

Review objectives (effectiveness and cost-effectiveness)

1. Examine the evolution of the concept of shared care

2. Develop a range of programme theories that describe how shared care works, who it works for, in what circumstances and why.

3. Identify and describe the most important mechanisms by which shared care is thought to produce better outcomes for patients, carers or health and other care services

4. Identify and describe the contextual factors which augment/activate or decrease/negate the impact of these mechanisms

5. Identify and describe the resource use and cost requirements or impacts of the different mechanisms and contextual factors

Investigate the effectiveness and cost-effectiveness of shared care using realist review methodology in order to explain better the circumstances in which shared care is likely to be effective and cost-effective.

Review methods - summary

• Conventional review of cost-effectiveness studies• Realist review methods:

– Phase 1: Identifying and developing programme theories

– Phase 2: Testing and refining those theories• Extracting info relevant to 3 sub-theories (or key elements)• Noting contrasting or competing explanations• Striving to identify relevant contexts.

What explains the Cost-Effectiveness of Shared Care?

Theory 1:

If care is shared in a systematic way between particular organisations or clinicians working across the primary/secondary care interface,

then there are efficiency gains within the health system, due to:• 1a. shifting care to less costly settings• 1b. better tailoring of resources to need• 1c. enhancing use of patient and carer resources

NB. Theory 2 & Theory 3 – theories to explain effectiveness

The (very) mixed cost-effectiveness of shared care6 ‘full’ economic evaluations

Effect Effect = Effect

Cost Q Pyne et alC Rost et alP Von Korff et al (Psych int. for

Major dep.)P Von Korff et al (CBT-based

int. for Major or Minor dep.)

P McGhee et alC Pyne et alP Von Korff et al (Psych int. for

Minor dep.)

Cost =

Q Davies et al

Cost Q Steuten et al = the only

model-based analysis)

The (very) mixed cost-effectiveness of shared care6 ‘full’ economic evaluations + 10 cost analyses

Effect Effect = Effect

Cost Q Pyne et alC Rost et alP Von Korff et al (Psych int. for

Major dep.)P Von Korff et al (CBT-based

int. for Major or Minor dep.)

P McGhee et alC Pyne et alP Von Korff et al (Psych int. for

Minor dep.)Q McCrone et al (Medium vs Low)

P Anderson & Haas*

Cost =

Q Davies et alQ McCrone et al (High vs Low)

C Smith et alC Swindle et alP&C Byng et al

Cost Q Steuten et al = the only

model-based analysis)P McCulloch et al

Q&Ѱ Buckingham et al/Drummond et alQ Hutubessy et alQ Ratcliffe et alQ Naji et al

Cost-effectiveness through …1a. Shifting care to less costly care settings?

• Imposed shift: Specialist vs primary care mix of routine appointments

Week 0 2 4 8 12 16 20 24 28 32 36 40 44 48 52UsualCare

Sp Sp Sp Sp Sp Sp Sp Sp Sp Sp Sp Sp Sp Sp SpShared

CareSp PC PC PC Sp PC PC Sp PC PC Sp PC PC Sp Sp

Cost-effectiveness through …1a. Shifting care to less costly care settings?

• Imposed shift: Specialist vs primary care mix of routine appointments• Or Shift due to effectiveness:

↑ in routine/planned appointments (mainly

primary care)

Better monitoring and evidence-based

treatment change

Fewer, less severe acute

episodes

↓ in unplanned/urgent appointments (i.e.

specialist/hospital = £££s)

Davies et al - Arthritis

Buckingham et al - Asthma

McGhee et al - Hypertension

Naji et al - Diabetes

0 1 2 3 4 5 6 7

Shared careUsual care

No. of Specialist appointments (per year)

Series1

0 1 2 3 4 5 6 7 8

No. of GP/Primary care appointments (per

year)

Cost-effectiveness through …1a. Shifting care to less costly care settings?

• Imposed shift: Specialist vs primary care mix of routine appointments• Or Shift due to effectiveness:

BUT …• Actual cost difference between a specialist and primary care appt.?• Is the starting point over-met need? Or under-met need?

– e.g. ‘Undermet’ – collaborative care for MH in USA– e.g. ‘Overmet’ – for Diabetes and Hypertension (UK 1990s)

↑ in routine/planned appointments (mainly

primary care)

Better monitoring and evidence-based

treatment change

Fewer, less severe acute

episodes

↓ in unplanned/urgent appointments (i.e.

specialist/hospital = £££s)

Cost-effectiveness through …1b. Better tailoring to need?

• At patient group level – where optimal care requires a planned mix of specialist and generalist/primary care input

• At patient level: ‘Temporal tailoring’?– e.g. for Depression – key issue of treatment adherance– e.g. for Hep C – detecting adverse side-effects early

• ‘Tailoring Up’ or ‘Tailoring Down’?– e.g. Disease management for asthma – to lowest level– e.g. Stepped care protocols – mental health (medications &/or CBT)

• BUT, was this revealed in SD/SE of costs? Not often

Cost-effectiveness through …1c. Increased self-care, by patients and carers?

Much less evidence:• Only 3 studies aimed to ↑ self-care/self efficacy (2 asthma, 1 Diabetes)• Only 1 showed ↑ self efficacy Steuten et al

• 2 others did not measure self-care or associated resources– McCulloch et al (in Diabetes) “emphasised the critical role of patient as self-

manager” but also stressed future effort needed on this.

Conclusions

Evidence to support our theories of cost-effectiveness?

1a. Efficiency through shifting to less costly settings -

Yes

1b. Efficiency through tailoring resources to need -

Some

1c. Efficiency through enhancing use of patient and carer –

Very little

Efficiency through other mechanisms?

some leads to follow – treatment adherence, avoiding duplication

→ Better explanations of the wide variations in cost-effectiveness?

Yes

Can these be attributed to having used a realist approach?

Harder to say (NB. followed a conventional review)

Wider challenges & Research recommendations• Not enough rich description of programmes in economic studies –

can their underlying theories be reliably identified?• Sufficient numbers of empirical economic studies (a) that used

similar outcomes, and (b) similar/same theoretical mechanisms?• Need to explore econ evidence in effectiveness studies• Working with complexity – how do we make contingent, provisional

review findings acceptable and useful for policy makers?

Hardwick, R., Pearson, M., Byng, R., Anderson, R. (2013). The effectiveness and cost-effectiveness of shared care: protocol for a realist review. Systematic Reviews, 2;12. http://www.systematicreviewsjournal.com/content/2/1/12

Anderson R. Systematic reviews of economic evaluations: utility or futility? Health Economics. 2010: 19(3); 350-364.

Thank you. R.Anderson@exeter.ac.uk

Some key questions for discussion

Should realist reviewers and evaluators set out to try and identify some mechanisms which are especially resource-sensitive or resource-impactful?

Or conduct separate or integrated reviews of effectiveness and cost-effectiveness?

Or just relax – if particular resources are important for Mechanisms, Contexts or as Outcomes then it will ‘all come out in the wash’?