1
RBF4MNH Impact Evaluation – Preliminary Findings
Presenters: Manuela De Allegri & Stephan Brenner
Study team: Adamson Muula, Don Mathanga, Jacob Mazalale, Christabel Kambala, Julia Lohmann, Danielle Wilhelm,
Jobiba Chinkhumba, Bjarne Robberstad, Till Baernighausen, Tom Bossert
2
Background
• Evidence available on impact of Performance-Based Financing (PBF) interventions on health service utilization
• Lack of comparable evidence on impact of PBF on quality of care
3
4
The RBF4MNH intervention
• Funded by KfW and Norway• Implemented by Malawi Ministry of Health with
technical support by Options• Feasibility study completed by 2011• Intervention launch 2012• Active in 4 rural districts• 17-28 Emergency Obstetric Care (EmOC) included
5
The RBF4MNH intervention
• Results-based financing intervention which includes supply-side and demand-side incentives
• Preceded (and accompanied) by infrastructural upgrade
• Supply-side: quantity and quality indicators targeting labour and delivery services
• Demand-side: conditional cash transfers to women upon delivering in health facility
6
Research question
What is the impact of a combination of supply-side and demand-side PBF interventions on
quality of maternal care services?
Scope: ANC, delivery, early PNC, and early neonatal careFocus: continuity of care around EmONC
Outcome: utilisation and quality indicators
7
vPROCESS OUTCOME
Provision of care
Experience of care
INPUT
MIXED METHODS APPROACH within framework ofCONTROLLED BEFORE & AFTER STUDY
Conceptual approach
8
Study design
• Controlled before and after study ...• ... which evolved into a stepped wedge design• Data from 33 EmOC facilities and their catchment
areas• 18(23) intervention and 15(10) control
9
INPUT PROCESS OUTCOME
Infrastructural assessment
Interviews with healthcare workers
Assessment of clinical records
Provider-patient direct observations
Exit interviews
Clinical records
Household survey
Interviews with healthcare workers
Experience of care
FGD & in-depth interviews with all stakeholders
Provision of care
10
Mixed-methods in practice
11
Household survey: methods
• Strategy: repeated cross-sectional: 2013 (baseline – BL), 2014 (midline – ML), 2015 (endline – EL)
• Objective: assess impact on health service utilization• Sample: 2 Enumeration Areas (EA) from each facility
catchment area & 25 women with delivery in past 24 months within each EA; 4 EAs from second-level facilities
• Analysis: DID regression model accounting for clustering at facility level and for covariates (age, literacy, wealth, gravidity, SES, switch)
12
Household survey: results (1)
Sample distribution
13
Household survey: results (2)
Sample distribution
14
Household survey: results (3)Means & DID estimates
BL mean control
BL mean interventi
onEL mean control
EL mean interventi
onDID BL-EL DID BL-EL
est. Sig.
ANC ANC utilization 96,2% 96,9% 98,7% 98,9% -0,005 -0,002 0,892
ANC in first trimester
15,1% 15,1% 22,2% 21,8% -0,005 0,013 0,770
ANC 4+ visits 44,1% 44,0% 52,4% 50,6% -0,017 0,046 0,419
Delivery Delivery at facility 91,4% 91,1% 96,7% 94,9% -0,015 -0,006 0,819
PNC PNC utilization 84,9% 81,9% 82,6% 75,7% -0,040 -0,016 0,811
Timley PNC within 7 days
45,8% 42,0% 38,0% 35,6% 0,013 0,031 0,715
Timely PNC with 6 weeks
83,3% 79,3% 83,2% 76,1% -0,031 -0,004 0,948
15
Household survey: results (4)
Control women using delivery services in intervention facilities
BL EL
% %
Non-movers 84.1 77.6
Movers 15.9 22.4
Total 100.0 100.0
Bonferroni multiple comparison test p-value = 0.019
16
Health facility assessment: methods
• Strategy: repeated cross-sectional: • BL – 2013, ML - 2014, EL - 2015
• Objective: assess impact on service readiness for facility-based delivery
• Sample: full sample of all 28 health clinics in intervention and control arms
• Analysis: DID regression model, accounting for covariates (switch)
17
Definition service readiness• Based on WHO SARA domains for facility-based delivery:
• staff & training (2 indicators)• equipment (10 indicators)• medicines & commodities (6 indicators)
• Percentage of meeting basic service readiness requirements– Domain scores: mean availability of items per domain for
each facility in percent– Readiness index: mean of domain scores for each facility in
percent• DID: comparison of average scores and indices across
facilities in each study arm
Health facility assessment: results (1)
BASELINE MIDLINE ENDLINEN % N % N %
Total sample: 26 100 26 100 28 100
Distribution by district:Balaka 7 26.9 7 26.9 7 25.0Dedza 5 19.3 5 19.3 6 21.4
Mchinji 7 26.9 7 26.9 7 25.0Ntcheu 7 26.9 7 26.9 8 28.6
Distribution by study arm:Control 13 50.0 12 46.2 10 35.7
Intervention 13 50.0 14 53.8 18 64.3
Sample distribution
19
Health facility assessment: results (2)
Average percentage service readiness & DID estimates
BL mean control
BL mean interventi
onEL mean control
EL mean interventi
onDID BL-EL DID BL-EL
est. Sig.
Staff & Training 96.2 % 69.2 % 85.0 % 86.1 % 28.0 % 27.8 % 0.07
Equipment 72.3 % 76.9 % 75.0 % 88.3 % 8.7 % 4.6 % 0.63
Medicines & commodities 79.5 % 88.5 % 86.7 % 87.0 % -8.6 % -14.7 % 0.09
Service Readiness Index 82.7 % 78.2 % 82.2 % 87.2 % 9.4 % 5.9 % 0.44
20
Clinical performance: methods
• Strategy: repeated cross-sectional: • BL – 2013, ML - 2014, EL - 2015
• Objective: assess impact on clinical service provision during facility-based delivery
• Sample: convenience sample of all non-complicated labor cases at health clinics during 5-day observation period
• Analysis: DID regression model accounting for clustering at birth assistant level and for covariates (time period between initial case presentation and entry of stage 2, switch)
21
Definition clinical performance
• Observed performance measured against clinical standards (IMPAC, national QA guidelines): • Complete assessment eclampsia risk (5)• Correct partograph use (7)• Correct performance infection prevention (7)• Correct performance AMTSL (5)
• Percentage of meeting routine care standards– Performance scores: mean of observed routine care
processes per domain for each observed labor case• DID: comparison of average performance scores
across birth assistants in each study arm
Clinical performance: results (1)
Sample distribution BASELINE MIDLINE ENDLINE
N % N % N %
Total sample: 45 100 102 100 87 100
Distribution by district:Balaka 11 24.5 26 25.5 22 25.3Dedza 179 20.0 24 23.5 14 16.1
Mchinji 15 33.3 33 32.4 28 32.2Ntcheu 10 22.2 19 18.6 23 26.4
Distribution by study arm:Control 16 35.6 51 50.0 38 43.7
Intervention 29 64.4 51 50.0 49 56.3
23
Clinical performance: results (2)
Average percentage clinical performance & DID estimates
BL mean control
BL mean interventi
onEL mean control
EL mean interventi
onDID BL-EL DID BL-EL
est. Sig.
Eclampsia risk assessment 31.3 % 19.3 % 31.1 % 40.9 % 21.7 % 35.3 % 0.01
Correct partograph use 40.0 % 24.4 % 44.5 % 57.6 % 28.7 % 21.9 0.13
Infection prevention 50.9 % 51.2 % 54.0 % 72.3 % 18.0 % 27.8 % 0.05
AMTSL 66.3 % 77.9 % 91.1 % 94.8 % -8.0 % 14.4 % 0.18
24
Perception of care: methods
• Strategy: repeated cross-sectional: • BL – 2013, ML - 2014, EL - 2015
• Objective: assess impact on client’s experience of care delivered during labor
• Sample: convenience sample of women exiting health clinics after facility-based delivery
• Analysis: DID regression model accounting for clustering at facility level and for covariates (age, literacy, wealth, gravidity, SES, switch)
25
Definition satisfaction with care
• Satisfaction with aspects of care received: • Interpersonal interaction with birth assistant (13)• Technical performance of birth assistant (7)• Structural/organizational set-up of labour service (11)
• Average satisfaction for each dimension– satisfaction scores: lowest-highest satisfaction on 10-
point scale for each dimension• DID: comparison of average 10-point satisfaction
scores across clients in each study arm
Perception of care: results (1)
Sample distribution BASELINE MIDLINE ENDLINE
N % N % N %
Total sample: 203 100 333 100 230 100
Distribution by district:Balaka 36 17.7 52 15.6 45 19.6Dedza 64 31.5 100 30.0 52 22.6
Mchinji 59 29.1 103 30.9 76 33.0Ntcheu 44 21.7 78 23.4 57 24.8
Distribution by study arm:Control 67 33.0 109 32.7 40 17.4
Intervention 136 67.0 224 67.3 190 82.6
Distribution by level of care:BEmOC 121 59.6 209 62.8 147 63.9CEmOC 82 40.4 124 37.2 83 36.1
27
Perception of care: results (2)
Average satisfaction with care received & DID estimates
BL mean control
BL mean interventi
onEL mean control
EL mean interventi
onDID BL-EL DID BL-EL
est. Sig.
Satisfaction with interpersonal
aspects9.3 9.2 9.2 8.9 -0.2 -0.1 0.86
Satisfaction with clinical/technical
aspects9.4 9.3 9.4 9.1 -0.2 -0.1 0.69
Satisfaction with organizational/
structural aspects9.2 9.3 9.1 8.8 -0.4 -0.2 0.57
28
Conclusions
• No measurable effect on utilization of delivery services, but important effect in redirecting demand/women’s choice
• No effect on ANC and PNC services – need for incentives that address the continuum of care?
• Weak effects on service readiness in spite of additional input upgrades
• Strong effects on incentivized clinical tasks – no measurable effect on non-incentivized tasks
• No measureable effect on client satisfaction – high satisfaction independent of intervention, satisfaction measures not sufficiently specific?
29
Thank You