Pbf malawi preliminary_analysis

1

RBF4MNH Impact Evaluation – Preliminary Findings

Presenters: Manuela De Allegri & Stephan Brenner

Study team: Adamson Muula, Don Mathanga, Jacob Mazalale, Christabel Kambala, Julia Lohmann, Danielle Wilhelm,

Jobiba Chinkhumba, Bjarne Robberstad, Till Baernighausen, Tom Bossert

2

Background

• Evidence available on impact of Performance-Based Financing (PBF) interventions on health service utilization

• Lack of comparable evidence on impact of PBF on quality of care

3

4

The RBF4MNH intervention

• Funded by KfW and Norway• Implemented by Malawi Ministry of Health with

technical support by Options• Feasibility study completed by 2011• Intervention launch 2012• Active in 4 rural districts• 17-28 Emergency Obstetric Care (EmOC) included

5

The RBF4MNH intervention

• Results-based financing intervention which includes supply-side and demand-side incentives

• Preceded (and accompanied) by infrastructural upgrade

• Supply-side: quantity and quality indicators targeting labour and delivery services

• Demand-side: conditional cash transfers to women upon delivering in health facility

6

Research question

What is the impact of a combination of supply-side and demand-side PBF interventions on

quality of maternal care services?

Scope: ANC, delivery, early PNC, and early neonatal careFocus: continuity of care around EmONC

Outcome: utilisation and quality indicators

7

vPROCESS OUTCOME

Provision of care

Experience of care

INPUT

MIXED METHODS APPROACH within framework ofCONTROLLED BEFORE & AFTER STUDY

Conceptual approach

8

Study design

• Controlled before and after study ...• ... which evolved into a stepped wedge design• Data from 33 EmOC facilities and their catchment

areas• 18(23) intervention and 15(10) control

9

INPUT PROCESS OUTCOME

Infrastructural assessment

Interviews with healthcare workers

Assessment of clinical records

Provider-patient direct observations

Exit interviews

Clinical records

Household survey

Interviews with healthcare workers

Experience of care

FGD & in-depth interviews with all stakeholders

Provision of care

10

Mixed-methods in practice

11

Household survey: methods

• Strategy: repeated cross-sectional: 2013 (baseline – BL), 2014 (midline – ML), 2015 (endline – EL)

• Objective: assess impact on health service utilization• Sample: 2 Enumeration Areas (EA) from each facility

catchment area & 25 women with delivery in past 24 months within each EA; 4 EAs from second-level facilities

• Analysis: DID regression model accounting for clustering at facility level and for covariates (age, literacy, wealth, gravidity, SES, switch)

12

Household survey: results (1)

Sample distribution

13


Sample distribution

14

Household survey: results (3)Means & DID estimates

BL mean control

BL mean interventi

onEL mean control

EL mean interventi

onDID BL-EL DID BL-EL

est. Sig.

ANC ANC utilization 96,2% 96,9% 98,7% 98,9% -0,005 -0,002 0,892

ANC in first trimester

15,1% 15,1% 22,2% 21,8% -0,005 0,013 0,770

ANC 4+ visits 44,1% 44,0% 52,4% 50,6% -0,017 0,046 0,419

Delivery Delivery at facility 91,4% 91,1% 96,7% 94,9% -0,015 -0,006 0,819

PNC PNC utilization 84,9% 81,9% 82,6% 75,7% -0,040 -0,016 0,811

Timley PNC within 7 days

45,8% 42,0% 38,0% 35,6% 0,013 0,031 0,715

Timely PNC with 6 weeks

83,3% 79,3% 83,2% 76,1% -0,031 -0,004 0,948

15


Control women using delivery services in intervention facilities

BL EL

% %

Non-movers 84.1 77.6

Movers 15.9 22.4

Total 100.0 100.0

Bonferroni multiple comparison test p-value = 0.019

16

Health facility assessment: methods

• Strategy: repeated cross-sectional: • BL – 2013, ML - 2014, EL - 2015

• Objective: assess impact on service readiness for facility-based delivery

• Sample: full sample of all 28 health clinics in intervention and control arms

• Analysis: DID regression model, accounting for covariates (switch)

17

Definition service readiness• Based on WHO SARA domains for facility-based delivery:

• staff & training (2 indicators)• equipment (10 indicators)• medicines & commodities (6 indicators)

• Percentage of meeting basic service readiness requirements– Domain scores: mean availability of items per domain for

each facility in percent– Readiness index: mean of domain scores for each facility in

percent• DID: comparison of average scores and indices across

facilities in each study arm

Health facility assessment: results (1)

BASELINE MIDLINE ENDLINEN % N % N %

Total sample: 26 100 26 100 28 100

Distribution by district:Balaka 7 26.9 7 26.9 7 25.0Dedza 5 19.3 5 19.3 6 21.4

Mchinji 7 26.9 7 26.9 7 25.0Ntcheu 7 26.9 7 26.9 8 28.6

Distribution by study arm:Control 13 50.0 12 46.2 10 35.7

Intervention 13 50.0 14 53.8 18 64.3

Sample distribution

19

Health facility assessment: results (2)

Average percentage service readiness & DID estimates

BL mean control

BL mean interventi

onEL mean control

EL mean interventi


est. Sig.

Staff & Training 96.2 % 69.2 % 85.0 % 86.1 % 28.0 % 27.8 % 0.07

Equipment 72.3 % 76.9 % 75.0 % 88.3 % 8.7 % 4.6 % 0.63

Medicines & commodities 79.5 % 88.5 % 86.7 % 87.0 % -8.6 % -14.7 % 0.09

Service Readiness Index 82.7 % 78.2 % 82.2 % 87.2 % 9.4 % 5.9 % 0.44

20

Clinical performance: methods


• Objective: assess impact on clinical service provision during facility-based delivery

• Sample: convenience sample of all non-complicated labor cases at health clinics during 5-day observation period

• Analysis: DID regression model accounting for clustering at birth assistant level and for covariates (time period between initial case presentation and entry of stage 2, switch)

21

Definition clinical performance

• Observed performance measured against clinical standards (IMPAC, national QA guidelines): • Complete assessment eclampsia risk (5)• Correct partograph use (7)• Correct performance infection prevention (7)• Correct performance AMTSL (5)

• Percentage of meeting routine care standards– Performance scores: mean of observed routine care

processes per domain for each observed labor case• DID: comparison of average performance scores

across birth assistants in each study arm

Clinical performance: results (1)

Sample distribution BASELINE MIDLINE ENDLINE

N % N % N %

Total sample: 45 100 102 100 87 100


Mchinji 15 33.3 33 32.4 28 32.2Ntcheu 10 22.2 19 18.6 23 26.4


Intervention 29 64.4 51 50.0 49 56.3

23

Clinical performance: results (2)

Average percentage clinical performance & DID estimates

BL mean control

BL mean interventi

onEL mean control

EL mean interventi


est. Sig.

Eclampsia risk assessment 31.3 % 19.3 % 31.1 % 40.9 % 21.7 % 35.3 % 0.01

Correct partograph use 40.0 % 24.4 % 44.5 % 57.6 % 28.7 % 21.9 0.13

Infection prevention 50.9 % 51.2 % 54.0 % 72.3 % 18.0 % 27.8 % 0.05

AMTSL 66.3 % 77.9 % 91.1 % 94.8 % -8.0 % 14.4 % 0.18

24

Perception of care: methods


• Objective: assess impact on client’s experience of care delivered during labor

• Sample: convenience sample of women exiting health clinics after facility-based delivery

• Analysis: DID regression model accounting for clustering at facility level and for covariates (age, literacy, wealth, gravidity, SES, switch)

25

Definition satisfaction with care

• Satisfaction with aspects of care received: • Interpersonal interaction with birth assistant (13)• Technical performance of birth assistant (7)• Structural/organizational set-up of labour service (11)

• Average satisfaction for each dimension– satisfaction scores: lowest-highest satisfaction on 10-

point scale for each dimension• DID: comparison of average 10-point satisfaction

scores across clients in each study arm

Perception of care: results (1)

Sample distribution BASELINE MIDLINE ENDLINE

N % N % N %

Total sample: 203 100 333 100 230 100


Mchinji 59 29.1 103 30.9 76 33.0Ntcheu 44 21.7 78 23.4 57 24.8


Intervention 136 67.0 224 67.3 190 82.6

Distribution by level of care:BEmOC 121 59.6 209 62.8 147 63.9CEmOC 82 40.4 124 37.2 83 36.1

27

Perception of care: results (2)

Average satisfaction with care received & DID estimates

BL mean control

BL mean interventi

onEL mean control

EL mean interventi


est. Sig.

Satisfaction with interpersonal

aspects9.3 9.2 9.2 8.9 -0.2 -0.1 0.86

Satisfaction with clinical/technical

aspects9.4 9.3 9.4 9.1 -0.2 -0.1 0.69

Satisfaction with organizational/

structural aspects9.2 9.3 9.1 8.8 -0.4 -0.2 0.57

28

Conclusions

• No measurable effect on utilization of delivery services, but important effect in redirecting demand/women’s choice

• No effect on ANC and PNC services – need for incentives that address the continuum of care?

• Weak effects on service readiness in spite of additional input upgrades

• Strong effects on incentivized clinical tasks – no measurable effect on non-incentivized tasks

• No measureable effect on client satisfaction – high satisfaction independent of intervention, satisfaction measures not sufficiently specific?

29

Thank You

Health & Medicine

Pbf malawi preliminary_analysis