IMPACT EVALUATION OF PERFORMANCE BASED

CONTRACTING FOR

A collaboration between the Ministry of Health, CNLS, SPH, INSP-Mexico and World Bank

GENERAL HEALTH AND HIV/AIDS SERVICES IN

RWANDA

1

04/21/23

2

Presentation plan

1. Country Profile2. Impact evaluation design 3. Results from the general health

baseline and Follow up4. Results from HIV baseline and

Follow up studies 5. Discussion

2

SECTION 2 :Impact Evaluation

Design

7

04/21/23

8

Study Rationale

No examples of rigorously evaluated bonus payment schemes to public sector health care providers in developing countries

No distinction between the incentive effect and the effect of an increase in resources for the health facilities

Link between worker motivation programs and quality of care

8

TRADITIONAL FINANCING VS PBF Traditional Way of Health financing in Rwanda

was based on 3 sources of finance:

GoR / Founders: Investment cost (Infrastructures & Equipment)

GoR / Private / Population: Recurrent cost (salaries, drugs, materials, trainings,..)

PBF: funds received for remuneration can be derived for enhancing human capacity building, HF materials, or any new strategy aiming at increasing quality.

04/21/23

10

Hypotheses

For both general health services and HIV/AIDS services, we will test whether PBC:

Increases the quantity of contracted health services delivered

Improves the quality of contracted health services provided

Does not decrease the quantity or quality of non-contracted services provided,

Decreases average household out-of-pocket expenditures per service delivered

Improves the health status of the population

10

04/21/23

11

Evaluation Design Make use of expansion of PBC schemes over time The rollout takes place at the District level Treatment and control facilities were allocated as

follows: Identify districts without PBC in health centers

in 2005 Group the districts in “similar sets” based on

characteristics: rainfall population density livelihoods

Flip a coin to assign districts within each “similar” to treatment and control groups. 11

12

04/21/23

13

Roll-out plan Phase 0 districts (white) are those districts in

which PBF was piloted Cyangugu = Nyamasheke + Rusizi districts Butare = Huye + Gisagara districts BTC = Rulindo + Muhanga + Ruhango + Bugesera + Kigali

ville

Phase 1 districts (yellow) are districts in which PBF is being implemented in 2006, following the ‘roll-out plan’

Phase 2 districts (green) are districts in which PBF is not yet phased in; these are the so-called ‘Phase 2’ or ‘control districts’ following the roll-out plan. According to plan, PBF will be introduced in these districts by 2008. 13

Baseline Health Facility SamplePhase I

(Intervention district)Phase II

(Control districts)

DistrictTotal Facilities District

Total Facilities

Kibungo 11 Kirehe 10Nyanza 2 Kamonyi 9Gakenke 11    Rwamagana 3 Nyagatare 13Gatsibo 8    Kayonza 11    Nyamasheke 2 Karongi 17Ngororero 6    Rutsiro 10 Nyabihu 10Nyaruguru 9 Nyamagabe 13Burera 10 Musanze 11Total 83 Total 83

TOTAL 166 HEALTH FACILITY

04/21/23

16

Program Implementation Timeline

Jan-06 Mar 06 Jun-06 – Sept 06 2007 Feb-08-Apr 08 Apr-08

Phase I Start of intervention

Program Implementation

Phase II

Start of intervention

Impact Evaluation

SURVEYS Baseline

(General Health) Baseline

(HIV/AIDS)

Follow-up

16

More money vs. More incentives Incentive based payments increase the

total amount of money available for health center, which can also affect services

Phase II area receive equivalent amounts of transfers average of what Phase I receives Not linked to production of services Money to be allocated by the health center Preliminary finding: most of it goes to salaries

18

The baseline has 4 surveys

General Health facility survey (166 centers)

General Health household survey (2,016 HH)

HIV/AIDS facility survey (64 centers) HIV/AIDS household survey (1994 HH)

19

Baseline Field Sampling: GH HH Randomly selected FOUR CELLS per HF

Done in SPH using Epi Info and given to team leaders Randomly selected THREE ZONES per cell

Done by GH HH team leader in field Obtained household lists for each of the zones and

randomly selected ONE HOUSEHOLD per zone Done by GH HH team leader in field with zone/village

leader Random sample of 12-13 households with children

< 6 years old per HF 2,159 households.

Baseline Field Sampling: GH HH

HEALTH FACILITY

CELL 1 CELL 2 CELL 3

HH 1

CELL 4

HH 4HH 3HH 2 HH 6HH 5 HH 7 HH 9HH 8 HH 10 HH 12HH 11

Validity of Sample

Require two different validations: Validate the sampling for the evaluation design

Diff in means tests between Phase I and Phase II to determine if intervention and comparison groups balanced at baseline

Validate the quality of data Compare descriptive stats to other sources of national

data (i.e.: 2005 & 2007 DHS, MOH data)

04/21/23

23

Analysis PlanAll analyses will be clustered at the district

level Compare the average outcomes of facilities

and individuals in the treatment group to those in the control group 24 months after the intervention began.

Use of multivariate regression (or non-parametric matching) : control confounding factors

Test for differential individual impacts by: Gender, poverty level Parental background (If infant : maternal

education, HH wealth) 23

04/21/23

24

Difference in differences models To test the robustness of the analysis Control sample (both observed and unobserved)

heterogeneity A two-way fixed effect linear regression:

Where:

is an outcome variable for facilitiy i in period t

or for individual i who lives in the catchment area of facility i,

is an indicator of whether facility i

it it k itk i t itk

it

it

y PBC B X

y

PBC

is being paid by PBC in period t,

are time varying control variables,

i is a facility fixed effect,

is a year fixed effect,

is an error term.

itk

t

it

X

24

SECTION 3 :Descriptive Results from General Health Baseline

25

GENERAL HEALTH FACILITIES

BASELINE SURVEY

26

27

General Health Centers Survey:Content

All 166 centers in General Health General characteristics Human resources module: Skills,

experience and motivations of the staff Services and pricing Equipment and resources Vignettes: Pre-natal care, child care,

adult care VCT, PMTCT, AIDS detection services

Exit interviews: Pre-natal care, child care, adult care, VCT, PMTCT

Classification of facility 28

Baseline Health Facility:Main sources of revenue

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Primary Source

Secondary Source

Tertiary Source

Baseline Health Facility:Human Resources

On average:

1 doctor for every 31,190 individuals,

1 nurse for every 4,835 individuals

30

Obs Mean Std. Dev Min MaxNurse A1 165 0.18 0.45 0 3Nurse A2 166 6.31 8.32 0 70Nurse A3 164 0.29 0.65 0 4Auxilliary Staff >1yr 166 1.14 1.61 0 12Auxilliary Staff <1 yr 161 0.78 1.27 0 6Laboratory Technician 165 0.89 1.65 0 20Full-time Support Staff 166 3.87 3.07 0 28

Baseline Health Facility: Services

Category Service% Facilities

Providing Service

Avg. Years Providing Service

Avg. Charge for Service (excluding registration fee)

BCG 98.81% 16.78 -DPT 98.81% 11.91 -Polio 98.81% 16.51 -Measles 98.81% 16.4 -Tetanus 98.21% 16.48 -Children <5 yrs 100% 20 RWF 158.99Adults 100% 19.76 RWF 164.48Child Grow th Monitoring 86.31% 18.03 RWF 16.46Nutritional Information 92.81% 16.83 -Prenatal Care 98.21% 15.51 RWF 35.31Normal Delivery 89.22% 18.6 RWF 717.76Ceasarian Delivery 1.95% 30.75 RWF 892.86Assisted Delivery 9.03% 17 RWF 513.64Home Delivery 14.19% 11 RWF 41.67Blood Transfusion 2.58% 36.5 -Pill 84.05% 10.96 RWF 10.23Injection 82.82% 10.72 RWF 12.59Implant 18.47% 12.2 RWF 85.29Male condoms 71.78% 9.05 RWF 2.78IUD insertion 12.74% 17.17 RWF 13.04Female sterilization 0.64% . -Male sterilization 0.64% . -Diagnosis 36.36% 10.17 RWF 59.09Treatment 49.70% 10.51 RWF 6.63Malaria Treatment 97.56% 17.01 RWF 332.91

Family Planning

TB

Curative Care

Immunizations

Summary of Services Provided

Delivery Services

Baseline Health Facility: Lab Tests

Number of Obs. cstd

Number of Obs. cstd

Anemia 80 37.50% 0.487 84 29.76% 0.460 -1.046

Pregnancy 80 56.25% 0.499 84 34.52% 0.478 -2.846

Blood Count 80 1.25% 0.112 84 3.57% 0.187 0.960

Urine dip 80 50.00% 0.503 84 60.71% 0.491 1.380

Stool Sample 80 8.75% 0.284 84 3.57% 0.187 -1.385

Parasites 77 89.61% 0.307 84 94.05% 0.238 1.029

Malaria 80 50.00% 0.503 84 53.57% 0.502 0.455

Malaria dip 79 55.70% 0.500 84 72.62% 0.449 2.277

TB 80 36.25% 0.484 84 40.48% 0.494 0.553

HIV 80 62.50% 0.487 84 55.95% 0.499 -0.849

Gonorrhea 80 18.75% 0.393 84 21.43% 0.413 0.425

Syphilis 80 10.00% 0.302 84 8.33% 0.278 -0.368

Pap smear 80 38.75% 0.490 84 36.90% 0.485 -0.242

Lab Tests

Phase I (Intervention) Phase II (Control)

T-Stat

Baseline Health Facility:Overall Patient Satisfaction

4.08%

4.83%

14.66%

76.44%

0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00%

Very Unsatisfied

Not Satisfied

Very Satisfied

Satisfied

Baseline Household Data: Education37

 Sample: ages 6 and up

Variable Nr. Obs Mean Std. Error Nr. Obs Mean Std. Error T-stat

Ever attended 3104 79.68% 0.011 4042 80.88% 0.010 -0.825

No schooling 2458 10.26% 0.007 3252 10.88% 0.007 -0.606

At least some Primary 2458 82.93% 0.010 3252 79.97% 0.010 2.195 **

At least some Secondary 2458 6.46% 0.008 3252 8.42% 0.007 -1.841 *

Attended school in last12 months 2391 41.13% 0.015 3181 41.34% 0.012 -0.112Able to read Kinyarwanda 2532 63.59% 0.015 3359 66.37% 0.014 -1.375

PHASE I (Intervention) PHASE II (Control)

Baseline Household Data: Assets

Variable Nr. Obs Mean Std. Error Nr. Obs Mean Std. Error T-stat

Complete sofa set 921 2.18% 0.005 1147 3.08% 0.007 -1.073

Radio 921 49.63% 0.023 1147 50.27% 0.017 -0.221

Radio-cassette or music system 921 4.29% 0.009 1147 5.89% 0.008 -1.274

Telephone Mobile 921 1.07% 0.005 1147 3.50% 0.007 -2.778 ***

Mosquito nets 921 22.53% 0.023 1147 28.00% 0.027 -1.542

Sewing machine 921 0.76% 0.003 1147 1.18% 0.004 -0.883

A bed 921 62.82% 0.023 1147 58.71% 0.022 1.290

Wardrobe 921 4.53% 0.009 1147 6.07% 0.010 -1.155

Metalic library 921 0.57% 0.002 1147 1.28% 0.004 -1.645

Table 921 63.47% 0.022 1147 63.40% 0.021 0.021

Chair 921 85.48% 0.018 1147 84.82% 0.019 0.255

A bicycle 921 16.37% 0.020 1147 17.26% 0.027 -0.262

PHASE I PHASE II

Baseline Household Data: Prenatal Care

40

Variable Nr.Obs. Mean Std.Error Nr.Obs Mean Std.Error T-stat

Of all births since Jan. 2005

In facility birth 1238 32.05% 0.026 1462 34.87% 0.025 -0.777

Of most recent pregnancy

Times received PNC 723 2.781 0.046 900 2.687 0.052 1.368

Injection to prevent tetanus 728 74.52% 0.023 900 72.56% 0.019 0.654

Phase I Phase II

Baseline Household Data: Child Immunization

42

Variable Nr.Obs. Mean Std.Error Nr.Obs Mean Std.Error T-stat

12-23 months oldfully_immunized 262 75.36% 0.03 295 77.04% 0.04 -0.33

12-71 months oldfully_immunized 1154 63.77% 0.03 1387 65.28% 0.02 -0.43

Phase I Phase II

Baseline Household Data: Child Recent Illness and Symptoms (<6 years)

30.79%

20.65%

16.37% 15.85%

11.71%

7.16%

2.77% 2.20%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

Fever Respiratory Diarrhea Headache Malaria Eye Infection Ear Infection Accident

Baseline Household Data:Child Biomarkers (<6 years)

18.67%

7.51%

22.16%

27.82%

19.53%

7.33%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

Anemic Malaria Fever and Malaria

Phase I

Phase II

Validity of Sample and Data46 Evaluation design

Of 110 key characteristics and output variables of HF,

the sample is balanced on 104 of the indicators.

Of 80 key HH output variables, the sample is balanced

on 73 of the variables. Quality of data

HH Results comparable to the 2005 DHS, MOH data

Follow-up Field Sampling: GH HH Result:

In total 2159 were suppose to be surveyed in the catchment area of 167 facilities.

1888 (87%) were interviewed in 2006 and 2008 267 (12%) were replaced 4 (0.2%) were not found and not replaced

% of replacement by region: South (24%), North (14%), East (4%) and West

(13 %)

HIV HEALTH FACILITIES BASELINE SURVEY

53

Baseline Health Facility Sample

Phase I Phase II MAP TOTAL SAMPLEFacility Level Data HIV/AIDS Services and Human Resources 18 18 28 64Provider Level Data General Providers 18 18 28 64 PMTCT Providers 17 17 27 61 VCT Providers 17 18 27 62TOTAL 52 53 82 187Patient Level Data PMTCT Patients 108 108 107 323 VCT Patients 116 87 209 412TOTAL 224 195 316 735

Assessment of the quality of ARV and HIV/AIDS service delivery: Health Facility Sample

Baseline Household Sample

38 ARV facilities for household sampling 13 MAP; 10 Phase I; 15 Phase II ARV sites for HH sampling

Original sample of 1,487 patients from health facilities and associations

Final sample consists of 1,961 households and 7,494 individuals

Patient Treatment: Distribution

Number %

No Treatment 196 13.18%

Cotrimoxazole 439 29.52%

ARV 852 57.30%

TOTAL 1487 100.00%

Household Level: Sample Size of Patients

We find there is an additional sample of individuals in our household sample who self-report HIV+ status

Based on this, final distribution of patients:

Patient Sample: Distribution

Number %

Phase I 701 39.47%

Phase II 694 39.08%

MAP 381 21.45%

TOTAL 1776 100.00%

Baseline Health Facility:Main sources of revenue

Obs Std Obs Std tstat

funding_centralgovt 17 11.76% 0.33 18 33.33% 0.49 1.53

ratio_centralgovt 16 6.25% 0.25 17 21.76% 0.35 1.45

funding_province 17 23.53% 0.44 17 5.88% 0.24 -1.46

ratio_province 17 10.67% 0.25 16 1.42% 0.06 -1.42

funding_healthdistri 17 0.00% 0.00 17 5.88% 0.24 1.00

ratio_healthdist 17 0.00% 0.00 16 1.67% 0.07 1.03

funding_admindistric 17 11.76% 0.33 17 0.00% 0.00 -1.46

ratio_admindist 17 2.15% 0.06 16 0.00% 0.00 -1.38

funding_userfees 17 82.35% 0.39 17 70.59% 0.47 -0.79

ratio_userfees 16 28.07% 0.37 16 27.11% 0.32 -0.08

funding_mutuelles_cu 17 52.94% 0.51 18 61.11% 0.50 0.48

funding_mutuelles_me 17 64.71% 0.49 18 61.11% 0.50 -0.21

ratio_mutuelles 17 50.48% 0.41 17 42.38% 0.40 -0.59

donsgov 17 4586525 3057811.00 16 1.55E+07 35600000.00 1.25

donsnongov_dum 6 100.00% 0.00 12 75.00% 0.45 -1.33

donsnongov 12 7974796 14100000.00 14 8591750 7649821.00 0.14

comite_gestion 18 33.33% 0.49 18 44.44% 0.51 0.67

comite_sante 18 55.56% 0.51 18 44.44% 0.51 -0.65

other_management 18 11.11% 0.32 18 11.11% 0.32 0.00

General Facility Information: Finances

Phase IIPhase I

Baseline Health Facility: Services

HIV/AIDS Services Offered

% Offer Service

HIV counseling 96.83%

Rapid HIV test 91.8%

Elisa HIV test 7.32%

CD4 Count 12.73%

Viral Load 2.5%

Systematic Testing of Blood Donations 19.51%

VCT 88.14%

Distribute Condoms 93.22%

PMTCT 93.10%

Preventative Cotrimoxazole Treatment 100%

Treatment of OIs 91.8%

ARV 80.7%

TB Diagnosis 84.75%

TB Treatment 92.98%

Nutritional Support 80.85%

Baseline Health Facility:Overall Patient Satisfaction

6.05%

0.47% 1.40% 1.14%

58.60%

49.77%

54.88%

48.86%50.00%

43.72%

49.77%

35.35%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

Time Attended Confidentiality Staff Attitude Cost medications

Not satisfied

Satisfied

Very Satisfied

HIV Patients vs. Non-Patients:Mortality in the Household

Death in the last 5 years

7.61%

16.84%

0.00%

2.00%

4.00%

6.00%

8.00%

10.00%

12.00%

14.00%

16.00%

18.00%

Patients Non-Patients

Patients vs. Non-Patients:Mortality in the Household in the last 5 years

35.08%

71.88%

15.79%

94.09%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

Illness related Death Sold Assets

Patients

Non-Patients

HIV Patients vs. Non-Patients:Health Care Utilization

Illness in last 6 months

69.67%

17.32%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

Patients Non-Patients

Patients vs. Non-Patients:Sexual Behavior

39.63%

76.74%

56.14%

37.42%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

Sex in the last 12 months Condom during last encounter

Patients

Non-Patients

Follow-up Field Sampling: HIV HH Objective:

Return to the same households to create panel data set (2006-2008)

Print baseline roster (names, codes) and keep consistent across waves

Account for household members who left and new arrivals from 2006-2008

Follow-up Field Sampling: HIV HH Result:

In total 1996 were to be surveyed in the catchment area of 38 ARV facilities.

1,716 (87.06%) were interviewed in 2006 and 2008

280 (12.94%) were not found and not replaced % of missing by region:

South (12.6%), North (12.7%), East (7%) and West (16.4%)