Embed Size (px)
Citation preview
Rio de Janeiro, 04/10/2014
Impact Evaluation of Social Protection Programs: household surveys and administrative data
www.ipc-undp.org
2
Introduction
The basic question of an Impact Evaluation of a Social Protection Program: “Have the Program made any difference in the life of the participants?”
As easy it is to ask as hard it is to answer.
3
Introduction
1. It implies the reformulation as: What would have happened to the participants’ life it they were not in the program?2. It implies a causal relationship between the programs and the outcomes 3. The question is usually asked after the program had been implemented 4. It requires a very sound methodological design5. It is a lengthy usually 2 years;6. It is costly (USD 200 thousand to over 2 million Grosh et al)
4
Introduction
The integration of the impact evaluation survey with administrative data and the national household surveys enhances the reliability of the results and lower the costs of the evaluation. Three examples: 1. The impact evaluation of the Bolsa Familia (AIBF in Brazil)- phase one.
2. The National Social Protection Monitoring (NSPMS in Yemen)
3. The spatial matching of administrative records with national household surveys)
5
AIBF Phase 1 (Brazil)
Year 2005
Sample Size: 15, 426 households 65,000 individuals
Design: •In the 41 largest municipalities the Primary Sampling Unit (PSU) was the Census Tract•The other municipalities were aggregated to have a minimum size of 50,000, they constitute the PSU and the census tract was the Secondary Sampling Unit (SSU)•Census tracts were screened to identify families beneficiaries, families non beneficiaries but registered in the registry system (CadUnico) and those who were neither beneficiaries or registered, composed as 3:6:1
6
AIBF Phase 1 (Brazil)
The individual and household records were linked to the CadUnico records with the objective of:
•Correct the information of the NIS of the survey•Assess the quality of self reported groups of study•Compare the effect on education on the survey and the matched groups•Compare the results on education of Propensity Score Matching and Regression-Discontinuity (Sharp) methods
7
AIBF Phase 1 (Brazil)
The matching of the AIBF survey and Administrative Data (CadUnico) combined the Probabilistic and Deterministic methods of record linkage
Information from CadUnico used were: Full Name, Municipality of Residence, Birth Date and Sex
8
A sketchy list of activities:
1. Cleaning and standardization of Data Sources2. Linkage by deterministic method 3. Classification of the record: Matched and Non Matched. (73.8% of NIS and 35% of families were matched)4. For the Non Matched : Blocks of Beneficiaries and Non Beneficiaries in the Survey and Soundex of the first name + soundex of the last name + municipality + sex5. Classification Total matching /Partial Matching6. Manual Revision7. Final Classification 73.5% of individuals 30% of families8. Reallocation of the Treatment and Comparison Groups9. Education: attendance, dropouts, progression, working, non progression by PSM and RD Sharp
AIBF Phase 1 (Brazil)
9
The overall results of effects on education did not differ greatly in the survey and in the matched databases by the PSM method, both indicating the same signals in the differences on drop outs, progressions, repetitions and out-schools children. The RD-Sharp the utilized the income declared in the CadUnico revealed more conservative results than the PSM.
SourceRacchumi Romero, J. R. Utilizando o Relacionamento de Bases de Dados para Avaliação de Políticas Públicas: uma aplicação para o Programa Bolsa Família. Tese de doutorado, Cedeplar, UFMG, 2008
AIBF Phase 1 (Brazil)
10
•A Longitudinal Quarterly Panel Survey to monitor socio-economic indicators and assess the targeting and possible impacts of the Social Welfare Fund (SWF)
•Year October 2012- September 2013
•Sample Size: 6,397 balanced size out of 7,152 households in the first round.
NSPMS-Yemen
11
Design: •Census Enumeration Areas (EA) are the PSU stratified by Governorates. 30 EA in each Governorate.
•In each EA, 12 households were selected using a stratified simple random sampling procedure. Households were selected from the three groups identified in the listing of the household in the selected enumeration areas, namely, beneficiary of Social Welfare Funds (SWF), potential beneficiary of SWF (registered, but not receiving it yet), and non-beneficiary and non-registered. •Each household was interviewed four times
NSPMS-Yemen
12
The SFW
•The SWF expansion was due to the incorporation of new beneficiaries into the programme. New beneficiaries were identified in the 2008 Comprehensive Social Survey (CSS) and selected through a proxy means testing (PMT), but were only systematically incorporated into the programme from October 2012 onwards.
•New beneficiaries correspond to about 33 per cent of the total number of beneficiary households.
•Some new beneficiaries received their first payment in the first quarter of 2011, after that, payments were suspended and only resumed in the last quarter of 2012. A lump sum payment varying from YER 30,000 to YER 60,000 corresponding to the 5 quarters in arrears was paid to them
NSPMS-Yemen
13
The matching with administrative date were necessary to:
•To assess the distribution of the different categories of SWF beneficiaries.
•To divide the sample of beneficiaries into old beneficiary (pre-2008) and new beneficiary (post-2008) since they were selected differently. Only the latter was chosen based on a PMT. It was necessary to use the administrative data because almost 50% of the sample replied they did not know when they had started receiving the SWF benefit. Thus, one could not classify the two groups (new and old beneficiary) based on the survey information.
NSPMS-Yemen
14
Matching of the survey info and SWF administrative information was based on “number of SWF card”, name of beneficiary, and when name did not match on other characteristics of the main beneficiary.
734 SWF beneficiaries in the NSPMS sample that were not matched with the SWF administrative database).
New parameters: the total amount of SWF transfers received during round 1 (October-December 2012) and the self-reported year of accreditation into the programme.
This procedure yield similar estimates of new beneficiaries in both admin data and NSPMS sample: 33%, out of 1,5 million total beneficiaries.
NSPMS-Yemen
15
Some results
•The propensity score estimates confirmed that new SWF beneficiaries were more likely to be poor (as identified by the PMT) and have higher predicted probabilities of SWF receipt than the comparison group members. •As for expenditures on food, we find that all of the estimated effects are positive, and most are also statistically significant, particularly, for old SWF beneficiaries.
•As for household income and agricultural production we find that income from work and from agricultural production are both significantly reduced among the old SWF beneficiary households.
NSPMS-Yemen
16
• New SWF beneficiaries are more likely to make investments in agricultural inputs and, they are also significantly more likely to possess livestock than non-beneficiaries.
• Higher rates of child labour and unpaid family work for female SWF new beneficiaries ages 6-11 (compared to non beneficiaries) while school
• Reductions in the probability that both male and female children of younger (6-11) and older (12-14) ages were absent from school
• Higher rates of unpaid family work for males 6-11 and 12-14 years (also new beneficiaries).
Source: Veras, Fabio et al. National Social Protection Monitorin Survey. IPC-IG/UNDP UNICEF-Yemen, 2014
NSPMS-Yemen
17
• New SWF beneficiaries are more likely to make investments in agricultural inputs and, they are also significantly more likely to possess livestock than non-beneficiaries.
• Higher rates of child labour and unpaid family work for female SWF new beneficiaries ages 6-11 (compared to non beneficiaries) while school
• Reductions in the probability that both male and female children of younger (6-11) and older (12-14) ages were absent from school
• Higher rates of unpaid family work for males 6-11 and 12-14 years (also new beneficiaries).
Source: Veras, Fabio et al. National Social Protection Monitorin Survey. IPC-IG/UNDP UNICEF-Yemen, 2014
NSPMS-Yemen
18
Multiple Cross Section Surveys In most of those surveys, the Primary Sampling Units (PSU) of the multiple cross section surveys are constant throughout the years. The spatial matching of the (future) beneficiaries with the PSU, using the address of the future beneficiaries. Matching procedure will define the comparison group in the same PSU. Additional questions on participation in the Social Programme should be added to the National Survey.
Spatial Matching of Administrative Records with National Household Survey
19
National Panel Surveys:The matching of (future) beneficiaries with the Panel subjects. The beneficiaries will constitute a sub-sample of the of the panel subjects. Matching procedures, among the Panel subjects, such as PSM will define the comparison group at the baseline survey (for the evaluation purpose). The frequency of the interviews will follow the same schedule of the larger project.
Spatial Matching of Administrative Records with National Household Survey
