1

When Support for the Poor is Poor Support:Income-tested social assistance programs

in Russia

Emil Daniel TesliucHNDSPWorld Bank

2

Objective

Using the example of an evaluation of 3 income-tested social assistance programs from Russia– (whose aim was to assess whether mediocre

targeting accuracy is due to program design or implementation)

Introduce two types of evaluation:– Process evaluation– Assessment of targeting accuracy

Highlight some innovative instruments used in the evaluation

3

The Issue Addressed by The Study: Russian means tested programs have low targeting accuracy Why?

Share of Beneficiaries from the Poorest QuintileNon-contributory cash transfers in US, LAC and ECA

0

20

40

60

80

100

Food

TA

NF

Bra

zil

Chi

le

Jam

aica

Mex

ico

Arg

enti

na

Rom

ania

Bul

gari

a

Lit

huan

ia

Hun

gary

Est

onia

Pola

nd

Mol

dova

Kyr

gyzs

tan

Alb

ania

Bel

arus

Serb

ia

Arm

enia

Rus

sia

Geo

rgia

Bos

nia

Uzb

ekis

tan

Mac

edon

ia

Aze

rbai

jan

Taj

ikis

tan

US LAC ECA

%

Programs using:

Income testing

Proxy-meanstesting

4

The Issue Addressed by The Study

Money for the poor end up in the pockets of the rich.

Деньги для бедных достаются богатым

Why? Examine the role of program design versus implementation. Source: Komsomolskaya Pravda, May 25, 2007

5

Objectives and Criteria

Objectives of the study– Assess the design and implementation practices of income-tested

programs in 5 regions

Criteria used to assess the effectiveness of income-tested social assistance programs in reducing poverty:

– Coverage of the poor– Leakages of funds to the non-poor– Generosity of the program– Cost-efficiency (share of administrative costs in program budget)– Horizontal equity (similar treatment for participants from different

locations)

6

7

Instruments and Methodology

Qualitative Quantitative Open-ended interviews with in-

take workers, rayon heads and oblast heads/ministers of the Social Assistance and Employment Offices

Focus groups with recipients Observation of eligibility

determination processes Observation of the premises Analysis of reporting and

information flows Regional labor market

assessments

Evaluation of targeting accuracy, based on representative household surveys (NOBUS, HBS, Tomsk and Leningradskaya oblasts)

Analysis of randomly selected case files

Analysis of administrative costs and use of time by program staff

Collection of administrative and statistical data on the programs

8

9

10

Using Administrative Data

Since 2003, poverty fell from 33% to 19% in the region of Tver, but caseloads remained the same Source:RosStat for poverty data, Tver SP Department for caseload data

0

20

40

60

80

100

120

Beneficiaries, Childallow ances

Beneficiaries, Housingallow ances

Nu

mb

er o

f b

enef

icia

ries

per

yea

r

2003 2005

0

5

10

15

20

25

30

35

2003 2005

Poverty rate, %

11

Large differences between program coverage and official poverty,by type of settlement, Tver oblast, 2005

1925

6

19

43

57

88

63

4

13

35

20

8

29

47

28

0

20

40

60

80

100

Tver city Otherurban

Rural Total Tver city Otherurban

Rural Total

Housing Allowance Child Allowance

po

pu

lati

on

(%

)

Coverage of the program Official poverty rate

12

Random Review of Applicants’ Files

What it is?– Similar with the Quality-Control Review process of SSN programs

in OECD countries– Assess whether the eligibility and recertification decisions have

been correctly made, given the documentary evidence available in the file

– How often do beneficiaries report changes in their circumstances?

Uses of the technique: – Estimate the level of error in determining eligibility or

recertification

13

14

15

Random Review of Applicants’ Files: Error level is low, regional variation

Beneficiary' files with incomplete documentation or errors

0 10 20 30 40 50 60 70 80

Tomsk oblast

KChR

Kalmikia

Tatarstan

Tomsk oblast

KChR

Tver oblaast

Kalmikia

Tatarstan

Chi

ld A

llow

ance

sH

ousi

ng a

nd u

tili

tyal

low

ance

s

% of case file reviewed

Incomplete documentation Error in estimating eligibility or benefit level

16

Comparing Findings from the Review of Applicants’ Files with Survey Data

0 10 20 30 40 50 60 70 80 90 100

Self-employment

Farming, hunting, fishing

Scholarships

Remittances

Wages

Self-employment

Farming, hunting, fishing

Scholarships

Remittances

Wages

Hou

sing

all

owan

ces

Chi

ld a

llow

ance

s

% families reporting a particular type of income

Quality control review, 2006 Difference in the NOBUS, 2003

Many beneficiaries do not declare all the incomes they earn(Example from one region)

Source:NOBUS 2003 and case-control review, 2006

Triangulate with survey data to assess whether information supplied by clients (and not verified) is reliable

17

Administrative cost survey / time use survey

Estimate how much is spent on administrative costs Estimate the time devoted to check documentary

evidence Estimate productivity indicators (beneficiary/staff ratios) Benchmark this information across sites and programs,

within and outside Russia Example of its uses: spot programs with unusually high

admin costs (TSA program in Tomsk city has admin cost ratio of 100%)

18

Payroll of staff, filing the applications, calculating allowances, or otherwise

dealing with the recipients

Share of other direct admin. costs (e.g.,

costs of post or bank services)

Share of payroll of heads of social services, office

and auxiliary staff, drivers etc.

Share of overhead costs

Direct administrative costs Indirect administrative costs

Administrative costs

Sums of transfers

Services for beneficiaries (for example, psychologist)

Other direct costs

Transfers to beneficiaries Total Cost of

the program

19

20

Administrative Costs per Beneficiary are small for the child allowance program

0% 50% 100% 150% 200% 250%

Albania

Armenia

Bulgaria

Lithuania

Romania

RF - Child allowances

RF - HUS Allowances

Brazil

Colombia

Median value

Index of cost per beneficiary(median value is 100%)

21

The Workload of the SP Staff is too high in Russia

Number of Beneficiaries Per One Full-time StaffSelected means-tested programs

93

180

1655

352

348

0 200 400 600 800 1000 1200 1400 1600 1800

Romania, GuaranteedMinimum Income

Armenia, Poverty FamilyBenefit

Russia, Child Allowances

Russia, Housing allowances

Russia, Regional targetedsocial assistance

22

Observation guide: High regional/ rayonal heterogeneity

Recall period of the income used to determine eligibility

For how long the benefit is assigned/ how frequent is mandatory recertification

Does the applicant have to submit the information on all types of income, stipulated by the law, or does he give the only the information he considers relevant

Does the applicant need to collect papers from other units of social assistance office

What agencies regularly give information that is used for verification

How often (what % of cases) are home visits conducted

Is income from land plot and subsidiary agriculture calculated?

How often do you check information submitted by applicants using the third party (calling tax inspections, enterprise etc.)

Are there appeal commissions

How many cases of clients’ fraud (or %) were last year

Duties defined by contract, instructions, other formal means?

Was there quality control by regional authorities of mistakes in granting the benefit this year?

23

24

Findings from the Process Evaluation: Why is targeting accuracy so low?

Under-reporting of income is widespread Verification of the reported income is infrequent or absent Mechanism to deter, detect and reduce fraud and error are

insufficiently developed Some program functions that are critical to ensure accurate

targeting are understaffed and underfinanced Arbitrariness in eligibility determination is widespread Quality control or procedural guidance from federal level is

lacking The authorities have not known about the magnitude of this

issue due to lack of evaluation and monitoring

25

26

DO SVIDANIYA!