Measuring the Quality of Education and Health Services:

The Use of Perception Data from Indonesia

Basab DasguptaAmbar Narayan

Emmanuel Skoufias

PRMPR- World BankApril 21, 2009

Motivation & Scope-1

Increasing trend of decentralization of service delivery to local governments Decentralization increases accountability, increases citizen participation and political

engagement, improves public service delivery, allocative efficiency and fiscal administration

Good measures for local government performance necessary for evaluating the impact of decentralization Many of these tools include subjective instruments that gauge citizen perceptions (citizen

report cards, community scorecards, facility exit polls, and citizen satisfaction surveys) , (Amin, Das, and Goldstein, 2007).

Can satisfaction-related questions be valuable in measuring the quality of public services, specifically in health and education?

Motivation & Scope-2

satisfaction surveys are appealing A quick and easy way for policymakers to measure the impact of

governance reforms on government performance, particularly for sectors where measurement of service quality is not easy, provided citizen satisfaction is closely correlated with the actual quality of services.

Less time and labor intensive than facility surveys and public expenditure tracking surveys)

BUT…. there is little consensus on whether citizens’ satisfaction reflects the actual quality of services satisfaction surveys Whether useful and if so How to use it

Motivation & Scope-3

Need better understanding of what factors influence satisfaction What household and community level factors (besides

quality) play a role? Decentralization improved outcomes is based on premises (i) increased accountability of service delivery increases performance and (ii) citizens are able to discern between good and bad gov’t and then influence local authorities.

Concerns w/ Satisfaction Surveys

e.g. Indonesia-GDS2 survey

May get High Reported Satisfaction due to cultural norms or social pressure rather than the superior quality of services.

Absence of a common baseline makes itDifficult to interpret the dataTricky comparing data-points across

regions and countries

Figure 1: High satisfaction with services in Indonesia

Figure 2: Positive perceptions of change in quality of services in last 2 years- Indonesia

Health services Health services

58.1

1.2 0.8

32.2

7.7

0

20

40

60

80

100Satisfied

Quite

Satisfied

Quite

unsatisfied

Not

Satisfied NA

% o

f h

ou

seh

old

s

20.6

3.0 5.8

71.7

0

20

40

60

80

100

Better Same Worse NA

Education services Education services

50.2

30.1

10.71.7

7.2

0

20

40

60

80

100

Satisfied

Quite

Satisfied

Quite

unsatisfied

Unsatisfied

NA

72.9

14.15.8 7.2

0

20

40

60

80

100

Better Same Worse NA

Source: GDS-2 (2006)

When most users claims to be “satisfied” with services...

Average satisfaction with public health and education facilities surprisingly high in GDS-2 Among five options (scale of 1 to 5), option 1 or 2 (somewhat satisfied

or better) chosen by around 90 and 80% of hholds for health and education respectively

Inconsistent with the poor reputation of health and education services in Indonesia, also supported by more objective measures of quality from surveys like GDS-2

Problem not unique to Indonesia, but not universal in surveys either High variation in satisfaction with education and health services in a

number of countries where CWIQ-type surveys have included questions on satisfaction

2006-07 survey in Pakistan showed 35% satisfaction rate for public health facilities; 2007 survey in Sierra Leone showed satisfaction rate of ~40% for public schools

High reported satisfaction in Indonesia probably attributable mainly to cultural norms

Key question: is there useful information content in the Indonesia satisfaction data?

7

GDS-2 survey--1 Survey in May-Sept 06 to assess governance and

local public service delivery in Indonesia collecting data on quality and satisfaction from households,

communities, and facilities

Household sample (8544 hholds) nationally representative – 1068 PSUs (hamlets or dusun), in 89 districts (kabupaten/kota), 267 sub-districts (kecamatan) and 534 villages

Facility data collected from a sample of health and education facilities – guided by which facilities were reported as most frequently used by households Health facility data from the 6 community health centers

(puskesmas) most frequently mentioned by households within each sub-district (kecamatan)

School facility data collected from the most frequently used public elementary school in a village and public junior high school in a sub-district 8

GDS-2 survey--2Thus data on both household satisfaction

and objective measures of facility quality are available for 52% of households using public health facilities and 57% of those using public education facilities.

Sources of possible selection bias Households choosing public facilities may systematically differ

from those choosing other types of facilities Restricting the facility sample to the most frequently used

facilities instead of a random sample of all available facilities can potentially add to the bias

9

Model--1 Simple expectancy disconfirmation model

S= f (Quality – Expectations) Where expectations=g(hh char & experience)

A modified expectations disconfirmation model Expectations play a role in the choice of the type of facility; expectations

proxied by household and community level factors Conditional on the choice of a service provider, reported satisfaction with the

service facility is a function of the actual quality of the service and governance (service is an “experience good” experience it only after choosing it).

Caveats: quality of service measured in terms of inputs in the production of services and NOT outcomes (e.g., achievement scores, z-scores).

Model--2Two-stage Heckman model to correct for selection

bias

1st stage selection equation predicts the propensity of households to use a facility for which objective data on quality are available

2nd stage equation examines how satisfaction (S) correlates with indicators of quality and governance, conditional on the choice of a service provider

Models estimated separately for samples from poor and rich districts, and in a pooled sample Accounts for difference in expectations between the

residents of rich and poor areas Rich districts = GRDP pc > median GRDP pc in sample of

88 districts

11

Defining the satisfaction variable in GDS-2 Useful information can be extracted by focusing on

variations in satisfaction level To exploit the variation, we define our dep. variable as a

binary S, which is =1 for all those who chose option 1, and =0 otherwise

For health services, S=1 for 58% of hholds; for education services, S=1 for 50% of households

Analyzing the determinants of S using two models

Simple heuristic model: correlating household characteristics with satisfaction, separately for health and education Illustrates the range of factors – that service providers have no

control over – influencing satisfaction with services among households

2-stage Heckman selection model: corrects for selection bias in facility choice

12

HEALTH

Health facility choice—1st stageExplanatory vars

S=1 if the household uses a public health facility (puskesmas) for which facility data is available, =0 otherwise

Explanatory variables Expectations proxied by household and community characteristics Other factors likely to affect choice of facility: location of facility

(in village or not), availability of information on corruption and the sources of that information

Wealth, rural-urban designation, and provincial location of district

14

Health facility choice—1st stageResults

Demographic/socio-economic characteristics play a statistically significant role in influencing whether or not a puskesmas is the facility of choice for a household

These characteristics are far more important for hh in poor districts than rich districts

Geographic location: relative to households in Java, households belonging to other regions are more likely to use puskesmas, with some differences between rich and poor districts

Household’s relative position in society (leadership position) or access to information do not seem to matter for choice of health facility

Caveat: these factors explain the choice of a certain type of public health facility for which facility data is available

15

Satisfaction with health-- 2nd stageExplanatory vars

Objective indicators of quality and governance environment Quality : (a) coverage area of puskesmas; (b) types of medical

support available from ancillary facilities; (c) quality of services (human resources and medical supplies); (d) infrastructure

For each category, multiple indicators combined into a single index created using Principal Component

Institutional /governance environment : (a) willingness to complain (voice), (b) govt responsiveness to complaints (accountability), and (c) an index of participation in the administration of health services

16

Satisfaction with health-- 2nd stageResults

Strong correlation between satisfaction and certain dimensions of service quality Namely, support available to the main puskesmas from ancillary facilities, the

quality of service care in terms of human and medical resources (for poor districts); quality of infrastructure has insignificant impact

Thus citizen satisfaction with health facilities, particularly in poor districts, correlated more with the availability of ancillary facilities, human personnel and medicines, rather than facility infrastructure

Institutional and governance indicators correlate with satisfaction, but with rich-poor differences Hholds in poor districts more likely to be satisfied when they participate more

in the administration of health services Hholds that complained about health services less likely to be satisfied with

health services; hholds in rich districts more satisfied when health services are responsive to complaints

All the governance indicators may be endogenous with perceptions of quality and satisfaction

18

EDUCATION

Education facility choice—1st stageExplanatory vars

Sample restricted to households that have at least one child of school age

Dep variable =1 if the household sends a child to a public school for which facility data is available, 0 otherwise; explanatory variables similar to 1st stage model for health

20

Education facility choice—1st stageResults

As expected, a household’s “selection” of a school is the result of a very different decision process from its choice of medical care Demographic and socio-economic characteristics matter less

for the choice of schools than puskesmas Access to information and social status of households seem to

matter for schooling choice and not for the choice of health facilities

Regional location of the household again plays a role: relative to households in Java, households belonging to other regions are more likely to send their children to public schools

Again, the model explains the choice of a certain type of school (public) for which facility data is available

21

Satisfaction with education--2nd stage

Explanatory vars Indicators of school quality : (a) quality of

infrastructure in school; (b) the quality of teaching staff; ; (c) student performance; and (d) coverage of students (enrollments, attendance) by the school

Institutional and governance environment: (a) information available to households about

bribery and corruption in education, (b) willingness to complain against service

providers, (c) provider’s responsiveness to complaints, (d) extent of participatory decision-making in school (e) coverage and implementation of BOS 22

Results from 2nd stage of selection model for satisfaction with public schools

All districts Poor districts Rich districts

Dependent variable: S =1,0 Coef. Coef. Coef.

Participatory mode of management 0.055* -0.043 0.124**

Index of Facilities 0.011 0.029** 0.002

Index of teacher qualities 0.007 -0.010 0.017

Index of BOS coverage 0.017* 0.002 0.027*

Index for student performance -0.009 -0.009 -0.016

Index for student coverage 0.002 0.026 0.001

Responsiveness of the service provider 0.013 0.075* -0.027

Complaints made (Voice) -0.087** -0.145** -0.063

Corruption information -0.100* -0.018 -0.169**

Bribery information -0.149* -0.062 -0.139

Dummy for urban -0.034 -0.015 -0.014

Region dummies (Java as reference):

Kalimantan -0.107** -0.214** -0.002

NTT -0.044 -0.080 -0.095

Sulawesi -0.123** -0.208** -0.051

Sumatra -0.149** -0.432** -0.036

_cons 0.610** 0.676** 0.496**

Satisfaction with education--2nd stage Results

Strong correlation between satisfaction and certain

dimensions of school quality in the pooled sample, but

important differences between rich and poor districts.

Satisfaction levels have significant correlation with: Better infrastructure facilities in schools (e.g. condition of classrooms,

availability of books, library, computers) – for poorer districts only

Participatory decision-making for school’s mission and vision (jointly by

school principal, teachers and the community) – for rich districts only

The coverage of a school by the BOS program and the progress in

implementation of BOS – for rich districts only

24

Satisfaction with education--2nd stage Results

Thus satisfaction in poor districts more influenced by the basic features of a school (e.g. condition of building facilities), whereas in richer districts influenced more by “second-generation” factors (e.g. participatory mode of mgmt) ?

Knowledge of bribery and corruption in education and complaints against schools have significantly negative effects on satisfaction, but again with differences between rich and poor districts

25

Comparing results: health vs. education--1

Key differences in what factors matter for satisfaction

with health and education services

While quality of infrastructure has no influence on

satisfaction with health facilities, quality of school

infrastructure has significant influence on satisfaction with

schools in poor districts

Indicators of quality (availability of personnel and medicinal

inputs) are key determinants of satisfaction in health

services; but teacher quality does not seem to correlate at

all with satisfaction in schools 26

Comparing results:health vs. education--2

For both education and health facilities, a more

participatory management of the facility induces

higher satisfaction. However, differences in how the

results vary across rich and poor areas For health facilities, households in poor districts are more likely to be

satisfied with higher participation in the administration of health services

For schools, households in rich districts are more likely to be satisfied when

management of schools is more participatory or the implementation of a

school-based management system is more advanced

Responsiveness to complaints about facilities improves satisfaction with

health facilities in rich districts only and satisfaction with schools in poor

districts only27

Comparing results:health vs. education--3

Important difference in how the regional

location of a household influences

satisfaction Matters only marginally for satisfaction with health facilities (and

has no effect for poor districts)

But satisfaction with schools likely to be much lower in the

poorer areas of the Kalimantan, Sulawesi and Sumatra regions

compared to the poor areas of Java region

28

Implications of our analysis

29

“Satisfaction” data has important information content

But requires finding meaningful variation in responses and models to account for selection bias and role of expectations in facility choice For health and education, satisfaction with facilities significantly

correlated (in the right direction) with measures of quality, governance and institutional environment of the facility

In many cases, collecting satisfaction data matched with facility level data is not practical Common practices: user surveys/citizen report cards, hhold

surveys

What does our analysis suggest for such “2nd-best” scenarios The design and use of instruments that just measure satisfaction

from hhold surveys, as a proxy for quality of services?

Implications for surveys collecting satisfaction data--1

30

Even if a linked facility survey is not possible, clear benefits in having a satisfaction survey collect as much information on the characteristics of households and communities as possible, including proxies for governance/institutional environment

Random sample administered at household level is likely to yield more representative results in most cases than a survey of a sample of users of a particular type of facility Better for correcting the selection bias arising from facility choice, in

contrast to a survey limited to the users of a particular type of facility Incorporating questions on satisfaction with basic services in household

surveys is becoming more common (E.g. CWIQ surveys combining questions on satisfaction with basic services with hhold and community characteristics)

In some cases where user surveys are the only practical option (e.g. when a service is used by a tiny % of the population), collecting hhold/community level information from users is recommended

Implications for surveys collecting satisfaction data--2

31

Even if most respondents appear to be satisfied (or not), useful information can still be extracted by using the variation in responses, rather than the strict meaning of the responses But the survey design must allow for that, for example through:

Multiple choice responses – variation in response more likely to occur when surveys phrase satisfaction-related questions with multiple options, as opposed to a simple “yes/no” or “satisfied/dissatisfied”

Being specific – More variation in responses likely if questions are specific to different aspects or features of a school or health facility, as opposed to a single “are you satisfied with school/health center” type question

Implications for education and health services in Indonesia--1

32

Which aspects of health and education services in Indonesia matter the most for user satisfaction and how these differ across richer and poorer districtsInfrastructure

Quality of personnel and inputs

Participation in decision-making, governance and accountability

Implications for education and health services in Indonesia--2

33

InfrastructureSatisfaction with health facilities related more

with access to ancillary facilities supporting the main puskesmas, rather than the physical infrastructure of the puskesmas

But for schools in poor areas, infrastructure appears to be high priority among parents of students

34

Quality of personnel and inputsQuality of human resource and medicinal inputs

in health facilities is a major concern among users in poor districts

Indicators of teacher quality and student performance do not seem to matter much for user satisfactionDoes not necessarily imply that households are

ambivalent about education quality– rather that these indicators may not reflect the aspects of “quality” households care most about

Implications for education and health services in Indonesia--3

35

Participation in decision-making, governance and accountability Greater degree of community participation in decision-making for

facilities and better responsiveness of service providers to complaints appear to improve satisfaction

In education, satisfaction positively correlated with the extent of implementation of BOS (decentralization of service – e.g. school-based management, allocating funds to schools, participatory planning)

Indicators of participation and decentralized service delivery may proxy the “quality” valued by users; may also reflect a special value users may attach to being involved in the management of the facilities

Key questions: why these indicators matter for satisfaction, what explains the variations between rich and poor areas, and what that implies for the priorities of a government ?

Implications for education and health services in Indonesia--3

Thank you

Simple heuristic model of household satisfaction OLS of S on key characteristics of households and communities,

using the full household sample, separately for health and education Household variables include education, gender, age, ethnicity,

religion; district level variables such as urban/rural and rich/poor districts

Restricted to characteristics that service providers have little control over; variables related to objective quality of services omitted

Certain hhold characteristics have statistically significant correlation with reported satisfaction E.g. gender of hhold head, religion, ethnicity, education level,

household composition Rural respondents less likely to be satisfied with public facilities

they frequent Information on governance seems to matter

(i) households less likely to be satisfied when they know about complaints, but more when there was a follow-up in response; (ii) information about corruption/bribery in education strongly associated with lower satisfaction, but not so for health services

But these types of information highly likely to be endogenous 37

Sample Selection Strategy 1 Kabupaten/Kota (district)

Sampling frame: 434 districts (–) following 26 districts =408 districts

(21 (Aceh) – 2 (Nias) – 3 (pre-test locations: Kabu- Maros (South Sulawesi), Kabu- Pontianak (West Kalimantan), and Kabu- Muaro Jambi (Jambi))

Procedure1. 89 districts are selected randomly using SRS2. Rest of the districts are top-offs from ILGR districts,

USDRP, SPADA, and SPADA-Justice (WBOJ); 3. And 2 training districts : Kota Salatiga and Kabu- Boyolali

(Central Java) Total districts selected = 135 (with 1 dropped)

Sample selection strategy 2

Kecamatan (sub-district)Sampling frame: all kecamatans within respected districts (data

source: PODES 2005) Procedure1. Random districts: 3 sub-districts using PPS with # HH as weight2. Total sub-districts selected = 134 * 3 = 402 Desa/Kelurahan (village)Sampling frame: all villages within respected kecamatans with 50+

HHs; 2. 16 households within the village Procedure: 1. Pick 2 villages using PPS Total villages selected = 402 * 2 = 804

Sample selection strategy 3

Dusun (hamlet) Sampling frame: all hamlets within respected village Procedure: sort hamlet names alphabetically; select 2

hamlets from the sorted list: (a) the first and (b) the middle of the list

Total hamlets selected = 804 * 2 = 1,608   Household Get the most recent household list from head of hamlet.

Randomly select 8 households from the list. Total household = 1,608 * 8 = 12,864

Note: We use 88 randomly selected districts

Sampling criteria: Schools & Puskesmases

Sub District (3)

District Education Unit=1

SD-1: Junior High (1) Puskesmas (2)

SD-2: Junior High (1) Puskesmas (2)

SD-3: Junior High (1) Puskesmas (2)

Village(6)

V-1: Primary (1) Pvt.Paramed (3)Pvt. Doctor (1)

V-2: Primary (1) Pvt. Paramedic (3)Pvt. Doctor (1)

V-3: Primary (1) Pvt. Paramed (3) Pvt. Doctor (1)

V-4: Primary (1) Pvt.Paramed (3) Pvt. Doctor (1)

V-5: Primary (1) Pvt. Paramedic (3)Pvt. Doctor (1)

V-6: Primary (1)Pvt. Paramed (3)Pvt. Doctor (1)

Selection of facility respondentsType of respondents Total respondents per district

Head of Education facilities 1

Head of Health facilities 1

Primary school

Principals 6

Secondary data 6

School Committees 6

Junior High school

Principals 3

Secondary data 3

School Committees 3

Health

Head of Puskesmas 6

Puskesmas secondary data 6