PNPM-Urban Process Evaluation :

Key Findings and Considerations

November 2011

For discussion only

Goals for process evaluation

For discussion only

Program objectives for PNPM-Urban

Source: Review of World Bank PNPM-Urban III Project Action Document

For discussion only

Evaluation focused on six themes emerging from the

results frameworkTo what extent is CDD realized in planning, decisionmaking and implementation? Are socialization activities sufficiently inclusive? Is PNPM-Urban meeting the needs of the poor?

Assessing the role of CDD

Assessing the role of CDD

How does PNPM-Urban link with local government and with sectoral programs? Is there much collaboration / duplication?

Links with local government

Links with local government

How effective are the current facilitators and do they adequately meet the needs of the communities? What characteristics contribute to strong facilitators versus weak?

Capacity building and effectiveness of the

facilitators

Capacity building and effectiveness of the

facilitators Overall how is the quality of the infrastructure investments? What factors contribute to high- and low-quality works? How cost effective are infrastructure investments?

Quality of infrastructure

Quality of infrastructure

How effective are the current governance structure/ control mechanisms that are in place? Are they adequate or could they be improved? If so, how?

Control mechanisms/governan

ce

Control mechanisms/governan

ce Do communities express a desire for larger grants?.How do experiences and outcomes vary under PAPG/ND and what are the differential impacts?

Adequacy of block grants and insights

from PAPG/ND

Adequacy of block grants and insights

from PAPG/ND

For discussion only

Methodology

Methodology

Desk and Literature Review

Desk and Literature Review

Subject Matter Expert

Interviews

Subject Matter Expert

Interviews

Formative ResearchFormative ResearchSite visitsSite visits

Review of PNPM-

Urban MIS Data

Review of PNPM-

Urban MIS Data

Analysis draws on multiple sources of information

For discussion only

Focus of Data Gathering: Site Visits

• Goal was to design and implement efficient data collection effort under time and resource constraints, while capturing

• sufficient number and variation in sites, including 3 ND

• subjective perceptions of multiple, different stakeholders

• objective assessments of PNPM-Urban project output quality

• Final fieldwork design reflects important trade-offs

• 16 field sites to be covered in 5-6 week timeline

• priority focus on infrastructure

• simultaneous dispatch of four field teams led by RAND and SurveyMeter, with four engineering teams after heavy pre-field preparation + common initial training and 2 site visits

• Fieldwork commenced May 28 through June 25, 2011.

For discussion only

Site Selection • Randomly selected from IFLS communities (80% of

national population) with projects in the MIS database > 2 years old

• Regional stratification by urban population/PNPM-Urban funds, wealth stratification by IFLS community wealth index

For discussion only

360-degree approach to data collection

• District Program manager• Korkot / Askot

PNPM-Urban ManagementPNPM-Urban Management

• Bappeda and district MPW leadership

• Kota OfficialsGovernment Government

• Facilitators• Consultants/plannersLocal PNPM-UrbanLocal PNPM-Urban

• Camat, Lurah, RT/RW leaders• NGO representatives

Local Government (and other civil

society)

Local Government (and other civil

society)

• BKM leader• Infrastructure KSM members

PNPM-Urban community leaders

PNPM-Urban community leaders

• Poor households in general• Infrastructure project beneficiariesCommunity membersCommunity members

For discussion only

360-degree approach to data collection

• Key informant interviewsPNPM-Urban

management and government

PNPM-Urban management and

government

Local PNPM-Urban, government and other civil society

Local PNPM-Urban, government and other civil society

• Key informant interviews• Rapid survey about BKM and KSM

PNPM-Urban community leaders

PNPM-Urban community leaders

• Rapid survey of poor households • Male and female focus groups +

interviews• General poverty• PNPM-Urban

• Infrastructure beneficiary interviews

Community membersCommunity members

• Infrastructure assessments• MIS data assessment Other Other

• Key informant interviews

For discussion only

Key Findings 1: Community socialization and participation

• What is working well?

• Knowledge of processes among local leaders and individuals within PNPM is good

• Community institutions like the BKM largely function as intended

• High level of recognition of PNPM-Urban projects among community (although not always PNPM-Urban)

• High level of expressed satisfaction with project outputs among community and beneficiaries

• High levels of community contribution to implementation, although role of paid and unpaid labor appears to vary

For discussion only

Key Findings 1: Community socialization and participation• What are areas for improvement or consideration ?

• Little process awareness among community members: rarely critical but tend to approve /ratify rather than propose project. Decision-making tends to proceed by delegation.

• Stated preferences = demand for social and economic projects: respondents identified loans/training as most important needs

• Targeting the poor via infrastructure projects challenging, because of geographical spread

• Voluntary administrative burden is taxing for BKM and KSM, may affect selection of leaders

• Women’s participation in decisionmaking is relatively low, potentially due to exclusionary norms and protocols

For discussion only

Key Findings 2: Links with Local Government

• What is working well?

• Involvement by local government varies across sites, from little substantive involvement (e.g., only signing off on projects) to substantial local government support (e.g., participating in decision making and oversight)

• Even in low-interaction sites, Musrenbang process is used effectively to convey information about efforts: there appears to be little or no duplication of activities or priorities.

• What are areas for improvement or consideration?

• Few respondents mentioned training for local governments

• Observe coordination, but no widespread collaboration or joint planning; some evidence of competitive feeling

• Support for more integration is not universal.

For discussion only

Key Findings 3: Facilitators• What is working well?

• Consensus view across respondents is that facilitators are playing a significant part in all stages of PNPM-Urban, and that facilitators’ contributions improve the quality of projects

• Rotation of facilitators limits cronyism and increases the exposure of skilled individuals

• What are areas for improvement or consideration?

• Facilitators universally appear overworked and overextended. The existing administrative burden on the facilitators for compliance purposes is substantial

• Most cannot contribute much to encouraging participation at the sub-Kelurahan level.

• Timing of rotations is disruptive to the project cycle

For discussion only

% Infrastructure Spending

For discussion only

Key Findings 4: Infrastructure Assessment

•What is working well?

• Assessments of PNPM-Urban project quality were high

• Subjective assessments compare PNPM-Urban project quality favorably to government projects; respondents attribute higher quality to participatory approach and avoidance of government management  (including some government officials)

• Community reports being more likely to maintain PNPM-Urban projects (although too early to objectively determine)

• Average levels of quality inspection reported in MIS tally with average independent site assessments, suggesting standards are adequate by local professional standards

For discussion only

Key Findings 4: Infrastructure Assessment

•What are areas for improvement or consideration?

• Most beneficiaries were not consulted or given opportunities to provide feedback or correction

• Although generally quality is perceived to be high, some respondents suggest voluntary contributions may lead to the use of lower quality materials

• Use of paid versus volunteer labor varies across sites, with some explanations e.g responding to the technical demands of the project, but reflects potential trade-off

• Actual projects can depart from plan and MIS reports (also related to control) without the appearance of reliable systematic protocols for reporting(e.g. cases of splitting projects, or bundling types of expenses together)

For discussion only

Key Findings 5: Control Mechanisms and Governance –

Funds Management• What is working well?

• Very few reports of complaints related to misuse of funds

• Auditing is generally completed on schedule and uncovers instances of inadequate project performance, selection and implementation at an early stage by facilitators.

• Most sites reported relying on the combination of multiple layers of audit, social norms and community awareness of projects to ensure accountability

• What are areas for improvement or consideration?

• Difficult to assess extent of reporting bias

• Previously noted: heavy burden of compliance, issues of control (versus governance)

For discussion only

Key Findings 5: Control Mechanisms and Governance –

MIS• What is working well?

• Relative to its initial goals, MIS performs well in making public a vast quantity of timely and usable data for tracking purposes

• Ability to analyze MIS has increased significantly thanks to motivated and highly-familiarized PNPM-Urban technical staff

• What are areas for improvement or consideration?

• Significant inherent challenges including harmonization of file formats across funders and cycles, training and motivation of data entry personnel at the community level

• Substantial room to realize full potential of MIS data for process evaluation: data quality and validation issues remain, many key outcomes e.g. poverty, population are only self-reported, users need significant guidance

For discussion only

Key Findings 6: Block Grant Size and ND Model

• What is working well?

• Relatively few sites reported size was inadequate for needs (however, this may be biased by local expectations)

• Large % of ND site funding is spent on planning and marketing, suggesting recognition of problems of participation

• What are areas for improvement or consideration?

• Larger grants may be used to undertake more small projects in different areas, not realizing economies of scale/scope. (One ND site even desired more flexibility to do so) Concerns expressed that larger projects are harder to manage and difficult to distribute benefits equitably.

• ND site planning and marketing efforts not viewed as useful (although sample is small)

For discussion only

Limitations of this study (planned and unplanned)

• Reiterate that focus is largely on infrastructure

• Short timeline precluded more intensive focus group discussions and community member interviews on specific issues that arose during evaluation e.g. expectations of community regarding delegated versus direct participation

• Resource and time constraints prevented larger sample of sites, including PAPG sites, potentially more meaningful sub-group analysis

• Cost-effectiveness analysis and analysis of sectoral program links could not be completed within the scope of the project

• MIS data provided useful insights into program at a high level but precluded robust regression and subgroup analysis; most analysis draws on subjective perceptions, should be interpreted as such.

For discussion only

Implications for Design and Experimentation

Implement universal invitations to meetings particularly for women. Consider alternative methods of eliciting preferences of poor households

Assessing the role of CDD

Assessing the role of CDD

Continue to focus on training and capacity building, emphasizing collaboration rather than competition and potentially consider more structured interactions

Links with local government

Links with local government

Build more facilitator capacity through recruitment and/or training that includes interpersonal skills building, reconsider rotation schedules

Capacity building and effectiveness of the

facilitators

Capacity building and effectiveness of the

facilitators Allow more direct channels of feedback for those likely to be affected. Improve balance between local responsiveness and control. Consider tradeoffs between community and hired labor.

Quality of infrastructure

Quality of infrastructure

Maintain multiple checks and balances with audit system incorporating local participation; increase attention to potential of MIS from bottom data entry to high-level database structure and user interafce

Control mechanisms/governan

ce

Control mechanisms/governan

ce Consider individual needs of communities and continue to monitor development of ND projects as they mature

Adequacy of block grants and insights

from PAPG/ND

Adequacy of block grants and insights

from PAPG/ND