.

FINAL COUNTRY REPORT:

September 2014Professor Don Matheson

EVALUATION OF ACCELERATING THE IMPLEMENTATION OF THE INVESTMENT CASE FOR MATERNAL, NEWBORN AND CHILD HEALTH IN ASIA AND THE PACIFIC PROGRAMME

INDONESIA

September 2014Professor Don Matheson

FINAL COUNTRY REPORT:

EVALUATION OF ACCELERATING THE IMPLEMENTATION OF THE INVESTMENT CASE FOR MATERNAL, NEWBORN AND CHILD HEALTH IN ASIA AND THE PACIFIC PROGRAMME

INDONESIA

CONTENTS

Abbreviations 6

Explanatory Note 7

Introduction 7

The Baseline Report findings 7

Programme Logic for EBP 9

Quantitative analysis for Indonesia 11

Qualitative evaluation for Indonesia 11

Responses to the evaluation questions 11

Evaluate the impact of the IC on how planning is undertaken, programmes are delivered, policies are

crafted and the processes by which budgets are decided 11

Evaluate the impact of the IC on government processes, including the political and subsequent budgetary 13

priority given to Maternal, Newborn, and Child Health and Nutrition

Document the current use being made of the IC in Indonesia

Additional impacts of the activity in Indonesia 19

Discussion 19

Conclusion 21

Recommendations 21

Acknowledgements 25

Interviewees 25

6

ABBREVIATIONS

AIPD Australia Indonesia Partnership for Decentralisation AusAID Australian Agency for International DevelopmentBAPPEDA Badan Perencana Pembangunan Daerah (Regional body for planning and development)BAPPENAS National Development Planning AgencyBOK Bantuan Operasional Kesehatan, which funds operational costs for PuskesmasDAK Dana Alokasi Khusus (Special Allocation Fund)DAU Dana Alokasi Umum (General Allocation Fund)DFAT Australian Government Department of Foreign Affairs and Trade DTPS District Team Problem Solving DPT3 Diphtheria, Pertussis, Tetanus Vaccine third vaccine DP Development PartnerEBP Evidence Based PlanningGAVI GAVI AllianceGFATM Global Fund to Fight AIDS, Tuberculosis and MalariaHSS Health System Strengthening IC Investment Case (for Maternal, Newborn and Child Health)iNGO International Non-Government OrganisationIP Implementation PartnerMBB Marginal Budgeting for Bottlenecks MCH Maternal and Child HealthMDG Millennium Development GoalOTSUS Otonomi Khusus (Special Autonomy)PAD Pendapatan Asli Daerah (revenue of regional governments)PONED Basic Emergency Obstetric and Neonatal CarePONEK Comprehensive Emergency Obstetric and Neonatal CarePPP Purchasing Power ParityRMNCH&N Reproductive Maternal, Newborn, and Child Health and NutritionUNICEF United Nations Children’s FundUKP4 Unit Kerja Presiden Bidang Pengawasan dan Pengendalian Pembangunan (the Presidential Working Unit for Supervision and Management of Development)UQc The University of Queensland Consortium WHO World Health Organization

7

In Indonesia, the ‘Investment Case’ is referred to as an ‘Evidence-Based Planning’ approach (EBP).

The structures/administrative organisations referred to in the report primarily refer to the Province of Papua.

EXPLANATORY NOTE

INTRODUCTION

THE BASELINE REPORT FINDINGS

Papua is composed of districts (also known as regencies). The political leader of the district is the Bupati. The Bupati reports to central government through the Ministry of Home Affairs, not the province. The BAPPEDA and BAPPENAS refer to the government’s provincial and national development and planning agency, respectively.

Community health centres are called Puskesmas.

This document reports on the evaluation of a UNICEF programme: “Accelerating the Implementation of the Investment Case for Maternal, Newborn and Child Health” in Indonesia.

The report follows on from the Baseline Report for Indonesia and should be read in conjunction with the Synthesis Report for the four countries: Bangladesh, Indonesia, Nepal, and the Philippines. The Synthesis Report includes a description of the methodology and the overall findings of this evaluation.

Indonesia appeared to be on track to achieve the under-five child mortality indicator Millennium Development Goal (MDG) 4. However, recent survey results suggest progress seems to have stalled and achievement of MDG 4 is now at risk.

There are disparities in the under-five mortality rate related to area of residence, socio-economic status, and mother’s education level. In 2012 the rate for the poorest 20 per cent, for example, was 70 deaths per 1,000 live births while for the wealthiest 20 per cent the rate was 23 deaths per 1,000 live births. Death rates are also dissimilar between urban and rural populations.

From a nutrition perspective, the number of underweight children has reduced by one-third; however, the reduction in stunting has been slow.

Although there has been considerable improvement over the last decade, it is likely that Indonesia will not reach the MDG 5 target. Maternal deaths are unequally distributed, with disparities in service provision that are greatest for the poor, those with low education, and those in rural areas. The leading cause of maternal deaths is haemorrhage, and to address this requires improvement in emergency obstetric care and the associated institutions (birthing facilities, district hospitals), attendance at birth by skilled health workers and improved referral systems. Quality of care is a major issue. One programme, Family Planning, stands out in its achievement of more equitable coverage.

The Indonesian government has a level of health expenditure of USD150 PPP per capita which is just above what is thought to be the minimum required to deliver on the MDGs. It has a health workforce of 23 (doctors, nurses, midwives) per 10,000 people, which is also assessed as the minimum required, although the dispersed island populations require a higher density than more tightly concentrated populations.

The funding problem is not so much the amount of funding available, as the complex mechanisms by which funds move within the system, from the centre to the periphery, as well as within provinces and districts. Unspent budgets, delays in budget fund release, weak accountability, and political overlay of the policy and prioritisation process have a serious impact at each step of the process. These difficulties are well recognised and apply to all sectors, not just the health sector. The major challenge is improving the allocation, flow, prioritisation of and accountability for resources at the periphery, a challenge that largely sits outside of the health sector.

The UNICEF country office labelled the intervention an Evidence Based Planning (EBP) approach. At the national level it planned to explore in detail one of the funding channels, the Special Allocation Fund (DAK) to help the government resolve the bottlenecks that a lack of flow of funds creates. In addition, the intention was to explore how this approach can be used to augment currently accepted district planning tools. Activities at the sub-national level focused on the Province of Papua, where the intention was to build capacity for conducting evidence-based planning activities at both the district and the provincial level.

8

9

PROGRAMME LOGIC FOR EBPDuring the first phase of the evaluation, the evaluator worked with the UNICEF country office to establish the programme logic for their activities.

19% 26%

14%

16%

BP Expenditure profile 2012

and 2013 Indonesia

Figure 1: EBP Expenditure profile, 2012 and 2013 - Indonesia

Research Institute

UNICEF Country OfficeGol National (BAPPENAS)

Gol Privince (BAPPEDA)

Between Phase I and II of the evaluation, the assumptions and context remained largely the same. However, the project’s resourcing in 2012 to 2013 was $1.4m,1 double that anticipated at Phase I of the project.

Figure 1 shows how the resources were spent in terms of the different institutions/ levels of government involved.

The commitment to explore the relationship between the EBP activities and the government’s longstanding MCH district planning process DTPS (District Team Problem Solving) was carried out by supporting the government to review the DTPS. This review has been completed and a new DTPS module is currently being considered by the Minister of Health. It contains a bottleneck framework analysis derived from the EBP approach.

The project had successfully developed a provincial team embedded in the provincial office with a focus on training districts in the use of EBP to prepare their plans and budgets. This team consisted of provincial employees, the UNICEF Papua field office and researchers from Jayapura Research Institute Cenderawasih University, with support from a national research institute, University Gadja Mada, and from the University of Queensland. The team’s activities included support for the identification of evidence, analysis and interpretation of district, provincial and nation level data to develop the equity gap analysis and bottleneck analysis relevant to each district.

1 EBP expenditure report supplied by UNICEF country office and University of Queensland. These figures were not independently audited.

10

The team conducted workshops at provincial and district level, and continued to support district level officials in analysis and document preparation outside of the formal workshops.

The district level work consisted of an original focus on three districts: Boven Digoel, Yapen and Jayawijaya. The EBP process began in these districts in 2011/12, so they have been the focus of this evaluation. The EBP process has now been extended to additional districts: Nabire, Sarmi, Mappi, Biak, Jayapura, Paniai and Supiori.

The synchronisation of the Indonesian government’s approach to district level planning of MCH is a work in progress as noted above.

The provincial team supporting district planning has been established in the provincial BAPPEDA office. Team members reported that the team would continue to operate even if UNICEF support ceased.2

Figure 2: Papua Districts (Regencies)

Figure 3: Changes in MCH funding in three Papua Districts – 2012-2013

Evidence from the districts show an 89% increase in MCH allocated funding occurring in the three original districts between 2012 and 2013.

In 2014 Otonomi Khusus (Special Autonomy) (OTSUS) was decentralised and the split between the province and the district changed from 60:40 to 20:80 in favour of the district.

In Yapen, despite the increase in funds from 70% (2012) to 90% (2013), the district was also able to improve MCH budget execution.3

District officials did not attribute the increase of MCH allocations to EBP activities, but rather to increased government funding.

The work undertaken with DAK evolved since the program logic was developed. The focus has been on DAK and the Bantuan Operasional Kesehatan (BOK, which funds operational costs for Puskesmas), as DAK is a matching grant where funds are spent only when there is a matching local contribution. There is now a proposal before government to incorporate DAK into BOK. There is also a broader initiative looking at the intergovernmental fiscal transfers in health more generally. This is being done in collaboration with DFAT in different parts of Indonesia.

2 Interviews with Provincial team members and officials. 3 Personal communication, Laxmi Zahara.

3,000

2012 2013

2,500

2,000

1,500

1,000

500

0Yapen Jayawijaya Boven Digoel

Change in MCH funding in three Papuan Districts 2012-2013

MCH funding in million IDR

11

4 DTPS was more effective in some regions (East Nusa Tenggara and Central Java) than others. (Director of Maternal Health)

QUANTITATIVE ANALYSIS FOR INDONESIA

QUALITATIVE EVALUATION FOR INDONESIA

RESPONSES TO THE EVALUATION QUESTIONS

Government administrative data on health service utilisation was sought for the Papuan provinces, as well as information held at the district level. Implementation

partners provided the financial information, and the district level financial information was sourced directly from four districts: Yapen, Boven Digoel, Jayawijaya and Tolikara.

Thirty-two people were interviewed in Indonesia over the two phases of the evaluation. They were from the National Ministry of Health, the Provincial Administration in Papua, District Health Offices (Yapen, Boven Digoel

and Jayawijaya), UNICEF staff (Jakarta and Jayapura), Development Partners in Jakarta, political representatives (Bupati and MP) in Yapen, and front line health workers including specialists and nurses in Yapen.

Evaluate the impact of the IC on how planning is undertaken, programmes are delivered, policies are crafted and the processes by which budgets are decided

(A) What was the process of planning and budgeting prior to the introduction of the IC?

Prior to the use of the IC in the three Papuan districts, data and information were not strongly considered in the local planning process, and in one district there was no awareness of the existing MCH planning tool:

“There were no planning tools prior to the IC. I am not aware of DTPS4.” (District Health Office official)

“There is very poor data. Budgets are just a duplicate of what was done in the past.” (Implementation partner)

A more formal planning process does occur at the central and provincial level. The President has formed a working unit for supervision and management of development (UKP4) to support the President at the national level. The committee meets the Ministry of Health every three months to assess MDG targets.

The provinces’ planning and budgeting process involves a technical meeting where data is presented and analysed, and there is a meeting between ministries, regional government and other parties to agree allocations. The process for ensuring the quality of data involves its review by a knowledgeable technical specialist:

“How is quality maintained? The profile results (from a regency) are first shown to provincial specialists in the particular topic to check that the results are consistent with their knowledge of the sector.” (Provincial Information Officer)

Coordination between different parts of the health sector is made difficult due to separate funding and accountability streams (see Baseline Report for a description of the funding streams). There is also a lack of coordination at the local level:

“There is lack of coordination between Dinkes (District Health Office) and the hospital. This is because the hospital is directly responsible to the Bupati.” (District health officer)

Prior to the IC there was not a planning and budgeting process at the district level that used data. There were developed processes at the provincial and national level that considered both evidence and political considerations in decision making and these are described in the Baseline Report. The direct oversight of the President’s Office for the MDGs has increased the focus on RMNCH&N.

12

(B) What was the understanding of the key challenges/ deficiencies of existing plans and budgets, especially in terms of addressing the needs of maternal, newborn and child health (MCH) and of the most deprived?

Senior government officials involved in MCH viewed district level planning processes as important but did not feel progress would be made by the introduction of another planning tool:

“The Ministry of Health finds that there is nothing wrong with the DTPS. We don’t need another tool.” (National MCH official)

In terms of the system’s key challenges and deficiencies, there were a number of different views expressed. The lack of adequate data has previously been mentioned, but most respondents emphasised the HR issues as being most important, followed by budget concerns:

“The challenges are lack of coordination between Dinkes and Hospital, lack of training and accreditation for physicians and midwives.” (District clinical specialists)

“Limited health personnel, limited housing for health personnel.” (Sub-national BAPPEDA Planning Officer)

“Not enough budget, problems with budget flow and delays.” (Regional MP)

The interviewees all had an understanding of who and where the main deprived populations were and saw the challenges in addressing these mostly in terms of overcoming geographical isolation, the difficulty in attracting health personnel to those areas, the formation of new local authorities and local political instability:

“Many of the most deprived districts (in Papua) are new. We need the same approach as in other districts. There is no track record of institution building.”

Data issues were also noted in terms of deprived populations:

“Inconsistent reporting is a major barrier to reporting on equity.” (Provincial Health Information officer)

(C) How was the IC introduced, process used, aspects of IC used, resources, timing and time taken, organisations and people involved?

The IC in Papua was developed by establishing a technical unit within the provincial government’s system-wide planning agency, BAPPEDA, to support the use of the IC approach at the district level. This approach built on the past experience of the use of the IC in Indonesia where the province was not involved, and the activities were subsequently not sustained.

The provincial-level unit and the district were supported by three university-based teams, at provincial, national and international levels: Cenderawasih University in Jayapura, University Gadja Mada in Yogyakarta, and the University of Queensland in Brisbane. The financing of this support is described in the resources section.

The process used was considerably simplified from that in the original concept of the IC. Originally there was use of a planning and modelling tool, the ‘Matrix’, which required input of extensive, detailed costing and service data, much of which was not available locally, and therefore using the tool was heavily reliant on outside assistance. This tool was modified, and more emphasis placed on the identification and development of local data sources. Elements of the approach, such as the use of tracers and bottleneck analysis remained as the core of the IC activities. There was no fiscal space analysis, estimates of coverage and lives saved, or scenarios undertaken.

The situation of the team within a multi-sector provincial government institution (BAPPEDA) enabled the approach to embrace sectors other than the health sector. The planning process at the district level involved other sectors as a consequence.

(D) How effective and efficient was the IC process?

The IC activities in Papua were effective in creating space for a district level multi- stakeholder planning process that involved the use and examination of local information. The impacts observed are discussed in more detail in the section below.

The EBP process also supported improved coordination within the district health office:

“Since the EBP there is intense involvement of the four DHO divisions; HR, CDC, infrastructure, MCH. Now we are all developing the 5 year plan, all four are involved.” (District health Planner)

13

Priorities outside of RMNCH&N were not directly included in the approach taken, as the priorities were framed around RMNCH&N issues, and this was shaped in the process by the use of tracers directly focusing on MCH as a priority. However given the above comments, its impact has flowed through to other planning activities.

District level participants did not report the process as cumbersome although it did involve increased workload and demanded more of their time.

“Cumbersome? No, it does not take too much time. What we need is time to understand the concept.” (District health planner)

In the approach taken, the bulk of the activity related to sourcing, cleaning and understanding the data. This was undertaken by the three research bodies. The EBP process has created the need for a parallel information system:

“The forms for the EBP are very specific and in depth. So there are two data collection systems, the district profile and the EBP.” (Implementation partner)

The effort involved from all parties was considerably greater than the previous budgeting exercise which was a simple ‘copy and paste’ from the previous year.

(E) What were the ‘products’ produced by the IC process? Who received them, and when?

The products of the IC activities at the district level were the districts reports, with bottleneck analysis, arising from the EBP process. These reports were then used by district planners in budget preparation and for advocacy, such as advocacy with the local political leader, the Bupati, on how they allocated resources.

They were also used at the provincial level to influence provincial planning and budgeting processes.

The participation of other sectors in one district (Boven Digoel) was a product in itself, as those sectors mobilised resources themselves to address the issues identified in the EBP workshop.

Evaluate the impact of the IC on government processes, including the political and subsequent budgetary priority given to Maternal, Newborn, and Child Health and Nutrition

(A) Did the IC impact on governments planning, budgeting and monitoring processes?

The biggest impact observed from the use of IC in Papua relates to the use of local data and information. A consistent view expressed by those directly involved was that the process was a revelation to them in understanding how data can be used and its meaning. Prior to the IC activities, data collection was a routine activity conducted throughout the district’s facilities, but there was no understanding or mechanism for its actual use at the local level.

“Before EBP we did not understand [data]. EBP emphasises the importance of data.” (District health planner)

“EBP has been useful, helped us use data, identify what the problems are, and use strategies to address them.” (District Health Office official)

“One advantage [of the IC] is the importance of data. This means when we go to the Bupati there is no need to debate as the data is there.” (District Health Office official)

There were also wider impacts noted from the increased use of data by the district political leadership:

“The Bupati is also keen on EBP. He asks ‘Where is the data?’ now when an issue comes up.” (District level health planner)

A key provincial official noted the improvement in data from two of the three EBP districts, but also made the general observation about iNGO influence on use of data:

“Do EBP regencies have better data performance? Data reporting is poor for Jayapura, good for Boven Digoel and Yapen. Regencies with active iNGO involvement (e.g. Global Fund) have better data reporting than those without.” (Provincial health data officer)

The improved planning process at the district level was noted at the provincial level but did not impact directly on the provincial planning process:

14

“EBP is a regency level planning tool. Its impact is the quality of the analysis that is fed into the provincial planning process is much greater from EBP regencies. Prior to this, budgets were a copy and paste exercise, now serious thought goes into it.”

The EBP did not impact directly on the provincial or the national planning process:

“The provinces’ planning process remains the same.” (Provincial Health Office planner)

(B) What particular aspects of the IC made the most impact on planning and budgets, and in what circumstances: the bottleneck and equity analysis? The fiscal space analysis? The estimates of coverage and lives saved? The scenarios? The relationships with IPs?

There is strong evidence for the biggest impact coming from the use of and engagement with data and the focus on a planning process at the district level, as noted in the responses above. There was no fiscal space analysis, estimates of coverage and lives saved, or scenarios undertaken.

(C) Has the IC approach helped to frame and better articulate the way RMNCH&N is characterised, and the way decisions are made?

The use of the EBP has enabled district level officials to engage in an evidence-based planning process where previously none existed. It also had an impact on the quality of the information that decision makers

then used at both the regency and the provincial level. However, it was not the only process active in the RMNCH&N area at the time.

As one provincial observer noted:

“It’s not the lack of process and mechanisms that is the problem, but too many.” (Provincial iNGO)

UNICEF itself is promoting similar but differently branded approaches in Papua, such as Integrated Micro Planning at the Puskesmas level, and ASIA (Analisis Situasi Ibu dan Anak), a mother and child situation analysis through the Ministry of Home Affairs.5 Other development partners such as DFAT through AIPD and USAID through Kinerja Papua6 are engaged with some of the same districts with related programmes. There is, in effect, a mini market of district level initiatives:

“This situation (multiple DP led district initiatives) makes the districts confused. So BAPPEDA will let districts decide which approach is easiest for them to be implemented.” (Provincial level, senior BAPPEDA official)

Equity considerations

The focus on equity can be looked at from two perspectives; were the EBP regencies those with the highest needs, and how were equity issues dealt with within these?

5 This is in its early stages in Papua. 6 http://www.kinerja.or.id/kinerja_expands_into_papua.asp?lang=en 7 Presentation of the research results of The Trees Health Foundation 2013. www.litbang.depkes.go.id

Figure 4: Service delivery (SBA) coverage across Indonesian provinces7

The proportion of deliveries by trained health workers. Indonesia 2010-2013

100.0

2010 2013

Papu

a

Mal

uku

Mal

ut

Sul

bar

NTT

P

abar

Kal

teng

Sul

tent

Kal

bar

Sul

tra

Sul

sel

Sul

ut

Jaba

r

Ban

ten

Kal

sen

Ria

u

IND

ON

ES

IA

Jam

bi

Sum

sel

Lam

pung

Kal

tim

Bab

el

Ace

h

Gor

onta

lo

NTB

Sum

bar

Sum

ut

Jatim

Ben

gkut

u

Kep

Ria

u

Jate

ng DK

I

Bal

i

DIY

80.0

60.0

40.0

20.0

0.0

15

8 Programme Evaluation and Strengthening of Health Care Systems in Papua Province Oleh: drg. Josef Rinta R., M.Kes. MH.Kadinkes Provinsi PAPUA, Jayapura, 20 November 2012

Figure 5: SBA attendance Papua Districts 2012 with EBP Districts marked

Figure 4 shows that Papua has the lowest level of coverage for birth attendance by a trained health worker. From this and other data, it is clear Papua is the highest need province in Indonesia.

Also of interest is the overall improvement seen in this indicator between 2010 and 2013 for most provinces. That improvement is minimal among three of the four provinces with the lowest coverage (Papua, Maluku, Sulbar).

Figures 58 and 6 show that for these indicators, the three initial districts chosen for the EBP activities were not the highest-need districts in the province.

Figure 5 shows the coverage for SBA to be above the average for Papua.

Figure 5 also shows there is a group of 11 districts with coverage less than 11 per cent for this indicator.

Slilled birth attendant coverage in Papua 2012: IC Districts identifild

16

Figure 6 shows the coverage for Antenatal Care, with the three districts preforming in the top third for the province.

The initial selection of districts involved consideration to ensure adequate capacity to maximize the chances of success in order to demonstrate the proof of concept of this approach. Districts that have been recruited since the evaluation began are spread more evenly across the spectrum of need.

Comparison First (K1) Antenatal Visit and Fourth Antenatal Visit (K4) Papua Regencices September 2012

Yapen: Changes in reporting and coverage for SBA

Figure 6: Antenatal Visits, Papua Districts, 2012

Figure 7: SBA coverage, Yapen, 2010 to 2012

The process conducted did have a focus on equity within the district. One district, Yapen, was examined in more detail to explore the extent that geographical equity was being addressed. However, the information available is only to 2012, which is at the beginning of the EBP process and it is too early to be seeing an impact. It is positive though to see reporting increase from 9 Puskesmas in 2010 to 13 in 2012, an indication of greater attention being paid to data at the district level. District officials were asked

17

Tolikara Distric reports of 4th antenatal and delivery by a health worker.

Figure 8: Maternity care coverage in Tolikara 2009 to 2013

about the impact of the EBP on the hardest to reach sub-districts in Yapen by 2014, but felt there was little change, because of a deteriorating security situation:

“There is no improvement in the more remote districts. The access there is hampered by road blocks that have been put up by the community.” (District level official)

Other Papuan districts not included in the EBP process are also reporting improved performance. Figure 8 shows change in Tolikara over the last five years in maternal care indicators SBA and antenatal care. This information needs to be treated with caution. It was collected directly from the district which had not been reporting its statistics to the province, so the

improvement may reflect more effective reporting or more effective coverage. Tolikara is a new district (established 2002), so there are also limitations in comparisons with other districts in the province. However it does indicate that system improvement is being reported outside of the EBP districts. Tolikara has a special program for all pregnant woman where they are given nutritious food every day except on Sunday. This program involves health staff particularly midwives in puskesmas, pustu and polindes, aiming to improve nutritional status of pregnant woman and use of SBAs.

Budgetary changes for MNCH are occurring in all districts in part at least driven by increased budget allocation generally.

100

90

80

70

60

50

40

30

20

10

0

K42009 2010 2011 2012 2013

Delivery by SBA

(D) What has been the impact on RMNCH&N programmes and policies?

One of the significant findings of the EBP process in Papua is the extent of the involvement of other sectors in the process.

“EBP improved coordination with other sectors. The plan we developed was better.” (District level health official)

The value of the inter-sectoral planning was commented on positively and frequently by interviewees at all levels in Papua.

“The strength of EBP is the involvement across sectors.” (Provincial level government official)

The approach also improved coordination within the health sector itself:

“It encourages Puskesmas to do integrated activities, such as immunisation and filariasis. This had not happened before.” (District programme officer)

The EBP process did not impact on national policy in regard to RMNCH&N.

18

The national government is, however, conducting a review of its existing planning tool, the DTPS. According to officials involved in this review, the use of some elements of the EBP process, such as the bottleneck analysis, are being considered for incorporation into the revised DTPS planning process. This would then be the government’s preferred MCH planning tool and would be rolled out to all districts. So, while the EBP process has not influenced RMNCH&N policy at the national level, it has influenced the national approach to sub-national planning for MCH.

(E) Is there a difference observed between districts involved in IC activities and those that are not?

As noted above, there is a background of improvement in RMNCH&N indicators in both EBP and non EBP districts. One non-EBP district, with a higher level of recognised need than the three EBP districts, showed similar improvements in performance as seen in Figure 8.

It also had increases in MCH budgets, consistent with the increases in EBP districts, reflecting the impact of national decisions on budget allocation.

(F) Has the IC approach influenced the way decisions are made?

The EBP, as noted previously, has improved the quality of the analysis at the district level, involved other sectors and increased the use and understanding of local data.

It has also trained a group of senior officials and a local university in the approach. Views were mixed as to its likely sustainability without UNICEF funding. Some senior provincial officials believed that it would continue without UNICEF funding, while others felt that it was quite vulnerable to changes at the political level and may not be prioritised by a new administration.

There was agreement at provincial and district levels that the EBP process had brought them new skills, and these skills would persist even without a formal programme.

The data requirements of the EBP process were outside of the routine reporting mechanism and had not been incorporated into the Health Information System.

(G) Has the IC approach strengthened the power of key actors engaged in RMNCH&N at national or sub-national levels? For example, increasing their

visibility, credibility, coordination, collective action, leadership, or available resources?

The EBP has strengthened the power of planners and managers at the district level by providing an opportunity for them to lead the planning process. It has also formalised the discussion of equity issues, within the planning process.

Document the current use being made of the IC in Indonesia

(A) What is the nature and intensity of IC activities at the time of evaluation?

The EBP in Papua has now been extended to 10 districts, including two with very high need, indicating a stronger targeting of high need areas since the selection of the first three districts. The expansion of the programme now means that one third of Papua’s districts are involved in the process. The evaluator attended a training session for the districts and observed a high level of commitment and enthusiasm from those present.

(B) Do different stakeholders, including DPs, have consistently different views on the usefulness of the IC methodology in promoting improved MNCH across countries?

Development partners had a sceptical view of the EBP process. They were not well informed of recent changes to the approach, and tended to base their views on negative reports of early engagement, before the current round of EBP. They also reflected the national government’s view that new planning tools were not required.

19

ADDITIONAL IMPACTS OF THE ACTIVITY IN INDONESIA

DISCUSSION

The involvement of the local University of Cenderawasih has enabled the university to provide a fifth semester health policy and administration paper based on the EBP for 90 students per year.

The qualitative findings in this evaluation support the assessment that the EBP process in the three districts in Papua has had a significant impact. District officers engaged in the EBP process are now getting a real sense of meaning and purpose from data and information systems. Other sectors have been successfully engaged in RMNCH&N and are contributing to activities in the districts. Advocacy at the local level has become more evidence- and information-based than in the past. These changes are likely to be sustained.

The quantitative information shows some improvement in data collection, improvement in budget allocation to MCH, and improved MCH related outputs. These changes could be due to a number of causes outside of the impact of EBP activities, and the positive trends often preceded the introduction of EBP. The timing of the evaluation is such that changes in the quantitative data are likely to occur over a longer time frame than that of the evaluation.

Papua is clearly the highest need province for RMNCH&N in Indonesia. The three districts chosen were not amongst the highest need districts within Papua; however, the extension of the programme has now included some very high need districts. There was some evidence of an equity impact as seen through the quantitative data, but the qualifications above concerning the interpretation of that data should be noted.

It is apparent from discussions in Papua that the planning and budgeting needs of the highest need districts are unlikely to be met by the EBP process alone. Those districts are not reporting health information, and have few capable administrative staff. Addressing these fundamentals may be required as a prerequisite to the EBP process.

At the provincial level there has been capacity built at the local university, as well as a team formed within the provincial development and planning organisation, and together they are providing dedicated support to the districts. These activities are directly funded by UNICEF,

and their longer term sustainability will depend on the extent that their financial support is picked up by the provincial administration.

At the national level, there is little awareness of the EBP activities, and there is no visible impact on national decision making (except that described below). The focus at the national level is on the government’s move to Universal Health Coverage, which if successful will have a significant impact on RMNCH&N.

At the national level, a review is being undertaken of the pre-existing MCH planning and budgeting process for the districts, known as DTPS. This review, which is not yet complete, is giving strong consideration to incorporating part of the EBP process, bottleneck analysis, into the revised planning tool. The government has expressed a consistent view over the time of this evaluation that it does not want a new planning tool, despite being offered many tools from DPs, but supports the upgrading and dissemination of its existing approach to planning.

“It’s like donors coming with a new car, abruptly changing to the new car each time, different from the last. Government wants the one they are used to and have experience of.” (National Government Adviser).

The perceived problem at the outset of the investment case activities was that Indonesia was not giving sufficient priority and funding to RMNCH&N and not making sufficient progress on MDG 4 and 5.

The design of the original IC activities was to advocate to government to increase the priority and the budget for RMNCH&N. The approach was to garner the lessons from international evidence, largely based on the Lancet series,9 and through the mechanism of a bottleneck analysis, combine evidence, cost, and service data. This would demonstrate that considerable progress could be made in relation to the MDG 4 and 5, with relatively small investments provided they were directed to the identified bottlenecks.

9 The Lancet Child Survival series, 2003.

20

This assessment of the problem has changed profoundly in Indonesia in recent years. The government has made MDG attainment of high national importance, including playing a leading role in the post 2015 agenda. It is investing significantly in health, such that it is well in excess of the minimum level thought to be required to attain the MDGs. It has chosen its flagship pathway to address health needs, through pursuit of Universal Health Coverage.

This changed context is reflected to some extent in the EBP activities. They have been primarily an exercise at improving coordination, planning and budgeting at the district level, rather than advocacy at the national level. The improved planning and data use at the district level has improved the advocacy powers of the officials involved. However, as its primary use is as a planning tool, and not an advocacy tool, its relationship with existing planning and budgeting tools becomes of prime importance.

There are a number of district planning and budgeting tools already in operation in Indonesia, supported by donors. UNICEF itself has a number of different tools that it uses within districts, the most common of which is its micro planning tool which it uses at the Puskesmas level (which also arose from the EBP engagement). Given the government’s clear direction in relation to planning, future work in this area should include a focus on reconciling the different planning and budgeting approaches, which have more commonalities than differences. These common factors include a focus on evidence, equity, use and analysis of data including local data, and supporting thoughtful processes to precede budget setting and service plans.

A further evolution in the move from an advocacy tool to a planning tool is the potential that rests in the cyclical nature of the planning process. It is in the re-examination of the system’s performance after the BNA has been done, the budget allocated, and the implementation completed that is likely to give the best insights as to the specific barriers and enablers to progress in that specific context.

The strong engagement of sectors other than health in the planning process is a significant achievement in these districts. As confidence in the planning process grows, consideration could be given to including the district hospitals and the private sector in the process. Such a move will increase the tension between the ‘evidence-based’ approach, and a ‘political economy-based approach’ to health sector planning and budgeting. This issue is discussed further in the synthesis evaluation report.

The re–examination by the government of its preferred approach to MCH planning is a window of opportunity that the EBP has capitalised on. The emergence of a revised planning guideline nationwide will be a major advance, but past experience suggests that it is likely to make little headway unless backed with consistent, sustained technical support from donors. UNICEF, through its EBP process, has positioned itself well to support this national initiative.

At its most fundamental level, the EBP programme in Papua has established the basis for a learning system. It is less about the application of a specific tool and more about an approach that learns from experience, adapting appropriately to the local context. Support for the continued evolution of this approach is highly recommended.

21

CONCLUSION

The EBP approach in Papua has had a significant impact on the Province. It has provided training in an evidence-based approach to service review and planning to local personnel at the university, the provincial administration and the district. It has created space for district-level managers and planners to engage in a structured, local, joint planning process and effectively use locally-relevant information and evidence in the process.

Although there has been little influence on the government’s approach to national planning and budgeting, there has been an influence on the redesign of local-level planning and budgeting guidelines for RMNCH&N, with the consideration of including elements of EBP such as bottleneck analysis.

The EBP approach has positioned UNICEF and its implementation partners as key actors in health planning in Papua, in the midst of a dynamic health transition that is occurring across Indonesia.

RECOMMENDATIONS

The IC activities in Indonesia have primarily focused on one province, Papua, and built significant capacity in support of this province. A continued focus on supporting district planning and budgeting processes is recommended. To achieve this, the following actions are recommended:

• Strongly support further IC (EBP) activities in Papua, building on the experience to date and further building the skills developed in provincial and district governments and implementation partners.

• The future activities under IC should align with and support the government’s revised DTPS MCH planning process, and not be positioned in competition with it, or be differently branded.

• Develop, with other development partners and within UNICEF, a consistent approach to district level planning and budgeting, utilising the strengths of the different approaches currently in use.

• Building on the experience of EBP, trial different approaches to addressing equity that would enable participation of the ten most deprived districts in Papua.

• Continue to explore the issue of how to involve the hospitals, private sector, and NGOs in the district planning processes.

• Develop a mechanism for moving from an RMNCH&N focus to a health system-wide focus of district-level planning and budgeting activities, building on the EBP experience, and inclusive of a mechanism for priority setting.

• Continue to monitor the performance of EBP districts over the next five years.

• Develop a consistent approach to IC planning reviews, following quality improvement principles, supporting a culture of long term, evidence informed cycles of planning, budgeting, implementation and review.

22

Recommendations Actioning agency/ body

Rationale and risks

Continued support for IC activities in PapuaStrongly support further EBP activities in Papua, building on the experience to date and further building the skills developed in provincial and district governments and implementation partners.

UNICEF, IPs, DFAT RationaleThe IC activities in Papua have built a critical mass of expertise in the province. Continued support and development of the local capacity, in both the government and the academic sector is required to make these initiatives sustainable. The tiered approach (Global/ National/ Provincial) with implementation partners has been effective in establishing more rigorous planning processes at the district level. Over the next three years there needs to be a managed transition of responsibility to local IPs, both national and provincial.

RisksA Papua-centric approach limits the likely influence that the activities will have on national planning and budgeting processes.

Alignment with GoI planning and budgeting processesThe future activities under EBP should align with and support the government’s revised DTPS MCH planning process, and not be positioned in competition with it, or be differently branded.

UNICEF, IPs RationaleThe government of Indonesia has strongly signalled it is not interested in competing district planning processes. The revised DTPS / MCH planning process, which in all likelihood will have a BNA element, is an ideal opportunity for the IC activities to be aligned and in support of the government-led process and not in competition with it. This will require supporting GoI branding and approach.

RisksThe government’s approach may omit some of the elements of the IC approach. However, the government is very supportive of enhancement of its approach – it does just does not want a competing planning and budgeting system.

Alignment of UNICEF and DPs planning and budgeting approaches

Develop with other development partners and within UNICEF a consistent approach to district level planning and budgeting, utilising the strengths of the different approaches.

UNICEF, DPs

RationaleWithin Papua, even within UNICEF in Papua, there are differently labelled planning approaches being used at the district level - the Integrated Micro Planning being an example, with its focus on Puskesmas planning. Other development partners are supporting their own planning approaches as well. The processes have more commonality than differences. Common elements relate to the use of evidence, the importance and use of local information, problem analysis, budgeting and planning, monitoring and review.Where they differ is in their scope. Bringing these differently labelled approaches into a common framework would simplify the processes and increase the potential for scale up across all districts.

RiskSome elements of the IC process may be compromised in this approach.

23

Recommendations Actioning agency/ body

Rationale and risks

Different approaches needed for weakest districts Building on the experience of EBP, trial different approaches to addressing equity that would enable participation of the ten most deprived districts in Papua.

UNICEF RationalePapua is clearly the highest need province in Indonesia. However the highest need districts in Papua have not been targeted in the IC approach thus far. The Province is well aware of the challenges of its poorest performing districts, but is yet to develop an effective approach to supporting them. The IC activities presuppose a degree of competency at the district level, in terms of leadership and information. Security concerns also figure significantly as a barrier to effective intervention in these areas. There is a need to develop an approach to support these districts that would be multi sectoral in nature, and would prepare the districts to engage in the IC activities.

RisksThere is a high degree of difficulty in working in these areas, in terms of security, isolation, and risk of failure.

Broaden the participants engaged in the IC process Continue to explore the issue of how to involve the hospitals, private sector, and NGOs in the district planning processes.

UNICEF. IPs, GoI RationaleThe IC activities in Papua have, in some cases, effectively engaged non health sector participants and the district hospital. As experience and confidence in the process grows at the district level, there is a need to involve a wider group of participants, including the private sector. The dual role that public employees play in the private and the public sectors will require management within this process. The nature of engagement with the private sector needs to be consistent with the strategic intent of the planning process. The private sector’s participation as a stakeholder should be managed so that it does not compromise any regulatory or contracting relationship.

Risks There is a risk that the planning process becomes so broad that it loses focus and becomes ineffective. Engagement of other actors should be prioritised to those with the greatest ability to influence the health outcomes in question.

24

Recommendations Actioning agency/ body

Rationale and risks

Broaden the health scope of the IC processDevelop a mechanism for moving from an RMNCH&N focus to a health system-wide focus of district-level planning and budgeting activities, building on the EBP experience, and inclusive of a mechanism for priority setting.

UNICEF RationaleThe current IC approach has an exclusive focus on RMNCH&N. The need at district level is to plan and budget across the health portfolio, as well as to prioritise activities across the sector. The advantage of a comprehensive system-wide approach to planning, is likely to outweigh the advantage of single issue or group of issue planning in isolation by addressing core system issues that impact across systems.

RisksThe risk of moving from a selective to a comprehensive approach is loss of the focus required to progress particular issues. This can be mitigated if the approach is based on a local needs analysis and a prioritisation process.

Continued Monitoring of IC impactContinue to monitor the performance of EBP districts over the next five years.

UNICEF RationaleThere have been impacts from the IC activities noted in this evaluation, and further impacts are likely to occur outside of the timeframe of this evaluation. Continuing to monitor progress of the IC districts over a longer period is recommended in order to document developments that have a longer timeframe.

Integration with the planning and budgeting cycle.

Develop a consistent approach to IC planning reviews, following quality improvement principles, supporting a culture of long term, evidence informed cycle of planning, budgeting, implementation and review.

UNICEF, IP, GoI RationaleThe original design of the IC was as an advocacy tool, to be used at the national level. There have been a number of adaptions to its current role as a support for subnational planning and budgeting processes. Central to these, is recognition of the cyclical nature of the planning and budgeting and implementation process – also known as the PDSA (Plan, Do, Study, Act) cycle in the quality improvement literature. The longer term use of the IC in support of district system improvement can support this action review cycle. It would then be less about ‘one off’ advocacy, and more about continual learning and influence at the local level. This development would also link the IC activities with the growing body of support globally for quality improvement approaches within health systems.

RisksThere is a risk that the support required to run this process as part of the regular planning and review cycle would overburden the system at the local level. The design would need to ensure it did not place too much demand on local management.

25

ACKNOWLEDGEMENTS

This evaluation has been greatly assisted by the engagement, enthusiasm and insights of those interviewed.

A special thanks to Laxmi Zahara for her assistance, and her supplementary work in exploring the financing of Tolikara.

Thanks also for the support from UNICEF offices, both in Jakarta and Jayapura.

Finally, I wish to acknowledge the leadership and innovation shown by UNICEF regionally and globally, DFAT for their foresight and confidence in exploring this important area of health development, and UQc, UGM and University of Cenrdawasih for their leadership on the ground.

INTERVIEWEES

Name Designation

Dr Tiara Marthias Researcher, Center for Health Policy and Management, Universitas Gadjah Mada, Yogyakarta, Indonesia

Melkior Tappy Lecturer in Faculty of Public Health, University of Cenrdawasih

Sutarman Staff member, Reproductive Health Division, Papua PHO. Areas of responsibility: Reproductive health, IMCI, DTPS

Sukemi Head of Public Health Unit for MCH, Nutrition, and Planning, Bovem Digoel District Health Office

Dr Sudhir Khanal UNICEF CSD Specialist, Field Office, Jayapura

Widya Setyowati Senior Programme Manager, MCH development cooperation. DFAT, Jakarta

Debbie Muirhead Senior Analyst, DFAT, Jakarta

Dr Robin Nandy Chief of CSD, UNICEF, Jakarta

Dr Joseph Rinta Previous head of Provincial Health Office, Jayapura, Indonesia

M. Iriyanto Pawika Health Planning District Health Office, Jayawijaya

Nimpan Tarigan District Health Office, Yapen, Programme Development (data management)

Pak Makbul Head of Social Welfare Unit of BAPPEDA, Papua Province

Dr Lukman Hendrolaksmono Independent researcher. Currently on contract to UNICEF working on an evaluation of the DPTS

Kadek Hermanta Staff member, Provincial Health Office, Data and Information

26

Name Designation

Juniarto Damanik Head of Programme Unit of District Health Office, Yapen, Papua Province

Agustinus Subaagio Head of Planning Division, BAPPEDA, Provincial Health Office, Jayapura

Deswanto Marbun UNICEF Office, Jakarta

Dr Agnes Ang Head of Planning and Budgeting, Provincial Health Office, Papua

Dr Rheinhard Riedel Semuel Ratulangi

Obstetrician and Gynaecologist, Yapen Public Hospital , Yapen

Dr. Susi Natalia Paediatrician, Yapen Public Hospital, Yapen

Margaret Sheehan Chief, UNICEF Field Office, Jayapura

Rory Marwani UNDP Peoples Centred Development Progamme, Yapen

Prof Laksono Trisnantoro Professor in Health Policy, Faculty of Medicine Universitas Gadjah Mada, Indonesia

Dr Nida Rohmawati Head of Standardisation Section in Sub-Directorate for High Risk Under-5s, Ministry of Health (MoH), Jakarta.

Marten Masuri Member of DPRD (House of Representatives at Regional Level): Head of Budgeting Committee, Representative of 6 sub districts, Yapen

Maria Tanawani Head of Dinkes (District Health Office), Yapen

Marijke Yosefina Watifa Cinderwasih University, Jayapura

Dr Gita Maya Koemara Sakti Director of Maternal Health, Ministry of Health (MoH), Jakarta

Dr Lukas C. Hermawan, M.Kes Head of Sub-Directorate Maternal Health, Ministry of Health (MoH), Jakarta

Dr Mudjadid, M.Kes Head of Standardization Section in Sub-Directorate for under-5s, Ministry of Health (MoH), Jakarta

Frans Sanadi Wakil Bupati of Kabupaten Kepulauan Yapen (Vice Regent)

Alexander Nussy Head of Bappeda (Development Planning Agency at Sub-National Level), Yapen