BSI evaluation report 2017 · MoU Memorandum of Understanding MTEF ... BSI also has a Research and Dissemination programme, ... Hext Consulting Ltd BSI Evaluation Report 2017 July

BSI EVALUATION REPORT 2017

Shaun Hext and Benjamin Chemouni Hext Consulting Ltd July 2017

Hext Consulting Ltd BSI Evaluation Report 2017 July 4th 2017 2

Acknowledgments

The evaluation team is grateful to all those who have provided support, information and

comments, and especially to Simon Gill, Ilinca Balan and Richa Okhandiar-MacDougall from

ODI. Full responsibility for the text of this report rests with the authors.


Acronyms AfDB African Development Bank AGD Accountant General’s Department AGI Africa Governance Initiative AWR Annual Workstream Report BSI Budget Strengthening Initiative BTI Budget Transparency Initiative CABRI Collaborative Africa Budget Reform Initiative CAPE Centre for Aid and Public Expenditure CMC Cash Management Committee CPIA Country Policy and Institutional Assessment DFID Department for International Development DRC Democratic Republic of the Congo ECOWAS Economic Community of West African States FCAS Fragile and Conflict Affected States GIZ Gesellschaft für Internationale Zusammenarbeit ICAI Independent Commission for Aid Impact IEV Independent Evaluator IFC International Finance Corporation IFMIS Integrated Financial Management Information System IMF International Monetary Fund M&E Monitoring and Evaluation MDTF Multi-Donor Trust Fund MICs Middle Income Countries MoF Ministry of Finance MoFED Ministry of Finance and Economic Development MoFEP Ministry of Finance and Economic Planning MoU Memorandum of Understanding MTEF Medium-Term Expenditure Framework NGOs Non-government organisations NRA National Revenue Authority ODA Official Development Assistance ODI Overseas Development Institute OECD Organisation for Economic Cooperation and Development OECD-DAC OECD –Development Assistance Committee P4R Program-for-Results PDIA Problem-Driven Iterative Adaption PEFA Public Expenditure and Financial Accountability PFM Public financial management SDG Sustainable Development Goals SIDA Swedish International Development Cooperation Agency SoC Story of Change TA Technical assistance TOR Terms of reference UNCDF United Nations Capital Development Fund UNDP United Nations Development Programme USAID United States Agency for International Development WTO World Trade Organisation


Table of Contents

Executive Summary ..................................................................................................................... 5

1. Introduction ...................................................................................................................... 10 1.1 Evaluation purpose .................................................................................................................. 10 1.2 Strategic focus and objectives ................................................................................................. 10 1.3 BSI approach ............................................................................................................................ 11 1.4 Programmatic scope and resource allocation ......................................................................... 13 1.5 Evaluation scope and approach ............................................................................................... 14 1.6 Evaluative framework .............................................................................................................. 14 1.7 Methodology ............................................................................................................................ 15 1.8 Research limitations ................................................................................................................. 15 1.9 Report structure ....................................................................................................................... 16

2 Relevance ......................................................................................................................... 16 2.1 Is BSI providing high quality relevant advice? ......................................................................... 16 2.2 How relevant is BSI to the peace building and state-building goals under the New Deal? ..... 17 2.3 How relevant is BSI’s research component? ........................................................................... 18 2.4 How does BSI ensure a gender and conflict sensitive perspective to its work? ...................... 19

3 Delivery ............................................................................................................................. 21 3.1 How demand-led has BSI been? .............................................................................................. 21 3.2 Is the quality of BSI’s relationship with counterparts conducive to effective technical assistance? ............................................................................................................................................ 23 3.3 How does BSI approach differ from other technical assistance providers? ............................ 25 3.4 How cost-efficiently and cost-effectively has BSI provided its inputs? ................................... 26 3.5 How well has BSI managed uncertainty and risk in fragile states? ......................................... 29 3.6 How well does BSI manage for results? ................................................................................... 30

4 Results .............................................................................................................................. 31 4.1 Has BSI had a catalytic effect? ................................................................................................. 32 4.2 Has BSI been able to increase uptake of good practice and promote innovation? ................ 33 4.3 Has BSI contributed to identifiable institutional change? ....................................................... 37 4.4 Has BSI had an effect on government strategy and policy on PFM reforms? ......................... 38

5 Sustainability ..................................................................................................................... 40 5.1 Is BSI having a sustainable impact on budget, policies, processes and systems capacity? ..... 40 5.2 Has BSI increased government capacity and capability sustainably? ...................................... 41

6 Conclusion and recommendations .................................................................................... 41

Annexes .................................................................................................................................... 45 Annex 1: Interview List (names available on request) .................................................................. 45 Annex 2: Bibliography.................................................................................................................... 46 Annex 3: BSI 2015 evaluation recommendations progress report and rating .............................. 47 Annex 4: Annual workstream evaluation and scoring ................................................................... 51 Annex 5: Validation and assessment of ‘Stories of Change’ .......................................................... 60 Annex 6: Logframe validation ........................................................................................................ 69 Annex 8: ToR .................................................................................................................................. 73

Executive Summary

Overall we found BSI to be a very efficient and effective programming initiative. Using a genuine demand-led, flexible and adaptive approach, BSI has strengthened PFM systems in fragile environments in ways that demonstrably contribute to service delivery, state and peace building. Informed by and contributing to high quality fiscal governance policy and research, BSI interventions have added significant value to enhancing fiscal governance in FCAS.

This report presents an independent evaluation of the Overseas Development Institute’s (ODI) Budget Strengthening Initiative (BSI) over the past two years (April 2015-March 2017). The evaluation builds on previous independent assessments undertaken in 2013 and 2015. Established by ODI in 2010, BSI aims to provide timely, trusted and tailored technical assistance, predominantly to national finance ministries and revenue authorities to improve effectiveness, transparency and accountability of budget processes, policies and systems. As an innovative programming approach, the BSI model aims to engage with the perceived limitations of conventional approaches to institutional reform in poorer countries. It recognises that many PFM reforms failed when underpinned by “best practice solutions”, often imposed by donors on countries that did not have the capacity or the political appetite to implement them, and yet are eager, for funding reasons, to demonstrate to donors that the reforms had occurred. The BSI model instead is premised on being ‘demand-led’ and ‘problem-driven.’ It deploys fiscal generalists and short-term technical specialists with the knowledge, expertise and relational skills needed to provide timely, contextually relevant strategic advice. BSI operates programmes in five countries (DRC, Liberia, Sierra Leone, South Sudan and Uganda). South Sudan, Uganda and Liberia are currently the three largest country programmes. In addition to country programmes, BSI also assists the g7+ group of fragile states through support to the g7+ secretariat (located in the East Timor). BSI also has a Research and Dissemination programme, which seeks to shape and influence PFM policy and practice in Fragile and Conflict Affected States (FCAS). The primary focus of this evaluation is the Liberia and Sierra Leone programmes (the only two countries visited). Our assessment of the g7+, research components, and South Sudan and Uganda country programmes are light touch (BSI work in these two country programmes are reviewed annually by DFID, as they are funded by DFID earmarked accountable grants), the DRC programme was evaluated by AfDB earlier in the year and the programme is now completed. The evaluation constitutes an assessment of BSI value and effectiveness. We engage with 16 questions included in the evaluation terms of reference with a view to validating results and testing the plausibility of BSI’s underlying theory of change. The report is structured in four sections: Relevance, Delivery, Results and Sustainability, adhering loosely to OECD-DAC criteria. Our evidence base is both qualitative and quantitative. Data collection tools included key informant interviews, a focus group, short questionnaires and analysis of programme documentation. We have arrived at a qualitative judgment regarding BSI’s plausible contribution to institutional change in FCAS but many of our findings are context specific and hard to generalise across the piece.


Relevance

BSI strategic objectives continue to remain relevant to the priorities of host government and donors working to tackle fiscal governance problems in FCAS. In the context of fiscal instability and growing fiscal crisis in many countries where BSI works, BSI efforts to improve PFM policies, systems and processes remain closely aligned to government priorities for resource utilisation, ODA and tax revenue protection and growth. There is clear evidence of BSI adaptability and flexibility, translating its ‘demand-led’ and ‘problem-driven’ approach into practice, acting on changing priorities. Given rising debt levels in most of the Sub-Saharan countries, the political incentives to use TA to resolve revenue and resource utilisation problems are likely to remain strong in the near future. It would make more strategic sense if BSI’s results frameworks were clearly aligned to this outcome. BSI support to the g7+ secretariat efforts to leverage greater ODA flows and align OECD donor policy to New Deal principles is still relevant in order for the international community to recognise FCAS special needs, in an environment where other groups of fragile states are competing for attention and resources and the use of country-systems is increasingly questioned. However, in the countries we visited, actual demand by partners and country donors for support in this area remained weak. The research agenda is highly relevant both to BSI objectives of influencing policy and practice and to address gaps in the literature, although it can be difficult to align the research agenda to host government priorities in the same way that TA is closely aligned to state demand. Government partners participate in research dissemination but are less involved in framing the research agenda. BSI has taken concrete actions to integrate a gender and conflict sensitive perspective in its programming. This has increased the likelihood that government resource utilisation and tax revenue protection and growth efforts will be more equitable and do no harm. Delivery

BSI has put in place a highly responsive and adaptable delivery system. Programmes are characterised by a large number of small bets, work streams that utilise the right people at the right time to help tackle some of the PFM policies, systems and process problems that constrain better fiscal governance outcomes. Whether these are to ensure a higher share of aid is ‘on-budget’, to improve the accuracy of revenue and debt forecasting, to ensure more credible in-year financial reporting or more equitable fiscal transfers to local government, among others tractable problems. As a result, expectations for all work-streams were largely met or exceeded over the two-year period. Of the 14 streams of work BSI undertook in 2015/16 all were validated and met or exceeded planned expectations. BSI performance was equally impressive in 2016/17 (with the exception of two South Sudan work streams, affected by the 2016 escalation in conflict) in generating incremental returns. BSI has demonstrated that a demand-led, problem-driven, tailored, technical process approach represents good value for money. BSI day-to-day management decisions are generally informed by sound analysis of the politics of each reform, mitigating risk, and enhancing likelihood of effectiveness. BSI uses potentially innovative monitoring and reporting tools. Outcome logs and annual work stream reviews enable BSI to report and discuss progress well internally and to report to donors. BSI ability to use M&E tools to ‘crawl the design space’ and ‘enhance the authorising environment’ is less developed. Work stream outcomes can at times lack clarity and as part of the BSI model, the M&E tools need to be more clearly aligned to problem-driven, iterative, analytic (PDIA) principles.


Despite a stated donor commitment to such PDIA approaches and recognition of the model’s comparative advantages, many donors remain wedded to large-scale, country-specific, core PFM systems programmes with multiple pre-planned output-based contracts. In this environment, BSI increasingly needs to manage a high level of funding uncertainty, particularly due to a reduction in available core funding (a key enabler of PDIA work undertaken by arms-length organisations such as BSI). The stop/start reality of most PDIA-type programming, coupled with funding uncertainty, has implications for how BSI manages programmes, potentially hindering the capacity of BSI to remain faithful to its design and delivery principles as much as ensure continuity of support, expand its portfolio or even exit from programmes without burning the bridges needed to step back in again. Results

Broadly, BSI interventions to enhance PFM policies, systems, and processes have all contributed in some ways to generating incremental improvements in resource utilisation, revenue protection and revenue growth. In some cases, results have also been transformative, generating high value short- term financial returns as well as profound improvements in public financial accountability. Our validation of 21 BSI ‘Stories of Change’ (SOC) identified 11 work streams where we considered the change to be ‘significant’. In our opinion this is a very positive achievement given the short time frame and highly volatile operating contexts. Validation of the reported SOC results echo the AWRs in terms of exceeding expectations regarding likelihood of contributing to institutional change. BSI has clearly had a catalytic effect in many of its operational contexts. It has often achieved this by cutting through complexity, designing simple technical solutions that help make systems work better, shepherding multi-agent processes that improve collaboration and providing timely trusted strategic advice, financial modelling to support negotiating process and influencing macro-economic and international tax and trade policies. In more stable post-conflict settings, BSI has been successful at role modelling, utilising technical processes that improve collaboration among actors who more often compete with each other rather than work together towards achieving common outcomes. This approach engages effectively with the collective action problems that can derail important budget processes or undermine desired revenue policy outcomes. In this way, BSI has helped protect ODA flows and domestic tax receipts but also gone some way towards shifting institutional norms. The contribution to such outcomes has been greater in the fragile contexts of Liberia and Sierra Leone than in the conflict setting of South Sudan, where political violence escalated in mid-2016. Although in this case the results were none the less impressive. BSI remained engaged for far longer, taking over the work of other departing technical assistance programmes, enabling finance leaders to stay on track, stopping PFM processes from falling apart, meeting budget deadlines and ensuring constitutional obligations in what was an increasingly unpredictable and volatile environment. It has been difficult to establish the extent to which improved budget revenue utilization, protection and growth of external ODA and domestic tax has had a bearing on service delivery outcomes (improved health and education allocation and expenditure) or on tackling societal grievances. BSI logical framework indicators at impact level were difficult to reliably validate. In our opinion BSI is more focused on process than the relationship between institutional outcomes and societal impacts. In the area of the support to the g7+, BSI has managed to provide timely and relevant support to the Secretariat to help it articulate fragile states’ interests through research and in high-level international fora, helping the g7+ to become a credible body internationally.


Regarding Research and Dissemination, BSI has produced high quality papers. We found that some have clearly contributed to influencing individual thinking and decisions though uptake has been more effective when combined with strong inter-personal relationships, suggesting that research dissemination processes would benefit from better integration to country programme processes. Sustainability

By adopting a flexible, behind the scenes delivery model, adhering to the fundamental principles underlying the PDIA approach, BSI interventions are more likely to contribute to developing enduring budget and revenue capacities than conventional output-based programming models. There is clear evidence of increased knowledge and skills, of practice change and mind-set change, even across finance policy decision-making networks. Yet the results are emergent and incremental, the authorizing space still limited and the practice changes still need embedding. It is difficult to confidently predict that all the results will be sustained. Some may be partially protected by legislative change and precedence but many are vulnerable to reversal, particularly in South Sudan as a direct consequence of conflict but also in Liberia and Sierra Leone due to political change. The concept of sustainable capacity, while desirable, is in itself problematic. Many finance leaders believe it to be more cost efficient and cost-effective to just draw down on global technical resources to tackle budget and revenue problems at the times when they are most needed. Such a capacity augmentation strategy has proven effective with high impact returns in Liberia and Sierra Leone. It is unlikely that demand for this type of advice will diminish in the short or medium term. Capacity augmentation should be seen as complementary to outcome-based PDIA and more conventional output-based capacity development approaches to sustaining PFM reforms in FCAS.

Recommendations We present only five recommendations. These are limited to tackling what we see as key value and effectiveness issues at the organizational level not to all issues or to each country programme. Recommendation 1 BSI should refine its outcome statement in order to more clearly articulate the link between efficient PFM policies, systems and processes and improved resource utilisation, revenue protection and revenue growth. The tighter coupling would also create the analytical space for BSI to articulate what the outcome actually looks like in different countries and institutional workstream contexts. Recommendation 2 Intervention-level outcomes also need to be clearer if BSI is to remain faithful to its PDIA aspirations. Outcome clarity will make it far easier ‘to trawl the design space’ and ‘enhance the authorising environment’, testing whether the particular strategic bet still makes political sense. Outcomes need to be measurable, providing leaders with a real sense of what they actually look like in practice. Recommendation 3 As BSI operates in an increasingly uncertain funding environment it needs to have risk mitigation policies and triggers in place that enable it to quickly and consistently take decisions with regard to exit, scale-down or scale-up. BSI should draw policy and practice lessons from recent experiences in South Sudan, Sierra Leone and DRC. Such analysis will have wider value (e.g. as many NGOs are currently operating in similarly unpredictable short-term fragmented funding environments).


Recommendation 4 BSI would benefit from experimenting with ‘Most Significant Change’ exercises (table 38). As part of this exercise it should revisit the purpose and process associated with its ‘Stories of Change’ tool.1 It should also explore means of introducing or tightening leadership feedback loops, ensuring the change dialogue travels through the hierarchy as well as across PFM policy outcome networks. Recommendation 5 BSI should ensure that its theory of change accounts more clearly for the role of capacity augmentation. Rather than seek to clarify the similarities and differences between BSI Advisors and ODI Fellows, BSI should document the strategic contexts and practice where Advisors and Fellows have worked together to achieve common outcomes and add overall value to PFM reforms.

1 Davies, R. and Dart, J (2005) The ‘Most Significant Change’ technique, a guide to its use, (also see AGI adapted MSC exercises and various organizational adaptations - www.unitar.org or www.click4iit.org).

http://www.unitar.org/


1. Introduction 1.1 Evaluation purpose This report presents an independent evaluation of the Overseas Development Institute’s (ODI) Budget Strengthening Initiative (BSI) over the past two years (April 2015-March 2017), focussing only on part of BSI’s work over the period. It evaluates the country programme in Liberia and Sierra Leone and conducts a “light touch” assessment of the Uganda and South Sudan programmes, the support to the g7+ Secretariat and the Research and Dissemination component. The evaluation builds on previous independent assessments undertaken in 2013 and 2015.2 Initiated by ODI, the evaluation assesses the value and overall effectiveness of BSI. While the evaluation supports ODI accountability, it is also focussed on generating learning around BSI’s approach to improving fiscal governance in fragile and conflict-affected states (FCAS). The report is written for ODI and for BSI’s funders, primarily Sida, whose core funding has enabled work in Liberia, Sierra Leone and on the g7+ and Research & Dissemination during the period. It is hoped the evaluation will also appeal to those interested in the practical experiences of applying problem-driven adaptive programming approaches to fiscal governance reforms in fragile states. 1.2 Strategic focus and objectives

Established by the ODI in 2010, BSI aims to provide timely, trusted and tailored technical assistance, predominantly to national finance ministries and revenue authorities to improve effectiveness, transparency and accountability of budget processes, policies and systems. Much of BSI’s work can be contextualised as part of wider international efforts to improve the efficiency and effectiveness of budget cycle management policies, processes and systems in FCAS (figure 1). An important strand of BSI’s work involves technical support to tax policy departments and national revenue authorities, with a view to tackling large and growing budget deficits in many of the concerned countries.

Figure 1: Budget cycle management

The hierarchy of objectives, or ‘results chain’, summarised as a causal logic in BSI’s overarching logical framework and echoed in the BSI theory of change diagram, posits that improved fiscal governance contributes to better service delivery, peace and state building (figure 2).3

2 Cox, M. and Robson, K. (2015) BSI Evaluation, Agulhas Applied Knowledge. 3 BSI Refined Theory of Change, (2016) and Logical Framework (2016).

External and

Domestic Revenue

Budget Policy

Budget Framework

Budget Approval

Budget Execution

Budget Reporting

Budget Oversight


Figure 2: BSI results chain (annotated)

Indicative of better fiscal governance are, a) increased government spending on social priorities, b) an improved budget process, engaging line ministries and other stakeholders, including politicians, and, c) improved budget planning and execution (evidenced by a decrease in variance between approved budget and executed expenditure). Donors are not without agency and improvements in international engagement with FCAS government systems is considered a pre-requisite to achieving progress towards sustainable development goals4, not just in terms of ODA flows to fragile states but also of alignment of OECD donor resources to New Deal principles (e.g. government budget systems).5 1.3 BSI approach As an ‘arms length organisation’6, BSI was conceptualised as an innovative way of engaging with the perceived limitations of conventional approaches to institutional reform in poorer countries.7 It recognised that most PFM reforms failed to achieve sustained improvements because “best practice solutions” had often been imposed by donors on countries that did not have the capacity or the conducive political environment to implement them, and yet were eager, for funding reasons, to demonstrate to donors that the reforms had occurred. As a result, many reforms have been formally undertaken, yet without altering day-to-day practices, evidence of form but not of functionality.8 Hence, the BSI model is premised on being ‘demand-led’ and ‘problem-driven’, deploying fiscal generalists and short-term technical specialists with the knowledge, expertise and relational skills needed to provide timely, contextually relevant strategic advice. They work iteratively alongside local counterparts, on their team, gaining trust, identifying and capitalising on emergent opportunities (figure 3). They focus on areas where there are actual or at least potential coalitions of interest for change, fostering small, incremental but nevertheless ‘significant’ institutional changes.9

4 Particularly SDG 16. 5 BSI Logical Framework (updated 2016). 6 Booth, D. (2013), Facilitating development: an arm’s length approach to aid. London: ODI. 7 Andrews, M. (2013) Limits to Institutional Reform: Changing Rules for Realistic Solutions, Cambridge University Press, Booth, D. and Cammack D. (2013) Governance for development in Africa. London: Zed Books. 8 Ibid. 9 Williamson, T. (2015) Change in challenging contexts: how does it happen? Executive Summary, September 2015, London: ODI.

Trusted, timely, tailored technical

advice and support

enhanced policy, process, systems

More effective, transparent and

accountable budget policy, process and

systems

Improved fiscal governance

improved serviice delivery weakens economic drivers

of conflict


Figure 3: BSI relational model 10

1.3.1 Relational behaviours Effective technical assistance is premised, according to proponents of such ‘Problem-Driven Iterative Adaption’ (PDIA), not just on iterating simplified data-driven solutions or ‘trawling the design space’ to tackle ‘wicked hard’ PFM reform problems11, but on ‘Craftwork’: a way of working that gives primacy to soft relational skills, e.g. acting technically but thinking politically (figure 4).12 Figure 4: Advisor ‘Craftwork’13

• Thinks politically – whose interest, why now, what are the risks

• Enlists the voice of trusted others to promote ideas for change

• Role modeling - demonstrates good practice and performance

• Challenges entrenched views and question old habits

• Helps identify multiple champions, fosters stronger ties and coalitions

• Is persistent, gathers hard evidence and makes compelling cases

• Encourages experimentation and learning, trial and error.

1.3.2 Roles of external actors BSI identifies four main roles external actors can play to support change processes.14 These resonate with the ‘craftwork’ proposed by other arms-length organisations (figure 5).15 Figure 5: Roles of external actors16

• The trusted advisor: - supports reformer to understand and address the problems they face.

• The facilitator and broker: - helps reformers to build coalitions, develop common understanding, create consensus on how problems can be solved.

10 Ibid. 11 Andrews, M. et al (2015) Building capability by delivering results: Putting Problem-Driven Iterative Adaptation (PDIA) principles into practice. Paris: OECD. 12 Hymowitz, D. (2016) Too much science not enough art, January 2016, Africa Governance Initiative 13 ibid. 14 Williamson, T (2015) Change in challenging contexts: how does it happen? Executive summary, September 2015, ODI 15 Hymowitz, D (2016), Shoulder to Shoulder, African Governance Initiative, November 2016. 16 Williamson, T. ibid.

AuthoriserPresident, Cabinet,

minister, top bureaucrat

Core team technicians,

civil servants, BSI technical

advisor

Team leader and convenor

Middle manager

Coalitions

Within institution,

government agencies, TA,

donors


• The ‘dot joiner’: helps identify and take advantage of opportunities, to build links, ensure consistency and complementarity between reforms, processes and systems.

• The technician: supports the development, implementation and management of processes and systems, and capacity building

1.3.3 Underlying BSI assumptions Likelihood of effectiveness relies strongly on BSI capacity to adapt its response to the realities of the environment (e.g. lack or attrition of skilled counterparts, low pay and poor counterpart motivation, a fragile authorising environment, a reactive, chaos-management culture, frequent leadership change, changes in political priorities, a preference for informal decision-making, among others). The PDIA approach is contingent on a wider range of relational assumptions holding true (figure 6). Figure 6: Underlying BSI assumptions17

• The programme has permission to pursue reforms in a flexible iterative manner

• A coalition of shared interests exists or can be built in favour of the reform

• The right technical advisors with the right soft skills can be recruited at the right time

• Ideas can be catalyzed by outsiders working closely inside government

• Large numbers of small bets are more effective than small numbers of big bets

• Leadership change will likely reduce reform openings and constrain progress18

• People want to share information, they are just not prepared to take the lead in doing so

• Improved availability of credible evidence will increase demand for credible evidence and reduce the likelihood that leaders take policy decision without assessing the evidence

At the higher level, the BSI model is also premised on the assumption of a causal relationship between service delivery and state-legitimacy, particularly the assumption that adequate and equitable health, education or agricultural budget transfers will contribute to tackling horizontal inequalities.19 1.4 Programmatic scope and resource allocation BSI has programmes in five countries (DRC, Liberia, Sierra Leone, South Sudan and Uganda). South Sudan, Uganda and Liberia are currently the three largest country programmes (figure 7). In addition to country programmes, BSI also assists the g7+ group of fragile states through support to the g7+ secretariat (located in the East Timor Finance Ministry). BSI also has a Research and Dissemination programme, which seeks to shape and influence PFM policy and practice in FCAS. Sida is a key funding partner for the period under consideration. Sida provided BSI with core funding (£2.15m) for a three-year period (2015-17). BSI allocated a high proportion of these funds to programmes in Liberia (£661k), Sierra Leone (263k), research and dissemination (£242k), g7+ (£251k), providing about 20% of total funding over the period. During the period BSI received grants from other donors; notably DFID fully funded the South Sudan programme and most of the Uganda programme. The African Development Bank was the main funder for DRC, and GIZ supported the resource governance work in Liberia. Other donors have provided smaller amounts of short-term country or component specific funding (World Bank, IPA, Make All Voices Count, European Union, CABRI, UNCDF, UNDP, and the OECD). BSI has also received a small grant from the Government of Sierra Leone. BSI spends between £4m and £5m per annum, with over 90% spent on direct technical assistance accounts and less than 5% on design and development, governance, M&E, management and direction (figure 7).

17 Select ODI and PDIA literature. 18 The opposite must also be an assumption. 19 BSI Theory of Change.


Figure 7: Expenditure by component and year

Component FY 15/16 % FY 16/17 %

South Sudan £1,501,181 37.20% £1,853,427 38.00%

Liberia £638,290 15.82% £642,442 13.17%

DRC £635,914 15.76% £466,635 9.57%

g7+ £231,945 5.75% £114,095 2.34%

Research & Dissemination £145,059 3.59% £93,935 1.93%

Uganda £329,428 8.16% £1,187,413 24.35%

Sierra Leone £182,183 4.51% £195,269 4.00%

Other £107,696 2.67% £95,675 1.96%

Programme direction, management, M&E £263,374 6.53% £228,425 4.68%

Total £4,035,068 100.00% £4,877,316 100.00%

1.5 Evaluation scope and approach BSI programmes in Liberia and Sierra Leone were visited, these two countries being the primary focus of this evaluation, as operations in South Sudan and Uganda are reviewed annually by DFID, whose country offices provide earmarked accountable grants (South Sudan - £4.3m, Uganda - £1.57m). Our assessment of other components (Research and Dissemination and g7+) was similarly light touch. DRC is not included as it was evaluated earlier in the year and is now completed.20 1.6 Evaluative framework In terms of our broader assessment of value and effectiveness, particularly the plausibility of BSI’s underlying theory of change, we engaged on 16 evaluative questions (figure 8). Fifteen were framed by BSI in the TOR and included supplementary questions (e.g. the relevance and contribution to peace building and state-building goals under the ‘New Deal’, the relevance, quality and uptake of the research and dissemination component).21 We have also assessed the results of efforts to integrate a gender and conflict sensitive approach, to enhance gender equality and to ‘do no harm’ as well as how BSI manages for results, a question that emerged inductively from our research. Figure 8: Evaluation questions

Evaluation Questions

1. Is BSI providing high quality and relevant advice?

2. How relevant is BSI to the peace building and state-building goals under the New Deal?

3. How relevant is the BSI research component?

4. How does BSI ensure a gender and conflict-sensitive perspective to its work?

5. How demand-led has BSI been?

6. Is the quality of its relationship with counterparts conducive to effective assistance?

7. How does the BSI approach differ from other technical assistance providers?

8. How efficiently and cost effectively has BSI provided its inputs?

9. How well has the programme managed uncertainty and risk in fragile states?

10. How well does BSI manage for results?

11. Has BSI had a catalytic effect?

12. Has BSI been able to increase uptake of good practice and promote innovation?

13. Has BSI contributed to identifiable institutional changes?

14. Has BSI had an effect on government strategy and policy on PFM reforms?

15. Is BSI having a sustainable impact on budget, policies, processes and systems capacity?

20 AfDB (2016) Evaluation of BSI in DRC 21 TOR. See Annex 7.


16. Has BSI increased partner capacity and capacity sustainability?

1.7 Methodology The evaluation was conducted over three months, between March 1st 2017 and June 1st 2017 by two independent consultants with no previous relationship with ODI. The methodology was outlined in the inception report and agreed with BSI management.22 A mixed approach involved the collection and analysis of a wide range of qualitative and some quantitative data. A variety of data collection methods were used (figure 9). These included key informant interviews, a focus group (Sierra Leone), short questionnaires (Liberia), and analysis of programme plans and reports, including Beneficiary Questionnaires (BQ), Annual Workstream Reviews (AWR) and Stories of Change (SoC). Figure 9: Data collection tools and validation processes

Country visits One-week visits to Sierra Leone and Liberia were undertaken during March 2017. More than 50 interviews were conducted during the two visits, including BSI advisors, government counterparts, other technical assistance providers, and in country donors. Key Informant Interviews 98 stakeholders were interviewed in total (annex 1). The evaluators used a semi-structured approach to encourage joint reflection. Simple interview guides and themed-based question frames were produced, while questions adhered to the wider question framework these were adapted, as themes and categories emerged from the data. Interview transcripts were produced for each interview and at times validated later by the interviewee. Quotes are unattributed unless with consent. Focus groups: A focused group discussion with 11 newly recruited graduate economists was held in Sierra Leone. The method adhered to the key informant interview protocols. Any individual written responses provided by participants were also anonymous and non-attributable. Questionnaires: A small number of BQs solicited from government partners by BSI in 2016 and 2017 were collated and analysed. A short questionnaire (10 questions) was also designed and used as an entry point for discussions with 14 government partners in Liberia. Documentation: The team drew extensively on a wide range of documents, including BSI country outcome plan and logs, BSI board reports, risk registers, donor proposals, government policy documents, former reviews and evaluations, among others. Validation and rating of BSI annual results: We also validated and rated three types of BSI monitoring and reporting instruments (BSI overarching logical frameworks, 28 AWR and 21 stories change, the latter two authored by each work stream). We assessed the extent to which AWR expectations were met and the significance of the SOC using the BSI assessment criteria (annex 4 and 5).23 Validation of progress against recommendations made in the 2015 evaluation Our data collection and analytic process has also been used to validate BSI’s assessment of progress against the eight recommendations made in the 2015 evaluation (annex 3).

1.8 Research limitations BSI acknowledges the overarching logical framework as more the legacy of a DFID funding agreement than a coherent logical model. While the output, outcome and impact statements provide an analytical steer it is accepted that they lack coherence and do not in themselves offer a viable theory of change on which to structure an evaluation. The TOR requirement to dive deeper into Liberia and Sierra Leone and the expectation of a comprehensive validation of all AWRs and SoC in order to populate the corporate overarching logical framework indicators replaced the need to develop a workstream sampling strategy. Assessments of Uganda and South Sudan AWRs and SOCs were lighter touch, uninformed by country visits. To mitigate the risks of positive bias during country visits (ODI’s long-standing relations with some counterparts and, excepting Sierra Leone, the generally cost-free nature of the relationship), we have triangulated opinions through the government hierarchy, across different departments and agencies,

22 BSI Evaluation Inception report, April 2017 23 definitions of AWR and SOC


as well as with other external technical assistance providers and donors. We have not been able to triangulate opinions expressed by finance ministries with sector ministries. Where possible we have drawn on hard data and actual observed behaviours to substantiate claims. We arrived at a qualitative judgment regarding the plausible contribution BSI has made to institutional change in the different contexts it operates but given the multiple drivers of change, attribution has not been possible. 1.9 Report structure The report flows according to the sequencing of the 16 TOR questions (e.g. we first ask whether BSI is providing quality relevant advice). These questions are organised in four sections: Relevance, Delivery, Results and Sustainability, broadly aligned to OECD/DAC and ICAI evaluative criteria.24 Structuring in this way offers a loose framework to test the validity of the theory of change. The final section presents a limited number of conclusions and related recommendations. Results validation tables (AWR and SOC) are presented in the annex and referenced throughout the report.

2 Relevance The section engages with four relevance questions included in the TOR (figure 10). The section aims to assess and provide insights into the extent that BSI advice has remained aligned to the strategic priorities of partner governments. We consider the strategic relevance of the peace building and state-building under the New Deal and of BSI research to address gaps in the literature on PFM in FCAS. We also assess how BSI has integrated a gender and conflict sensitive perspective to programming. Figure 10: TOR questions

• Is BSI providing high quality relevant advice?

• How relevant is the BSI to the peace building and state-building goals under the New Deal?

• How relevant is the Research and Dissemination component?

• Does BSI ensure a gender and conflict sensitive perspective to its work? 2.1 Is BSI providing high quality relevant advice? According to our validation of 28 AWRs and the analysis of BQs, BSI provides high quality and appropriate technical assistance, including strategic and operational advice (section 3 and annex 4). Leaders we spoke with believed that BSI technical and policy advice had helped them tackle some of the problems ‘that keep them awake at night’. Such problems were complex, usually associated with a deepening fiscal crisis, often caused by factors outside their direct control (figure 11). Figure 11: The big issue that matters to most partners - macro-economic instability

Tackling macro-economic instability has become a key priority for many Sub-Saharan states, particularly those affected by fragility and conflict. Reductions in the revenue and limited fiscal space often precipitated by a collapse in global commodity prices (e.g. Oil in South Sudan, Rubber in Liberia, Iron Ore in Sierra Leone), but also a shift in ODA flows away from general or sector budget support to direct humanitarian interventions have incentivised many leaders to prioritise the protection or increase of revenue flows and necessitated policy change.

24 http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm .

http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm


Lack of revenue makes it harder for finance ministers to prioritise, particularly sector budget allocations and transfers to sub-national entities as many interest groups are demanding a share from a smaller pot. In some cases, service sector salaries are being paid late and there is growing tension between discretionary and non-discretionary payment policies. While the maintenance of core budget systems remains important to ministers, the yawning budget deficits in many countries have increased the need for higher quality and more frequent forecasting and reporting of revenue, expenditure and debt, among other fiscal governance priorities. Country leaders have also placed greater emphasis on adjusting policies in order to mobilise direct tax revenues, particularly with a view to protecting and growing the natural resource revenue base.

For partner governments, more ‘effective, transparent and accountable PFM policies, systems and process’ is increasingly seen as a means of mobilizing domestic and external resources and improving resource utilisation. BSI has responded to this shift. It is agile and adaptive ‘going with the grain’, tailoring advice to the shifting strategic and operational priorities of each country. In some cases, BSI has re-aligned more of its advisory effort in order to offer high quality macro-economic advice, borne of a sound understanding of the countries’ economic policy context, intended, for instance, to tackle structural impediments (e.g. exchange rate reform analysis and fuel subsidy analysis in South Sudan). In other cases, BSI has further refined work stream focus. For instance, instead of just building capacity in revenue units, it also provides highly specialist short-term technical advice to aid critical government negotiations with multi-nationals and multi-laterals (e.g. agricultural conglomerates in Liberia or IMF relations in Sierra Leone). We noted in both Liberia and Sierra Leone that BSI was offering more advice intended to grow and protect the domestic tax revenue base (e.g. audits of artisanal mining companies, tax policy analysis on state-owned enterprise fee income) and domestic resource utilisation (e.g. public investment tracking and short-term domestic debt sales). BSI has also been politically savvy, recognising the need for governments to make trade-offs, particularly when the strategic priorities of country donors are less closely aligned to the strategic priorities of their partner government. For instance, aligning economic strategies to all SDG goals, strengthening civil society organisations to ensure education budget accountability, or promoting gender equality through health sector use of ‘gender budgeting’ tools were relevant for government as they constituted a means to keep donors happy. As a recent article on Liberia noted, “they’d rather be getting money for something that isn’t a priority than not be getting any money at all’’.25 While such actions could potentially mean that governments’ commitment on these topics may wane overtime, we however feel that BSI has been right to provide support on them. This has allowed BSI to foster trust while helping governments to keep good relationship with donors. In addition, it may help to demonstrate the usefulness of these reform areas to partner governments. 2.2 How relevant is BSI to the peace building and state-building goals under the New Deal? BSI reinforcement of PFM systems in FCAS was relevant to the peace building and state-building goals under the New Deal as it helped countries to move out of fragility through building resilient and accountable institutions. BSI intervention was consequently in line with the New Deal TRUST Principles26 of improving transparency, strengthening capacities, improving predictability of aid and, by improving PFM, encouraging the use of country systems by donors. In addition, BSI interventions

25 The Economist, July 1st, 2017, Foreign Aid: Fading faith in good works. 26 These are Transparency, Risk Sharing, Use and Strengthen Country Systems, Strengthen Capacities, Timely and Predictable Aid.


on macro-economic stability, revenues and fiscal transfers for service delivery contributed to a more stable economy, revenues and services, in accordance with two of the New Deal’s Five Peacebuilding and Statebuilding Goals (PSGs).27 Support to the New Deal also involved support to the g7+ secretariat and its international engagement and OECD influencing role. We consider this support relevant as the g7+Secretariat is focussed on helping fragile states to leverage external resources (e.g. as demonstrated by the IDA replenishment which was favourable to fragile states), as well as keep the principles of the New Deal alive in international forums (e.g. in the Third International Conference on Financing for Development in Addis), while making the international community aware of specific FCAS challenges. This would be very difficult for individual countries to achieve without the representative role of the Secretariat. According to the secretariat, the relevance of BSI support has been enhanced by its capacity to adapt to the fast-changing landscape of aid. While fragile states remain a high priority on donors’ agenda, other groups of states are competing for international attention (G77, Small Island Developing States). In such an environment, BSI support to the g7+ in making its voice heard and increase its capacity to clearly articulate its need was strategically relevant. In addition, we found that BSI engagement with the g7+ secretariat enabled it to realign itself to the changing strategic context, as donors become more interested in trade and private finance and SDG harmonisation. For instance, BSI supported research on investment in fragile countries, the g7+ engagement with the IFC on investment, and the creation of a g7+ SDGs tracking system. However, the officials we did engage with in the g7+ member countries (Sierra Leone and Liberia) we visited did not consider the g7+ initiative to be a priority. This resonates with findings from a recent New Deal evaluation.28 Rather than a challenge to the relevance to g7+ principles of TRUST and FOCUS, which remain high on the agenda of many countries, this rather shows that the ownership of the g7+ initiatives remains largely at the higher level, among a few top politicians. In our opinion the initiative remains relevant but is still nascent and highly vulnerable to shifts in national political incentives and changing priorities in the international donor arena. 2.3 How relevant is BSI’s research component? BSI research has aimed to explore the link between fiscal governance and state building, and influence practitioners to successfully deliver PFM reforms in fragile environments. We found that the research agenda was relevant as it tackled topics pertinent to practitioners and because these topics address recognised gaps in the fiscal governance and state building literature (figure 12).

Figure 12: Relevance of BSI publications (2015-17)

Papers Relevance to BSI objective Relevance in the wider literature Public financial management introductory guides

• Public investment management

• Organising a strategic phase in the budget process

• Fiscal decentralisation

Provide introductory guides to practitioners on reforms in different fragile, low-capacity environments in key PFM areas.

Guides for practitioners are mainly from the World Bank and the IMF and treat these issues from a “best practice” perspective. The BSI guides address second-best option approaches to reform, which is much more relevant in fragile environments.

27 These are Legitimate politics, Security, Justice, Economic Foundations, and Revenues & Services. 28 Hearn, S. (2016) Independent Review of the New Deal for Engagement in Fragile States. New York: New York University Center on International Cooperation for the International Dialogue on Peacebuilding and Statebuilding.


• Cash management in cash-constrained environments

• The ministry of finance “challenge function”

The capabilities of finance ministries

• Report

• Report UK

• Report Sierra Leone

• Report Uganda

Open the “back box” of ministries in charge of devising and implementing PFM reforms and identify drivers of / obstacles to effectiveness

The Ministry of Finance are not normally taken as an object of analysis in the literature

Change in challenging contexts - how does it happen?

• Report

• Executive summary

• Briefing paper one

• Briefing paper two

Analyse not the “what” but the “how” of PFM reform in fragile/low-capacity context

Similar objectives to pieces in development studies calling to “work with the grain” and “thinking politically”29, but more detailed, grounded in the concrete experience of BSI, and directly relevant to PFM practitioners.

Basic enough budgets: what should you do when you can’t do everything?

Provide framework of analysis and evidence on prioritising aspects of budget reform in low capacity environment.

Conventional approaches to budget reform are based on importing international best-practice models but the literature hardly provide analysis on “real-life” prioritisation in budget reform

PEFA: what is it good for? The role of PEFA assessments in public financial management reform

The paper challenges the appropriateness and underlines the perverse incentives the PEFA framework can create in PFM reforms.

Directly address the issue of ensuring that PEFA diagnostics translate into more effective PFM reform.

In Sierra Leone, the Finance Secretary had found the ‘capabilities of finance ministries’ research particularly relevant to his current organisational development agenda. The Budget Director also considered the PEFA research to be relevant in terms of informing his IMF engagement. There was less evidence of research question priority in Liberia. Some counterparts believed that the BSI research agenda could be better aligned to their current priorities rather than PFM per se, for instance examining the constraints to revenue mobilisation and resource utilisation. Others thought counterparts should have played more of a role in defining the agenda in the first place. 2.4 How does BSI ensure a gender and conflict sensitive perspective to its work? We noted that since the 2015 evaluation BSI has taken concrete steps to see all its work through a more gendered and conflict sensitive perspective. 2.4.1 Gender sensitive BSI’s is no longer as ‘gender blind’ as it was two years ago. It has started taking some small but none the less important steps to ensure it integrates a gender sensitive perspective to its programming. To this end, an initial ‘gender screening’ report is said to have acted as a catalyst for mainstreaming gender across the programme.30 An introductory guide to gender-sensitive PFM is currently being

29 Booth, D. (2011). Introduction: Working with the grain? The Africa power and politics programme. IDS bulletin, 42(2), 1-10; Booth, D., & Unsworth, S. (2014). Politically smart, locally led development. London: ODI. 30 Barnes-Robinson, K. and Mansour-Ille, D. (2016) Rapid review: why should it be an how could gender be better integrated into BSI’s work. ODI: London.


written and, following BSI retreats and country management meetings, there is more evidence of action across the country programmes. There has been a moderate increase in demand, with requests from government for BSI advisors to support work related to gender equality (figure 13).31 Figure 13: Budgets and gender equality

Gender equality is often a development principle but has seldom been a strategic priority for finance ministries. It may be framed as a ‘cross-cutting’ objective in national development agendas but is less likely to feature in specific PFM strategies.32 While governments usually rely on equal opportunity law to cut gender inequality, laws are recognized as important yet insufficient for ensuring women’s rights, particularly in terms of the ways scarce resources are allocated.33 There is evidence that some finance ministries are now prioritising budgeting activities intended to promote gender equality. In Uganda BSI has supported government to introduce a gender perspective to a Local Government Performance Assessment Manual. In Liberia, BSI contributed to terms of reference for piloting gender responsive health budgeting processes and presented the work at an IMF funded conference in Washington. Seeking better gender outputs in local government or in the health budget is a start, for Finance Ministries it still largely reflects a shift in the donor agenda. For BSI, it is a means of engaging on a key developmental issue without compromising their fundamental PDIA design principle.

2.4.2 Conflict sensitive BSI has also taken steps to integrate a conflict sensitive capability into programming, ensuring that its support to governments’ intended PFM reforms ‘do no harm’. These actions can be attributed to a related 2015 evaluation recommendation that BSI should ‘explore the use of scenario planning and other risk management tools to help manage its engagement in volatile contexts’ and to concern expressed by Sida at the same time, suggesting BSI needed to be more ‘conflict sensitive’. BSI has previously supported a number of ‘country fragility assessments’ (through support to the g7+ secretariat), notably in Sierra Leone. BSI supports PFM reforms that have the potential to reduce the structural drivers of conflict (e.g. budget transparency and service delivery equity in Uganda or natural resource governance in Liberia). In South Sudan, following an escalation of the conflict, it has undertaken a far more rigorous conflict sensitivity analysis. It subsequently convened a regular expert advisory group meeting to track the unfolding conflict dynamics in real-time. BSI’s recent decision to withdraw from South Sudan was intended to mitigate the risk of doing harm. BSI acknowledges a greater focus on analysing the drivers and constraints to PFM reforms than assessing the horizontal inequalities or the structural or proximate causes and interests that fuel civil war. It had not previously explicitly ‘joined the dots’, for instance, between its support to budget allocation systems and the timely payment of solders. If such analytical work was to be undertaken it would more likely to be in collaboration with the humanitarian or governance theme groups in ODI. It is our opinion that ODI as a whole has become more conflict sensitive and the different parts, including BSI, are now more joined up around the principle that interventions ‘Do no harm’.

31 BSI November 2016, Gender Assessment 32 For instance, see Liberia Agenda for Transformation 33 The Economist, February 25th, 2017, Gender Budgeting: the fiscal mystique.


3 Delivery The section examines the quality of the BSI delivery system. We engage with six delivery-focussed questions outlined in the terms of reference (figure 14). Collectively we aim to assess the extent that the BSI delivery system is fit for purpose. Figure 14: TOR questions

• How demand-led has BSI been?

• Is the quality of the relationships with counterparts conducive to effective assistance?

• In what way is BSI support distinguishable from other TA providers?

• How cost-efficiently and cost-effectively has BSI provided its inputs?

• How well has BSI managed uncertainty and risk in fragile states?

• How well does BSI manage for results?

3.1 How demand-led has BSI been? For BSI, being ‘demand-led’ is primarily a ‘supply-side’ question, a fundamental principle that informs where it agrees to work and what it agrees to work on. BSI adheres to a belief that if there is little real government demand for international technical assistance then PFM reforms are more likely to fail. Hence, being ‘demand-led‘ helps to mitigate the high risk of ‘isomorphic mimicry,’ by ensuring the technical solutions are aligned to the prevailing political interests and incentives.

BSI argues all work-streams are ‘demand-led’, given that an official request was needed to establish them in the first place. But being ‘demand-led’ is not just about ensuring alignment to different pillars and objectives of PFM sector strategy or poverty reduction and economic transformation agendas. It is more about being responsive to government requests to tackle specific tractable problems that undermine the efficiency and effectiveness of resource utilisation and protect external revenues flows, including ODA on which FCAS governments have become more reliant. There is strong and consistent demand for fiscal governance specialists with experience and skills in macro-fiscal policy analysis, critical thinking, advanced systems solutions, data capture programme design and financial modelling (figure 15). Such competencies are hard to find and retain locally. The large ODI community of fiscal governance specialists and the network of former ODI fellows (many are macro-fiscal specialists) makes ODI very well placed to cater to this demand. Figure 15: Typology of BSI strategic inputs

• Ensuring budget and revenue data quality

• Translating, simplifying, adapting and designing budget data tracking tools

• Brokering technocrat buy-in to a range of budget processes and tools

• Providing high level macro-economic and tax policy advice

• Undertaking and disseminating research on fiscal governance policy gaps

• Facilitating engagement on OECD donor policy in fragile and conflict settings.

We noted that BSI always seeks to validate the legitimacy of government demand (undertaking needs assessments, assessing political feasibility, seeking insider opinions from ODI Fellows and other like-minded embedded technical assistance providers, AGI for instance).


BSI acknowledges the risks are usually ‘high’. The original statements of intent made by finance leaders may not always translate into a real commitment to change practice. The underlying authorising environment may be weak, designed to signal commitment to deliver, for example, on an IMF structural benchmark or with a view to influence a World Bank investment decision. Although demand-driven, BSI has sometimes had an active role in ‘creating’ such demand. In Uganda for example, BSI was instrumental in convincing the MoFED to tackle problem of inadequate and inequitable fiscal transfers for service delivery (e.g. education and health). Through its support to the g7+, BSI was key to sensitising the Secretariat on the importance of preparing in advance for the Third International Conference on Financing for Development (see annex 5). In some cases, BSI has ‘catalysed demand’ from below, usually by working alongside teams to translate, design and iterate simple technical solutions to complex PFM problems, helping partners grasp the value of a data tracking tool, at times creating ‘ah ha’ moments (figure 16). Figure 16: Typology of data-driven technical problems (non-exhaustive)

• Producing credible in-year financial plans and reports on time

• Producing accurate revenue, debt and expenditure forecast regularly

• Assessing relationship between plans, budget allocations and cash availability

• Using IFMIS to reliably report in-year aid spend

• Benchmarking commodity prices and assessing transfer pricing practice

• Collecting, collating and analysing data to assess policy option costs and benefits

• Tracking actual transfers against public investment commitments

• Designing budget formula to calculate adequate and equitable fiscal transfers

BSI country managers can be reluctant to push if there is no demand, particularly if requests are not forthcoming. This has been particularly true of the g7+ initiative in Sierra Leone and Liberia where BSI had been conscious of limited demand for BSI support on the New Deal. Two interviewees from international organisations associated with the g7+ and New Deal initiative also supported the view that country ownership was weak despite high-level representation at international forums. BSI is good at building demand when it has the license to ‘reach out’ and find others who may support the reform, aiming to forge a wider coalition of support, one premised on collaboration, on identifying shared interest, usually among data holders, across a practice network (e.g. officials in the central bank and the revenue authority in Sierra Leone) but also among policy networks (e.g. those parties with an interest in reframing oil storage fees or mineral export fees as taxes in Liberia). BSI responds well to short notice requests. In Liberia, for instance, partners said that major concessions deals happened so infrequently that they did not need to retain all the skills and knowledge in house, just to draw down specialist advice when needed, usually at very short notice. Similarly, the g7+ secretariat team lauded BSI for responding to TA requests at very short notice. In the few cases where demand has been weaker than anticipated we noted that BSI has been agile, quickly making course corrections. In some cases, this has meant shifting TA to more promising areas. In others, advisors have few choices but to support conventional output-based work (e.g. training, guidelines, manuals), areas where the delivery incentives for government are stronger. Often it is a case of balancing conventional output-based approaches with PDIA type work that is more clearly focussed on a specific intervention outcome.


3.2 Is the quality of BSI’s relationship with counterparts conducive to effective technical assistance?

The BSI model is underpinned by a causal assumption: that likelihood of effectiveness is improved by the quality of the relationship between government counterparts and BSI advisors in variously ascribed roles (e.g. ‘trusted advisor’, ‘technician’, ‘facilitator and broker’, ‘dot-joiner’).34 Broadly our findings support this assumption, recognising the utility of the respective roles but also, and perhaps more importantly, their potential complementarities and synergies. 3.2.1 The Trusted Advisor In the four countries we studied, as well as in the g7+ Secretariat, there often existed a BSI advisor that had the ear of senior management, whose opinion was frequently sought. In the countries studied, the topics were typically those of fiscal policy or bigger macro-economic policy decisions (e.g. reforming the exchange rate or putting the relationship with the IMF on a firmer footing). Such trust had often been cultivated over many years at times when advisors had been ODI Fellows. We noted that Trusted Advisors are likely to be more experienced, undertake short-term periodic visits and less likely to be embedded. They may have played a role in establishing the programme, in validating the demand for work streams, in interpreting the political economy of reform. A personal recommendation from a Trusted Advisor also paved the way for quickly establishing new advisors. Trust has been maintained through a mix of fostering strong personal ties, advisor utility, and appropriate practice over time. Trusted Advisors ‘take our view’, as one budget official noted, ‘when we have donor meetings they sit on our side of the table’. The positioning helps government ‘translate’ the donor relationship (a role also of value to donors if partner relations are strained). There are risks to such a personalized model. Some counterparts can be jealous of Trusted Advisors, particularly if their competencies and access to leadership pose threats to their own status. As trust is the result of a personal relationship it can also be undermined if the patron moves on. BSI has sought to mitigate this risk by cultivating wider relations particularly across finance departments. 3.2.2 The Technician Less experienced advisors, those with general budget and revenue competencies or specialist technical skills (e.g. data base programming, financial modelling) are also central to the BSI model. These embedded generalists and short-term specialists are respected for ‘thinking like us’, by ‘sitting with us’ and ‘working alongside us, as a result of being ‘very hands on’, they ‘understand us better’. BSI systems technicians work on the premise that problems are not well defined, so solutions need to be more iterative and emerge over time, often on the basis of trial and error. One official noted that BSI takes the TOR as the starting point for understanding a problem better, ‘what we want’ and ‘how to help us fix it’. They do not come with ‘readymade (software) solutions’. This is a defining characteristic of the BSI model, distinguishing it from other technical assistance providers (figure 18). Some partners noted little difference between ‘BSI technicians’ and ‘ODI Fellows’, particularly if BSI technicians were younger and less experienced. They believed that one or two of the younger technicians had been technically highly proficient but had not fully grasped the political economy of the instrument; ‘technically intelligent but politically naïve’ as one said. In our opinion BSI country programmes are generally highly attuned to the micro-politics of each intervention. Some advisors think through the implications of the politics while others choose to try and work around them.

34 Williamson, T. Ibid.


3.2.3 The Facilitator and Broker Advisors also need to do things that constitute conventional output-based capacity development (e.g. review codes, deliver trainings, coach teams, draft guidelines or manuals). Most counterparts appreciated the opportunity to learn ‘on the job’, rather than at large off-site training events; the value of BSI’s ‘hand-holding’ approach to building skills and knowledge was mentioned by many. At times, BSI advisors are expected to ‘gap fill’, for instance, to craft a compelling case on their own or to submit a report on time. But on most occasions advisors ‘enlist the voice of others’, usually middle managers from within the budget directorate or revenue authority, and at times across broader policy networks. While BSI advisors may well need to ‘augment capacities’ particularly in those areas where the cost of local expertise is too high to retain in-house, advisors generally aim to empower their counterparts, for instance, encourage the use of credible evidence to substantiate a case, whether a tax policy paper or during a budget allocations presentation and discussion. A priority focus of BSI work is on the use of simple data-tracking processes in order to foster closer ties between people, people that have previously opted to work separately despite the potential benefits of working together. In Sierra Leone, junior officials told us how they were more motivated ‘to go down the road’, to collect data from other ministries or agencies, primarily because the BSI advisor ‘walked with me’, ‘stopping to chat with people on the way’. Most of the government officials we spoke with acknowledged that without the ‘persistence’ and constant ‘prodding and nudging’ by BSI advisors, the computer-based solutions (e.g. web-sites, excel tracking, tools, data management templates, among others) while technically innovative, had limited convening power. 3.2.4 The Dot-Joiner Embedded advisors are well placed to help counterparts identify how different budget or revenue problems interact with each other, although country programme managers are better placed to take advantage of opportunities to build the links and ensure complementarities and synergies. For instance, in Liberia BSI advisors strengthened counterpart linkages by collaborating with ODI Fellows and like-minded AGI advisors at different levels in the ministry hierarchy, helping to land budget briefs or ensure sector budgets cost and prioritise the President’s flagship projects (figure 17). Figure 17: Collaboration with other arms-length TA providers

Finance Ministry Hierarchy Technical Assistance

Minister’s office AGI governance advisor

Deputy Minister BSI budget planning advisor

Assistant Minister/Director ODI macro-fiscal fellow

In Liberia, Sierra Leone, South Sudan and Uganda counterparts recognised BSI advisor support in linking operational issues to wider strategic priorities of World Bank (P4R) and IMF (Article IV) programmes. BSI ability to ‘join the dots’ had at times helped ‘retain a minimum level of order despite all the chaos’, advisors’ ability to make the connections helped ‘stop things from falling apart’, noted one Deputy Minister, whether recovering from catastrophe (such as Ebola) or just keeping the budget process on track during a phase of escalating political violence. Joining up the dots is not always an easy task, particularly when officials are overly focussed on the details and only see the bigger picture in very vague terms, even irrelevant to their daily lives. We noted that technical advisors, embedded and short-term, can also get caught up in the same weeds, either losing sight of the intended outcome or unable to clearly articulate it, hindering their ability to ‘muddle with a purpose’, attempting different routes to the same destination.


Country programme managers are placed to play the ‘dot-joining’ role, but an increased pressure to manage more donor relations can also constrain their ability to keep strategic outcomes in sight. The need to monitor the dynamics of an evolving mix of budget and revenue processes in a short time frame also means a priority can be placed on managing the inputs and not the outcomes. 3.3 How does BSI approach differ from other technical assistance providers? As some donors increasingly look to ‘mainstream PDIA’ through large commercial contracting arrangements, we found the distinction between BSI and other TA to be less marked. BSI was clearly an innovator, but now most TA providers claim to be ‘demand-led’ and ‘problem-driven’. In our opinion, despite blurred distinctions, there remain important differences, particularly between BSI and large private consulting firms (figure 18). Our conclusions were influenced by counterpart insights but also reflect views of the donors and advisors we spoke with. Figure 18: Summary of counterpart views on difference between BSI and other TA providers

• They take our perspective, sit on our side of the table, respect our interests

• More flexible, iterative and adaptable, less focused on outputs (less deterministic)

• More open and collaborative rather than competitive

• Less commercial, less worried about managing operational margins

• Use their learning to influence policy and practice not to support new bids or pipeline work

Some donors believed that large private technical assistance-led consulting firms are so heavily vested in managing their client base that it is hard for them to adopt a ‘partner perspective’. Others were conscious that use of ‘Payment by Results’ contracts had incentivised delivery of tangible outputs rather than more iterative, trial and error-based processes. Outputs were much easier to value, targets and milestones easier to track and measure, such pre-planned results worked against finding any suitable pathways to change, despite a desire ‘to go with the grain’. A key distinction for some concerned ODI’s primary role as a think tank and BSI use of research and practice in order to positively influence aid policy and practice. The g7+ and the research and dissemination components were both indicative of an intention to influence the size and direction of aid flows as well as how TA led programming is undertaken in FCAS. Other donors saw a value in a model that involves ‘multiple small bets’ and fosters change through ‘incrementalism’, but noted that the small size of many of the contracts often meant the donor transaction costs far outweighed the benefits. Although BSI offered a more efficient procurement channel for individual technical advisor placements, and in some cases the trade-off was worth it. Most donors acknowledged that BSI was not originally set up to compete for the large value contracts that are now used to deliver on donor strategies to tackle corruption or build resilience in fragile states. They are ‘more interested in doing their work than in selling it’, one noted. Even BSI board members are seen as a ‘good community of practice’, a body more interested in linking practice to research and vice versa. As an arms-length organisation set up with core funding from DFID, BSI was never designed with a view to bidding for large commercial contracts. In most cases BSI’s was pre-selected by donors who saw value in supporting a more innovative approach, premised more on strengthening government process rather than delivering pre-defined outputs. Yet increased reliance on earmarked donor funding can at times compromise these values.


3.4 How cost-efficiently and cost-effectively has BSI provided its inputs? During the period, BSI supported 17 streams of work.35 In 2015/16 all workstreams met or exceeded expectations.36 Performance was equally impressive in 2016/17 with the exception of 2 work streams in South Sudan, both affected by an escalation in the conflict since July 2016 (figure 19).37 Figure 19: Annual workstream value (£) and independent evaluator (IEV) performance rating (2015/16-2016/17).

Country Workstream 15/16 15/16 Budget

16/17 16/17 Budget BSI IEV BSI IEV

Liberia Budget Transparency A A 140,818 NA NA NA

Department of Budget Planning

NA NA NA A+ A+ 137,500

Tax Policy A+ A+ 127,562 NA NA NA

Natural Resource Tax Administration/Audit

A+ A+ 193,220 A+ A+ 138,000

Concessions Negotiation NA NA NA A++ A++ 77,400

Sierra Leone

Cash Management and Forecasting

A+ A+ ~100,000 A A++ 131,000

South Sudan38

Aid Coordination and Management

A+ A+ 745,000

B B 70,000

Budget A+ A A B 145,000

Local Service Support (Fiscal Transfers)

A++ A+ A A 267,000

Treasury A++ A+ 37,000 A A 158,000

South Sudan Systems Development

NA NA NA A A 157,000

Strategic advice to government

A+ A+ 254,000 A A 199,000

Advice to DFID South Sudan A++ A+ 58,000 NA NA NA

Revenue/Tax Administration A+ A 95,000 A A 141,000

Uganda Budget Transparency Initiative

A A 120,000 A++ A+ 254,660

Improving equity and accountability in financing local government service delivery

A++ A++ 250,000 A++ A+ 914,650

g7+ Support to the Secretariat A+ A+ 231,945 A+ A+ 111,000

The 2015 evaluation noted BSI achieved higher ratings than comparable PFM reform programmes funded by DFID.39 It is our opinion BSI continues to outperform the sector averages.40

35 BSI defines a ‘workstream’ as a technical assistance valued at more than £100,000. Exceptions to this rule were noted in South Sudan, mainly due to BSI in-country arrangements required for managing the large number of smaller work streams as BSI absorbed the work of other technical assistance providers who had left the country earlier due to the conflict. 36 The rationale for the independent evaluator (IEV) scoring is in Annex 4. Workstreams are colour-coded in figure 19 to identify the ones used in the BSI logframes (figure 20). AWRs that are not colour-coded means that they do not count towards the logframe output areas. 37 South Sudan accounts for more than half the total (8) followed by Liberia (3), Uganda (2), Sierra Leone (1) and the g7 secretariat (1). The research and dissemination work undertaken by BSI is not considered a ‘workstream’ and no annual workstream review is written. 38 AWRs for South Sudan are also being assessed as part of DFID’s own annual review. The ratings for the 2015-16 AWRs are similar to those of the April 2016 DFID review. DFID’s review for FY 2016-17 is on-going at the time of writing. 39 Cox, M and Robson, K (2015), ibid. 40 These have been recently estimated by DFID at ‘A’


Table 20: Annual Workstream Reviews used in the logframe as indicators, scoring ‘A’ or above, cumulative 2015/16 and 16/17.

Logframe output areas

AWRs scoring ‘A’ or above

2016/17 target (cumulative) Results

1. Improved budget policy, budget transparency, process capacity and financial management systems in partner Ministries of Finance

4 10

2. Improved management of domestic revenue and external resources by Ministry of Finance

4 6

3. Financial management systems facilitating government led local service delivery

4 4

4. More effective international engagement of fragile states being facilitated through the g7+ Group of Fragile States

2 2

Figure 21: Year-on-year AWR ratings by scores

In the context of a PDIA-led programme, the question of whether BSI is ‘cost-efficient’ or not is less relevant. For PDIA proponents, measures of cost-effectiveness matter more as PDIA is concerned with turning inputs into outcomes rather than just outputs which in themselves hold no guarantee of contributing to behavioural change. That said, process-related efficiencies do matter and in this sense we consider the delivery model to be efficient for the following reasons (figure 22). Figure 22: BSI cost-efficiency drivers

• BSI is able to recruit from a large network of loyal former ODI Fellows. Inputs are often procured at

below market rates for PFM advisors and macro-fiscal specialists. Advisors frequently work in excess of contracted days, particularly when providing remote reach back support.

• Short-term specialists are deployed at very short notice, exactly when needed. Capacity augmentation of this type has demonstrated tangible direct returns to effectiveness not just efficiency (in terms of international influence, budget resource utilization and revenue protection and growth.)

• Close links between BSI and other ODI groups (ODI Fellowship programme, humanitarian and conflict, governance and politics) has enabled efficient access to appropriate resources

1

8

5

0

0

2

5

5

2

0

0 1 2 3 4 5 6 7 8 9

A++ substantially exceeded expectation

A+ moderately exceeded expectation

A met expectation

B did not meet expectation

C substantially did not meet expectation

Number of AWR

2015-16 2016-17


• BSI country support footprint is light. Advisors usually ‘squat in counterpart offices’ rather than rent their own offices, are accommodated in small rather than large hotels, hire local taxis when needed rather than buy or hire their own vehicles.

• Being demand-led and problem-driven means that partner traction around a process is usually good, and use of trusted advisors makes it easier to access and navigate the stakeholder networks

• Close collaboration between BSI and other embedded TA particularly at different levels of the finance ministry hierarchy ensures complementarities and synergies are realized.

• Programme management and direction costs constitute less than 5% on total spend despite higher transaction costs associated with increased requirement to service more smaller country donors.

• BSI adopts a proportional approach in its use of structured programme management tools (results frameworks, risk registers, political economy and conflict analytic tools).

BSI has not developed a set of ‘effectiveness indicators’ against which costs can be measured. There have been some informal efforts among advisors to determine the economic gains achieved as a result of BSI inputs (e.g. saving of £10m in lost tax revenue for an investment of $50,000) or leveraging £200m in ODA as a result of investing £80,000 worth of TA on strengthening revenue and debt forecasting and reporting systems, but sole attribution to BSI advisors is always problematic. If the assumption that efforts to protect and grow the domestic tax base are more likely to produce a monetary return to budget than broad-based budget process, then BSI revenue work streams investments could be construed as highly cost-effective (figure 23), although such analysis does not value the synergies between the two programme components and the less tangible benefits. Figure 23: resource allocation to work stream outcome

The BSI model is premised on an assumption that if workstreams are demand-led, problem-driven, and outcome-focused then they have a greater likelihood of contributing to significant institutional change or limiting the likelihood of losing earlier gains. Our analysis of BSI Stories of Change (see results section) suggests this assumption holds true. There is also some truth in the assumption that programmes funded from core grants are more cost-efficient and cost-effective than those whose funding is earmarked and country-specific, as country programme managers can spend more time managing the strategic relationship with partners (e.g.

706467

1,397

2,434

0

500

1,000

1,500

2,000

2,500

3,000

2015/16 2016/17

Tho

usa

nd

s £

AWR expenditure by outcome

Tax revenue protection and growth Ressource utilisation


‘trawling the design space’ and ‘enhancing the authorising environment’ for reforms) with fewer transaction costs associated with managing the in-country donor relationship. We noted that country donors are less likely to value strategic inputs from BSI central management, considering periodic visits to be a cost rather than a benefit, reducing the potential to improve cost-effectiveness through the provision of ‘critical friendship’ and ‘thought partnership’. An increase in country-based funding also creates fewer opportunities for wider programmatic learning. 3.5 How well has BSI managed uncertainty and risk in fragile states? Since 2015, BSI has prioritised the development of a more robust risk management system. There is greater clarity of risk management responsibility between the BSI Advisory Board, Programme management and Country Programme Managers. Day-to-day responsibility for identification, prioritisation, mitigation and monitoring of risk is largely delegated to those closest to the risk, in most cases Country Programme Managers, although those higher up the delivery chain are also more focussed on risk management. Central management regularly discusses evolving risks with country managers and facilitates structured decision-making with the Board. For instance, recently, after much deliberation and assessment of exit triggers by an expert country panel, ODI decided to close the BSI programme in South Sudan (given the high risk to safety, strategic delivery and as a result of the escalating human rights violations in the country). BSI does not apply a one size fits all approach to risk management. The approach is proportionate to the value, size and context of each country programme. In South Sudan and Uganda, BSI monitor a comprehensive risk register while in Liberia and Sierra Leone the approach is less structured. While BSI has rated overarching output risks ‘high’ in its logical framework, risks inevitably vary significantly by country, with strategic risks being higher in some contexts than in others (e.g. rapid shifts in political appetite for reform, political instability, funding uncertainty, partner and staff churn, among others). In country programmes with a lighter touch approach to risk management (e.g. Liberia and Sierra Leone) the probability of a high impact risk being realised has been less than in countries with robust formal risk management tools in place. Funding uncertainty continues to pose a significant risk to programme continuity (figure 24). The stop start reality of most PDIA type programming coupled with a reduction in core funding, and funding uncertainty more broadly, has serious implications for how BSI manages programmes, particularly the policies and decision-making processes that ensure BSI can either maintain or grow its portfolio or exit without burning the bridges needed to step back in again. Figure 24: Funding uncertainty

The availability of core funding was a foundational principle of arms-length organisations (e.g. BSI, Trade Mark East Africa, the African Governance Initiative), enabling BSI country programmes to be established.41 Core funding afforded BSI the opportunity to demonstrate ‘proof of concept’. Consequently the IMF, World

Bank, ADB, DFID, GIZ and host governments have used their own funding to sustain BSI technical support.

41 Booth, D. (2011), ibid.


3.6 How well does BSI manage for results? We noted that while BSI uses innovative monitoring and reporting tools, the alignment of M&E systems to PDIA principles is weak. While it reports well on results to donors, BSI ability to use M&E tools ‘to crawl the design space’ and to ‘enhance the authorising environment’ is less developed.42 BSI is at times sceptical about the value of comprehensive planning, monitoring and reporting frameworks as epitomised by logframes, particularly in more volatile environment as South Sudan. It believes, rightly in our view (see Annex 6), that rigid results frameworks can do more harm than good, leading technical assistance providers to give primacy to a small number of output led milestones and targets and pay less attention to the outcomes of each work stream intervention. BSI’s overarching logical framework is recognised as more the ‘legacy of funding agreement than a coherent logic model’. At the outset, DFID and BSI management tried to design a results framework that ensured compliance to DFID contracting requirements without compromising PDIA principles. As we noted in the introduction, while the output, outcome and impact statements provide an analytical steer they do not suitably reflect BSI’s theory of change model. Logical frameworks are used in South Sudan and Uganda. Loosely aligned to the overarching framework, they help mediate donor relations, and are revised to reflect annual delivery priorities, although plans can be difficult to forecast, particularly in unpredictable settings such as South Sudan. Teams in Liberia favour ‘outcome mapping’43 planning and reporting processes, these are better suited to PDIA. The Sierra Leone team uses neither, citing lack of client demand and a preference to use scarce government resources on direct delivery. AWR templates are useful for corporate reporting, requiring advisors to describe the ‘situation’, summarise the ‘assistance provided’ and highlight ‘progress against expected results’ but they do not require the adviser to clarify what was actually expected in the first place. In our opinion this is a shortcoming as such outcome ambition and clarification is central to PDIA (annex 4). As interventions usually designed to engage with ‘collective action problems’, the AWRs often fail to flesh out the specific outcome (e.g. to unify bank accounts in different revenue departments within two years). Seldom are outcomes framed in terms of real-time measures (e.g. reduced variance between monthly debt forecast and actual debt). Neither are indicators developed to support the typology of data-driven technical problems we outlined earlier (figure 16). These shortcomings may be the legacy of logical framework aversion or just limited team and stakeholder conversations. In a conventional M&E sense this lack of outcome clarity makes AWRs very difficult to ‘validate’ (see annex 4), and cost-effectiveness almost impossible to determine with any degree of robustness. In a PDIA sense, since the causal logic underlying the specific process is under-articulated, despite being ‘demand-led and ‘flexible’, advisors are likely ‘muddling along’ and being ‘adaptive’ but without a clear sense of purpose, ‘problem-driven’ but not well sighted on a desirable outcome. While BSI continues to see ‘effectiveness monitoring’ as a compliance mechanism rather than as a practical tool with which to ‘trawl the design space’ and enhance the ‘authorising environment,’ it reduces likelihood of effectiveness and limits the ability to demonstrate the effectiveness of the model. We propose a solution to mitigate this effectiveness risk in the recommendations section (page 43).

42 Pritchett, L., Samji, S., & Hammer, J. S. (2013). It's all about MeE: Using Structured Experiential Learning ('e') to crawl the design space. Cambridge: CGD. 43 Jones, H., & Hearn, S. (2009). Outcome mapping: A realistic alternative for planning, monitoring and evaluation. London: ODI.


4 Results This section aims to assess the effectiveness of BSI interventions. We engage with four related questions outlined in the TOR (table 25). To do this we draw on BSI ‘Stories of Change’ (SOC). Engagement with the SOC has also helped interrogate the plausibility of the BSI theory of change.

Figure 25: TOR effectiveness questions

• Has BSI had a catalytic effect?

• Has BSI been able to increase uptake of good practice and promote innovation?

• Has BSI contributed to identifiable institutional change?

• Has BSI had an effect on government strategy and policy on PFM reforms?

According to BSI a ‘Significant Change’ is defined as ‘a significant shift, or series of small interlinked changes in policy discourse, attitudes, behaviours, process or content, contributing to a wider impact at the policy, institutional, financial or service delivery level’. Collectively such changes should bring into focus the agency of BSI advisors, reflecting a causal link between the technical and relational processes advisors engage on and the institutional outcomes and even societal impacts that follow. Our validation of BSI ‘Stories of Change’ suggests that BSI is a highly effective programme. BSI exceeded its SOC target by 100% recording six significant changes against a plan of three (figure 26). Figure 26: Significant changes against logframe targets

Logframe outcome indicators

‘Significant changes’

Target 15/16

Actual 15/16

Target 16/17

Actual 16/17

1. ‘Number of significant changes or prevention of losses in the broader area of fiscal governance in BSI’s partner countries (including work on aid co-ordination, cash management, forecasting and revenue).’

3 6 3 5

2. ‘Number of significant changes in the area of global engagement in fragile states (demonstrated by increased alignment and take up of the principles set out in the New Deal)’

0 1 1 0

In most cases change exists at an individual and institutional level, often the product of small teams being motivated by a mixture of the technical, political and personal: the existence of the necessary authority (the initial authorising space), the technical know-how and personal incentives, often down to the presence of someone whom you trust and know will step up if you can’t. In some cases, the stories reflect a commitment to be better, usually in the sense of integrating good practice, drawing on international PFM norms (e.g. charts of accounts) and even international social norms (e.g. gender equality). Though there are no guarantees that good practice always leads to changing behaviours. In other cases, change has involved teams doing things differently, testing new approaches and then scaling them up, setting more ambitious targets. Yet in some contexts effectiveness might have little to do with change, just a case of maintaining systems, trying not to lose too much of the ground gained earlier, just about managing to stop it all from falling apart.


In our opinion the majority of BSI work streams are very effective. They foster the kind of incremental change which has the potential to be embedded over time and they help keep things on track in contexts where there is a high risk that earlier gains will be lost and systems will break down. 4.1 Has BSI had a catalytic effect? Central to the BSI model is the notion of advisors having a ‘catalytic effect’. Here we noted much evidence of government officials acting on BSI’s technical ideas, of department directors or unit heads championing a new or re-packaged idea to solve an old or emergent problem. BSI has clearly played a motivational role, aiding the understanding of the purpose and helping teams to grasp the proposed technical solution but also instilling them with the confidence to be the change. We noted that BSI has usually been successful at catalysing ideas when it has found the right person to work with on the right problems (ones that are likely to make a real practice change). This has often been a single champion in a newly formed, reinvigorated or re-focussed unit, motivated by pressure from above to quickly find a technical solution and make processes or systems work better. We saw evidence of this in both Liberia and Sierra Leone. A catalytic effect has also been contingent on being granted permission to work in a flexible adaptive manner, for teams to have the authority to continue to ‘trawl the design space’, to undertake practice diagnostics but not spend too much time on initial design and planning work. Terms of reference are loose, allowing advisors to iterate with a core team, adjusting the solution on the basis of collective real-time learning, ensuring buy-in as the trial and error approach plays out. The catalyst has often been down to an ability to ensure alignment of the technical and the political, initially at the strategic level. That is to make clear the contribution a simplified technical solution (usually an excel or web-based data management tool or template) might make in terms of helping finance leadership overcome a problem associated with delivery of a political priority (figure 27). Figure 27: using technical tools to repair reputational damage in Uganda

In Uganda, the Director of Budget first bought into a BSI idea to start up a budget website to disseminate government financial information to the public, for two reasons. First he was convinced by a compelling argument made by a trusted BSI advisor (in terms of fostering better financial transparency and accountability) and secondly because it offered an opportunity to protect the reputation of the finance ministry. He wanted to distance the ministry from line ministries and local government implicated in a large corruption scandal involving misuse of aid. The initiative was effective as It went some way to appeasing parliamentarians, journalists and civic activists but also helped to repair Uganda’s reputation among the international finance community, helping to restore relations and reinstate ODA flows at a key point in the political cycle.

We found that technical solutions are not just more likely to motivate others when there is a political imperative to make an existing system operate better, but when those responsible also have a personal interest in making the solution work. For instance, in Sierra Leone, one of the catalysts for generating buy-in by members of a small team in the office of the Accountant General for a simplified cash forecasting model, was not just the knowledge that better cash forecasting would improve the allocation of sparse cash but that their salaries would be more likely to be paid on time and they would learn a range of excel skills that would enhance their prospects (figure 28). Figure 28: catalysing a simple cash forecasting model in Sierra Leone

BSI advisors, skilled in conceptualizing simple real-time technical solutions to local cash management problems, coached a small team inside the Accountant General’s Department (AGD) to design an excel tool


that could forecast revenue, expenditure and debt on a weekly basis. The team adopted a trial and error approach, adapting the forecasting model as they progressed, ensuring user and senior management support along the way, engaging constructively with a network of data providers, potential ‘veto players’ from within the Ministry of Finance and from outside (the Revenue Authority and the Central Bank). After a year of sustained input, the team produces the cash forecasts on a weekly basis with less external support. It collates reasonably credible data, collected (often cajoled) from the Debt Management Unit, the Central Bank and the Revenue Authority. The team does not always present the data at the weekly Cash Management Committee meeting (as the meeting is not always convened if the Chair is busy) and policy makers still favour a more informal approach to deficit-financing decision-making (usually in a ‘closed circle’). But the results have contributed to timely payment of finance staff salaries and gone some way to reassure the IMF and the wider external financing community that Sierra Leone remains committed to trying to support expenditure control through regular presentation of clearer and better quality cash information.

In some cases teams have been motivated by the simplicity of BSI solutions, excel trackers designed to cut through complexity, tools that hold out the hope of achieving a higher level of systems functionality. We noted this to be particularly true for those officials, often local consultants, working as part of IFMIS and Aid Management teams, often frustrated, tied to screens in dark rooms, not just aiming to produce credible financial reports on time but to use their advanced excel skills on other contracts they were working on, benefiting themselves and the ministry (figure 29). Figure 29: Improved budget policy decisions in Liberia

BSI advisors, excel-skilled in data flow problem analysis, helped Liberia’s budget preparation team design and broker sufficient buy-in to populate a revised fiscal table. The tool enabled the team to better track and assess implications for the allocation of resources to expenditure items with highest perceive need and ensured the impact of policy decisions could be factored into budget expenditure plans. The initiative contributed to more evidence-based analysis of financing options and helped the budget directorate improve their budget policy decisions. While the data has yet to inform budget execution decisions, the tool created a platform for better resource utilisation. Donor confidence in the credibility of the budget data has improved; aid allocation is becoming more efficient and most importantly the government post Ebola recovery programme is still largely on track, instilling other donors with more confidence.

It follows that a key factor undermining the ability to catalyse counterparts is the absence of pressure from above, often as a result of misalignment, competing priorities or conflicting interests. When high-level interest is weak, lower levels of the hierarchy had fewer incentives to act, particularly when they were heavily vested in the current system and conventional way of working. It can be hard to motivate middle managers and junior officials in such settings. Their incentives to remain wedded to complex ‘best practice’ software solutions (with limited functionality) can be strong and even innovative excel solutions can fail to fix real problems. Our evidence of ‘catalytic effects’ was more forthcoming from new departments and units, these are not always ‘greenfield sites’ but they are usually less encumbered by the need to maintain complex technical solutions. 4.2 Has BSI been able to increase uptake of good practice and promote innovation? 4.2.1 Good Enough International Financial Standards In our opinion BSI has contributed to an increased uptake of good practice and promoted innovation. Budget teams in Liberia and Sierra Leone noted the progress made towards aligning their systems to


international PFM standards as a result of BSI support, recognising the need to settle for ‘good enough’ while holding on to the hope of becoming ‘one of the best’. For instance, in Liberia BSI support to cleaning-up of the chart of accounts had helped improve alignment to international public expenditure classification standards (figure 30). There are many other examples of such results (annex 4). Overall, this in-country effort has been reinforced by BSI research, for which the adoption of ‘good enough’ over ‘best practice’ is a central theme. Figure 30: Liberia - standardising charts of accounts in Liberia

BSI technical and soft skills in budget systems diagnostics, problem analysis and facilitation all helped the budget directorate team to adapt the design, clean up and improve the data quality of the finance ministry’s budget coding system. According to budget officials, there are now fewer data inconsistencies in budget classification and chart of accounts, which is more aligned to international public expenditure classification standards. Improved alignment, consistency and credibility of finance data contributed to better budget and expenditure decisions made by technocrats and policy-makers. Budget allocation efficiency improved. Risk of donor attrition is lessened and Liberia is a better placed to finance budget deficits from external sources.

Integrating international social norms on gender In addition to aligning budget systems to international financial reporting norms there is some evidence of aligning budget processes to international social norms and values, for instance by integrating a gender perspective to budget management practice. For example, In Sierra Leone, guidelines published by Public Investment Unit recommended local governments consult women farmers when undertaking infrastructure project proposal assessment process. In Uganda, a local government performance assessment manual published by the Fiscal Decentralisation Unit recommended adding a gender specialist to the assessment team. In Liberia, a Budget Call Circular, published by the Budget Directorate recommended that sector working groups become ‘gender responsive’, ensuring ‘gender equality and equity in all stages of the budget preparation process.44 Equity has also informed sub-national budget transfers in South Sudan and Sierra Leone, integrated to education capitation grants and local government performance assessment criteria in Uganda. 4.2.2 Research uptake There is some evidence that BSI research recommendations influence good practice. A finance leader underscored the value of BSI research on ‘Capacities of Finance Ministries’. It had influenced his approach to inducting a new intake of graduate economists, to pay more attention to improving their practical knowledge of macro-fiscal measures and the importance of IMF fiscal stabilization role. In Sierra Leone, this led the minister to engage in a reflection about restructuring his ministry. We observed that one Budget Director had highlighted key parts of the 2016 BSI policy brief that had used critical theory to assess the distortionary effects of PEFA measurement systems on state behaviour.45 The knowledge influenced his dialogue with visiting IMF missions. The paper helped him understand their thinking. The PEFA research also created the opportunity of an engagement of BSI with the PEFA secretariat, which has commissioned ODI to do some further research. Personal involvement in the research dissemination process has usually been more effective than structured electronic dissemination of research products. For instance, by inviting the Sierra Leone

44 Government of Liberia (2016) Budget Call Circular 1 (BCCI) FY 2017/18, Department of Development Planning (5) November 2016 45 BSI (2016) PEFA, What is it good for? Policy Brief, ODI


Finance Secretary to join an ODI panel discussion and meet UK treasury officials, BSI had a clear influence on Finance Secretary decision-making with regard to organizational development plans. Online access of BSI research outputs has been limited. BSI set a target 12,500 downloads of BSI papers per year but downloads were lower than anticipated, reaching 4,000 only. The Beyond Budgets Blog page was viewed more than 8,000 times in 2015/16, exceeding the target of 6,500 views per year, but downloads decreased dramatically the following year (Figure 31). Figure 31: Research product downloads and blog page views (2015-17)

The loss of the BSI research lead may have had an adverse effect on downloads during the period. In our opinion standalone websites might not be the most effective dissemination strategy especially as there is a requirement to maintain the BSI website (and BSI blog Beyond Budget). We noted that links between the BSI website and its parent ODI website could also be stronger. Given the excellent quality of the research, BSI would have benefited from a well thought through and better-resourced dissemination strategy. While it organized a large number of conferences, workshops and other events where the BSI research products were discussed, there could have been more effort to use other channels; notably more targeted communications with user groups in FCAS, through for instance a targeted mailing lists or utilizing the BSI community of practice. It was also problematic that the BSI website, arguably a key entry-point for people interested in BSI research output, had not been updated since 2015. Furthermore, it was not automatic for visitors to have the reflex to go to the ODI website (were all papers were available) in the absence of an updated BSI website. This may explain the severe drop of downloaded papers since 2015. 4.2.3 From uptake to institutional change ‘Uptake of good practice’ is no guarantee of institutional change. The production of research, guidelines, manuals, or even the alignment of codes of accounts and the revision of budgets may change the formal rules but insufficient to translate into a ‘significant change’ in actual behaviour. Our evidence suggests that such ‘uptake’ (in terms of shifting behaviour) is more likely to be influenced by practice than products. Involving a mix of personal relations and daily interaction, BSI ‘role modelling’ by trusted advisors and competent technicians. For instance, by initially taking the lead on

8205

19782452

3907

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

Beyond Budgets Blog Page view BSI papers downloads

2015/2016 2016/2017


fiscal data needs discussions, whether with members of a budget sector working group in Liberia, a local grant transfer allocation committee in Uganda or a cash forecasting committee in Sierra Leone, BSI had contributed to shifting some of the main incentives that had previously worked against sharing information. As one BSI advisor noted, each knew it would be better to have the data available and collated consistently but none had had the incentive to take the lead. Pressure from above also matters. Such network ties have been engendered where there is real leadership endorsement, through the hierarchy, inter-departmental interaction has been authorised, usually by formal bi-lateral MOU’s between ministries and agencies, reflecting not just leadership commitment but leadership approval to ‘go ahead with my backing’. In the absence of such a blessing it can be easier for some middle managers to revert to conventional technical outputs (e.g. manuals, guidelines, codes), actions that may have short-term incentives (triggering donor payments, pleasing a Minister) but with marginal risk (of not treading on others toes). Yet with all the data collection, collation, entry and analysis involved in regularly feeding these technical templates, small teams can find themselves running around, caught up in the weeds. It is not always easy to remain sighted on the bigger differences they are making. Unit heads acknowledged that they often had no incentive to see the bigger picture because no one had helped them understand what the picture actually looked like. For instance, in Sierra Leone the cash management team did not track the year-on-year variance in finance, revenue and debt forecast, because no one had asked them to. As a result, they did not know that improvements in ‘horizontal accountability’ had contributed to improvements in the accuracy of debt forecasting (figure 32). Figure 32: undocumented story of change: more efficient domestic debt sales in Sierra Leone

During our visit to Sierra Leone, we noted the cash forecasting initiative had produced two unrecorded results. Evidently more frequent personal visits from the people in the AGD’s office (‘the only way we can collect the data’) had helped strengthen ties with the people in National Revenue Authority (NRA). On one level, an MOU mandating regular presentation of finance data to the Cash Management Committee (CMC), and authorising data flows to the AGD’s office, had led senior Revenue Authority officials to feel a sense of greater scrutiny by their parent ministry (‘we are supposed to be ‘semi-autonomous’, said one). On another, regular contact had improved trust between staff in revenue and finance and a willingness to share information. Yet challenges remained: leadership demand for credible cash data remains weak, revenue and expenditure forecasts are distorted by different departmental incentives and the government is still issuing cheques without cash, creating a backlog of unpaid cheques at the central bank. However, year on year analysis of the data table (Feb 2016 versus Feb 2017) indicated a significant reduction in the variance between forecast debt and actual debt. This may suggest higher demand for the debt information, used to inform treasury bond sales, and ultimately make a small contribution to more efficient utilisation of Sierra Leone’s scarce financial resources.

We noted that likelihood of effectiveness had improved when leaders had an appetite for regular progress updates (often the case in National Revenue Authorities but less evident in Budget Directorates) and when there was a shared and strong commitment to the same outcome. Positive results were borne of closer ties between individuals and departments, which had served to strengthen accountability at different points in the finance ministry network and hierarchy.


4.3 Has BSI contributed to identifiable institutional change?

4.3.1 Institutional change In line with the intentions of the PDIA-led model BSI has contributed positively to identifiable institutional change. Some of these changes are outlined above. BSI has realistic ambitions, seeking to foster incremental rather than transformative change. Such change can be interpreted as one in a series of small steps, often small teams cutting their teeth on simple tasks before graduating to more complex ones, even setting more modest targets before aiming for higher ones (figure 33). Figure 33: Prioritising natural resource tax audits for better results in Liberia

The Commissioner General of Liberia’s nascent tax authority, inspired by BSI advisors’ capacity development efforts to improve team skills and knowledge regarding natural resource tax auditing and the principle of risk based auditing, empowered a small natural resource tax unit to audit a selection of established artisanal gold mining firms. The pilot was successful, leading to demands for more than $3m in unpaid tax liabilities from the firms. While the demands are still being contested in the courts, the process built the team’s confidence to take on the bigger Chinese and Indian mining firms and these might yield larger returns in the future.

On other occasions, such incrementalism can be more profound in nature. In Uganda for instance, BSI support is contributing to gradually re-shaping the whole approach to service delivery funding, part of a wider political commitment to fiscal governance reforms (figure 34). Figure 34: Adequate and equitable education and health grant transfers in Uganda

BSI advisors were instrumental in raising awareness in the Ugandan Ministry of Finance on the inadequacy and inequity of budget transfers to local government, hindering equitable and efficient local service delivery and investments. As a result, BSI supported the establishment of a rationalised grant system to local government that ensured that funds are geographically distributed according to objectives criteria of equity and performance. In parallel, the intervention strengthened the accountability of local governments in the use of these funds. The role of BSI has been pivotal in the success of this difficult reform. It facilitated the negotiation of the new World Bank Programme for Results (P4R). The programme opened a window of opportunity as its future funding could be used to ensure that the reform would create no losers, and hence bringing veto-players on board. In addition, besides providing technical support, BSI has played a role of broker within government to ensure that all concerned ministries understood the rationale and implications of the reform, which mitigated the risks for spoilers to emerge. As a result, the reform appears set to be successful as fiscal transfers grant have been redesigned, grant allocated according to the new formulae (with some sectors till pending the P4R signature) and the new accountability system is being rolled out. This is even more noteworthy that similar complex and sensitive reforms have failed in the past in Uganda.

4.3.2 Safeguarding gains Rather than institutional change, some of the results can be more appropriately understood as improved ‘systems maintenance’ (rather than ‘systemic change’). This is particularly true in states affected by conflict, where the intention is to prevent loss, economic, reputational and even personal, with benefits accruing to the state budget as much as to the budget directorate leadership and staff. The rationale being that, on the long run, it is more cost-effective to improve institutions that have not completely unravelled through conflict rather than starting with a blank sheet.


Figure 35: Staying on track in South Sudan In South Sudan, stopping the budget process from going off track, submitting the Budget Book and Budget Speech on time to parliament, during what has been an escalating period of political violence, was not just a significant achievement for the Finance Minister (in terms of his priority to deliver on a constitutional obligation), but the result also had a significant effect on staff morale, and helped the Budget Directorate to retain a sense of professionalism and ‘esprit de corps’. This was particularly important when treasury systems were clearly less resilient and public officials, include finance officers, were increasingly not being paid. On the issue of Fiscal transfer, while conflict and the creation of new sub-national governments undermined the progress made about local government capacity, BSI continuous engagement at the centre enabled the protection of gains. Despite the chaotic situation, officials in the Ministry of Finance are still committed to submit realistic budget. The principles supported by BSI of transferring non-wage grant locally for service delivery, such as capitation grant in schools, is now widely understood on in the Ministry and, for now, unquestioned.

4.4 Has BSI had an effect on government strategy and policy on PFM reforms? There is evidence BSI has had some positive effects on government strategy and policy on PFM and macro-fiscal reforms more broadly from the countries we visited. Our examples draw mainly on Liberia. The policy outcomes are to some extent contingent on a range of complimentary processes. 4.4.1 Influencing counterpart decision-making We noted examples where BSI had made a significant contribution in shifting views and ultimately changing minds. For instance in Liberia, convinced by a compelling presentation made by a junior finance ministers and a BSI advisor that the long-term costs far outweighed the short-term benefits, the Finance Minister and Foreign Minister moved from their earlier position regarding the merits of signing a dual tax treaty with another country (figure 36). In the context of short-term budget deficits, the decision was difficult, requiring a new perspective, one that prioritised the interests of the country and not just the personal networks of those involved in the treaty negotiation. Figure 36: Getting a better deal for Liberia

As part of a wider review of Liberia’s Revenue Code, BSI advisors, embedded in Liberia’s Tax Policy Division, supported the head of the Direct Taxation Unit to improve the team’s tax policy knowledge and analytic skills. Confidence boosted, and with the benefit of higher-level authority, the team reached outside the finance ministry and formed better relations with key stakeholders (Legislative Budget Office, Liberia Revenue Authority, National Investment Commission and representatives from the small community of large taxpayers) ensuring better information sharing and access to tax revenue data. With help from a suite of BSI-designed excel tools, the team tracked review progress, documented policy discussions and undertook tax policy modeling. Having someone they trusted working alongside them helped. The advisor also afforded access to a wider network of international tax policy experts and donors. Better evidence-backed analysis informed timely policy note submissions to Ministers, enabling tax policy amendments to be factored in to the budget preparation process and even influenced cabinet decision-making. As a result, Liberia’s tax policy framework is more aligned to standard WTO policies and ECOWAS Common Expenditure Tariffs. Tax policy revisions discussions helped surface the thorny issue of fee rather than tax income with some incremental results (fuel storage income now goes to the consolidated account, used for the public good, such as maintaining roads, rather than just state owned enterprises ($2m extra tax generated in the 2015/16 fiscal). The Cabinet also decided against signing a double tax treaty, a classic tax avoidance strategy that would have been a very bad deal for Liberia in the long run. While Liberia continues to rely heavily on ODA, tax as a % of GDP is rising.


Although yet to reach tipping point there is also some evidence of a change in mindsets about domestic tax policy, a wider societal viewpoint seems to be shifting a little in Liberia. It is not just the people in the tax policy unit and the revenue authority that have been convinced that diamond exports or fuel import storage costs should be paid as taxes rather than fees, directed to the consolidated account, benefiting public goods such as road maintenance, instead of being paid into the account of private enterprises (used to support operational costs and prop-up inefficient businesses). We noted evidence of a similar shift, albeit still private discourse, in the way officials are talking about the merits of current state policies on fuel subsidies and exchange rates. Frequently contested, macro-economic policy issues are never that easy to resolve, particularly in fragile and conflict affected settings. For instance, in South Sudan, ministers have refused to countenance decreased fuel subsidies and exchange rate reforms were made too late to make any difference. Several Liberian money dealers we spoke with on the streets of Monrovia believed that, in context of a budget allocation and expenditure system that didn’t really work for the people, propping up enterprises that employed large numbers of poor people and keeping everyday fuel prices down was the right thing to do. It ensured a fairer distribution of limited public resources. BSI advice has an impact in contexts where political incentives are closely aligned. For example, some external observers felt that the g7+ secretariat ‘punched above its weight’ at the 5th Global Meeting of the International Dialogue on Peacebuilding and Statebuilding in Stockholm and the Third International Conference on Financing for Development in Addis Ababa as a result of BSI augmenting their capacity at both events. Yet forum Declarations do not always translate into increased flows or ensure donor funds are channelled through government systems, particularly when domestic politics in most OECD countries work against adhering to New Deal principles. Capitalising on windows of opportunity While there has been some BSI influence on OECD ‘PFM strategies and policies’ either through research, specific support to g7+ or country-based engagement, it has not always translated into action. When BSI has made a difference, it is because it had identified windows of opportunity, because political interests were aligned, for instance the World Bank needed to have fragile state representation at their table to help legitimise Fragile State Fund decision-making. While the New Deal principles are embraced by FCAS, the political incentives to operationalise the g7+ agenda can be limited. The priority for governments is often first to manage the donor relation in-country, which may mean New Deal implementation is put on hold, especially as donors do not necessarily feel accountable to the New Deal Principles in country. Nonetheless, the impressive results of the g7+ Secretariat in terms of international engagement demonstrate the value of BSI support to the Secretariat even if country incentives to implement the New Deal are weak.


5 Sustainability This section engages with two related questions from the terms of reference. In so doing we consider the extent to which the results outlined in the previous section will endure. Figure 37: TOR sustainability questions

• Is BSI having a sustainable impact on budget, policies, processes and systems capacity?

• Has BSI increased government capacity and capability sustainably?

5.1 Is BSI having a sustainable impact on budget, policies, processes and systems capacity? BSI interventions on budget, policies, processes and systems capacity are likely to contribute to making BSI impact sustainable. We identified four contributory factors. First, BSI has been able to support reform through a collegial approach to change. Such role modelling and brokering has encouraged veto-player on board and prevented the emergence of spoilers. The Uganda fiscal transfer reform epitomizes this process. While similar reforms have regularly failed in the past, the one BSI supported in Uganda is likely to succeed thanks to its capacity to bring relevant stakeholders, notably line ministries, around the table. In the specific case of Uganda, the World Bank ‘Programme for Results’ (P4R) also offered a window of opportunity by creating the fiscal space for such reform to occur without creating obvious losers. The Uganda experience stands in contrast to Liberia where such planning processes were less successful, derailed by line ministry scepticism about the likely continued absence of fiscal space and the merits of fiscal planning in a context where allocation and expenditure decisions continued to be made by a small presidential circle. Second, demonstration of success can provide the basis for sustainability overtime of BSI support. Budget and revenue officials were of the opinion they would continue to use many of the tools and templates BSI had catalysed. The evidence that newly designed or simplified data sharing processes in Liberia and Sierra Leone will be institutionalised however is flimsy. If BSI exits, some initiatives will likely be taken on by other TA programmes as BSI has done itself in South Sudan, this is not sustainability per se. Shifts in incentives also pose a risk to sustainability. For instance, sustained quality and timeliness of reporting has often been sustained not as a result of just utilising new skills and knowledge but when individuals have an incentive to cooperate with each other around the data. Third, some BSI reforms may endure as they have managed to instil new norms. In South Sudan for instance, despite the extreme volatility of the context, mid-level officials all understood the constitutional importance of submitting the budget and were committed to ensuring that recurrent grants, such as capitation grant for schools, were transferred on time. A similar professional culture was slowly emerging in the Liberia Revenue Authority, instilled by a leader committed to tackling corruption. Such nascent shifts in norms were of course heavily dependent on leadership and just as likely to change if leadership no longer remained committed and incentives changed. Fourth, the sustainable impact of BSI reform is sometimes simply ensured by the fact that the change is difficult to reverse. In Liberia, the revenue code review had put in place some laws that created the conditions for longer-term gains. There was also evidence of a changing policy conversation spurred on to some extent by the legislative review process (regarding the relationship between tax and fees for instance on mineral exports or fuel imports) that would have longer-term implications in terms of


generating revenue to public goods. Similarly, there was some evidence of decision-makers saying no to an international tax deal that that was not in the long-term national interest of the country even if some individuals may have benefitted in the short term. Of course, legal change does not always lead to behavioural change and gains can be lost if leadership changes. 5.2 Has BSI increased government capacity and capability sustainably? BSI has increased capacity and capability mainly through its model of trusted and skilled technical advisors embedded in the bureaucracy. A mix of training and on-the-job application of learning, one-on-one, as well as team coaching, has improved confidence to utilise the variously adapted and simplified tools and systems. Directors and managers we spoke with thought their teams would continue to use new excel (e.g. pivot tables), analytical (e.g. expenditure options modelling) and critical thinking skills (e.g. budget policy options) after BSI advisors had left. By tailoring skills and knowledge to the local context they were becoming less dependent on long-term advisors though they were still needed to embed some of the practices. Yet, for leaders, whether the new technical skills and knowledge would ensure sustained results were contingent on resolving bigger problems: low salaries, unskilled staff, staff churn. For leaders in conflict-affected settings, the problem was exacerbated. Leaders, particularly in budget and planning departments, voiced concern about their ability to attract good staff unless on enhanced, often donor-supported, salaries. This had distorted incentives and created internal divisions. Gap filling or capacity augmentation In some cases, the sustainability question is less relevant. For instance, BSI advisors are often expected to help ‘manage the chaos’, to firefight. Or in the case of the g7+, despite a significant increase of capacity, the Secretariat will always remain a fragile organization because of its limited resources.46 In these settings ‘being one of us’ or ‘working alongside us’ is not just about skill transfer but more often a means of ‘gap filling’ or ‘capacity substitution’, doing the work, particularly if the required skill sets are always in short supply, when they are too costly to buy in locally. It is an open question whether, in the context of a globally inter-connected economic policy environment, it makes a great deal of sense for poor country governments to try and retain all the requisite skills and knowledge in house. While leaders remain politically wedded to a skill and knowledge transfer approach to capacity development, many just want to be able to draw down on trusted advisors with the right specialist skill sets and knowledge at the right time. Such a capacity augmentation strategy has produced enduring high impact returns. For instance, in Liberia, the NIC secretariat members noted immediate financial returns associated with short term technical advice on financial modelling and questioned why they needed to keep an advanced trade tax policy modelling capacity in house when they only really needed the support when bi-lateral trade negotiations were in play. It is highly unlikely that demand for this type of advice will diminish.

6 Conclusion and recommendations Conclusions and recommendations are presented by section. Our recommendations are limited to tackling key value and effectiveness issues at the organizational level, not individual programming.

46 Another factor of fragility for the g7+ Secretariat is the absence of a strong political champion for g7+ initiatives.


Relevance Broadly, BSI’s model remains strategically adaptive and flexible, ensuring strategic objectives are relevant to the shifting priorities of host governments and donors. The g7+ and research and dissemination components are similarly relevant to the programme as a whole; they exhibit coherence at the higher strategic level (in terms of peace and state-building) and add overall value. It is our opinion that BSI’s framing of its outcome level objective (more effective, transparent and accountable PFM policies, processes and systems) is no longer as relevant as it was when the organisational logical framework was first agreed. In the context of limited fiscal space, partner governments are increasingly focussed on improving resource utilisation, and protecting and growing the revenue base, whether from domestic taxes, ODA receipts or other means and it would make more strategic sense if BSI interventions were clearly aligned to this outcome. Recommendation 1 BSI should refine its outcome statement in order to more clearly articulate the link between efficient PFM policies, systems and processes and improved resource utilisation, revenue protection and revenue growth. Such outcome framing is more relevant to the current strategic context and indicative of better fiscal governance (e.g. improved accountability) as well as being more tightly coupled with higher-level service delivery, state and peace building goals. The tighter framing would also create the analytical space for BSI to articulate what the outcome actually looks like in the different country and institutional contexts where it operates. Delivery BSI’s PDIA-led model of technical assistance, focused on utilising technical and relational skills, is well suited to helping partners tackle many PFM problems in fragile state contexts. Annual work-stream progress is impressive given the difficult and volatile operating contexts. The model tackles many of the limitations associated with conventional approaches. The BSI approach is clearly very cost efficient and offers significant advantages and complementarities when compared to larger technical assistance programmes utilising more conventional output-based approaches. Yet it can be difficult to assess how tightly coupled interventions are to outcomes while specific work-stream outcomes remain poorly framed with an overemphasis on process. Closer attention to articulating the desired change informing each work-stream processes will ensure BSI interventions are not just demand-led and problem-driven but also outcome-focussed. The stop start reality of most PDIA type programming coupled with a reduction in core funding, and funding uncertainty more broadly, has serious implications for how BSI manages programmes, particularly the policies and decision-making processes that ensure BSI can either maintain or grow its portfolio or exit without burning the bridges needed to step back in again. There are two related delivery recommendations: Recommendation 2 First, intervention-level outcomes need to be made much clearer if BSI is to remain faithful to its PDIA aspirations. Such a framing process could be led by country managers drawing on country advisory teams, both embedded and periodic. It is our opinion that work streams underpinned by clear outcomes are more likely to produce positive results than those with ambiguous objectives, where outcomes are hard to specify. Outcome clarity will also make it far easier ‘to trawl the design space’ and test whether the particular strategic bet still makes sense, helping allocation decisions.


Such processes are of course contingent on the nature of the authorising environment, on political feasibility. When ambiguity is required to move forward it can be difficult to negotiate outcome clarity and associated effectiveness indicators. But If the political and technical incentives are truly aligned then BSI should use ‘process tracing’ or problem diagnostics to identify, frame and find common ground with their partners on the specific outcome the intervention is focussed on. They should try and work together to discuss and frame a small number of associated indicators of effectiveness, taking into account the politics of the measures not just the framing process. These outcome measures will likely be technical and quantitative (often ratios). They will also be measurable, recognising that it may not be possible to collect data against them at first but finding opportunities to discuss the measures with senior partners, particularly with a view to adapting the design or validating and expanding the authorising space. If possible the desired intervention outcome and effectiveness indicators should be set out at the start of each AWR. As development partnerships often require outcome ambiguity this should be considered as work in progress with outcome clarity considered a key result. Recommendation 3 As BSI operates in increasingly uncertain funding environment it needs to have a risk mitigation policies and triggers in place that enable it to quickly and consistently take decisions with regard to exit, scale-down or scale-up. BSI should draw policy and practice lessons from recent experiences in South Sudan, Sierra Leone and DRC. Such analysis will also have wider value (e.g. as many NGOs are currently operating in similarly unpredictable short-term fragmented funding environments). Results It is our opinion that the BSI model has been very effective at catalysing interest and motivating partners to try out a range of simplified technical solutions to what are often complex collective action problems, encouraging people to work together, in some cases gradually fostering incremental change in institutional behaviours and norms, not just the formal rules but daily practice. In other cases, protecting gains previously made, preventing losses or derailment. These results are nascent but nonetheless impressive. They have been achieved through an effective mix of ensuring alignment of political interest and high level oversight, improving awareness of key issues, iterating simple technical solutions, even redesigning large processes, modelling technical and relational behaviours, and fostering closer ties between people with a common interest in change. Yet BSI can at times be too caught up in the detail and overlook the bigger picture. BSI has taken innovative steps to embed evaluative learning processes in its practice with mixed results. The story of change process has largely been interpreted as a monitoring and reporting tool rather than a ‘Most Significant Change’ process undertaken to enhance and embed effectiveness. Many of the stories do not make the link to higher level results, particularly societal impact in terms of state legitimacy or peace building. Feedback loops to partners remain weak and this has limited potential to enhance the authorising environment. Recommendation 4 BSI would benefit from experimenting with ‘most significant change’ exercises (table 38). As part of this exercise it should revisit the purpose and process associated with its ‘stories of change’ tool’.47 It should also explore means of ensuring tighter leadership feedback loops into workstreams.

47 Davies, R. and Dart, J (2005) The ‘Most Significant Change’ technique, a guide to its use, also see AGI adapted MSC exercises and various organizational adaptations - www.unitar.org or www.click4iit.org

http://www.unitar.org/

http://www.click4iit.org/


Figure 38: Most Significant Change

MSC methodologies are narrative-based in that they would require BSI advisors to craft compelling short stories of change that detail and demonstrate effective engagement on a particular institutional problem (e.g. the tendency to set unrealistic targets, to not share information). These are not long reports listing lots of different activities and results but tighter character-driven narratives demonstrating a link between BSI inputs, processes, outcome and impact, engaging with BSI’s underlying theory of change. The stories are narrower than reports in that they pursue a particular story line rather than try and capture everything the advisor claims to have achieved. Where possible, these short articles draw on effectiveness indicators (rather than outputs) to validate a claim (e.g. the accuracy of debt sales forecasting, or the creation of a Treasury Single Account - see figure 16). The stories are almost journalistic, not unlike Economist articles. The stories are short-listed and scored by BSI country teams using a small set of outcome criteria (e.g. improved resource utilization, protection of gains, prevention of loss, strengthening ties/improving collaboration). The iterative process helps foster wider team understanding and when undertaken with

partners can help expand the authorizing environment and institutionalize practice change.

Sustainability The evidence suggests that, as BSI-led improvements in budget, policies, processes and systems capacities are largely incremental, they are also fragile and nascent. Unlike new rules, sustaining new habits requires daily practice and ongoing support. In this respect capacity augmentation and capacity development models have been complementary, just as the sum of the BSI advisory model and the ODI Fellowship model is greater than the parts. When combined, they do not just offer a highly cost-efficient and effective way of keeping aid players on board and securing international or domestic revenue flows, but also help challenge entrenched practices, encouraging people to work together on common problems, strengthening ties, enhancing accountabilities and fostering better performance, strengthening fiscal governance as a whole. Recommendation 5 BSI should ensure that its theory of change accounts more clearly accounts for the capacity augmentation function. Rather than seek to clarify the similarities and differences between BSI Advisors and ODI Fellows it should document the strategic contexts and practice where Advisors and Fellows have worked together to achieve common outcomes and add overall programme value.

Annexes

Annex 1: Interview List (names available on request)

ODI

ODI-BSI Management 5

BSI Country Directors 4

BSI Advisors48 15

ODI Fellows 2

Multi-lateral and Bi-lateral donor organisations

African Development Bank Liberia 1

DFID South Sudan 1

European Commission Liberia 1

GIZ Liberia 2

IMF (AFRITAC) 2

OECD 1

Sida 1

UNDP 1

World Bank 1

Government partners’ organisations

G7 Secretariat 2

Liberia Finance and Development Planning Ministry 16

Liberia National Revenue Authority 2

Sierra Leone Finance and Development Planning Ministry 35

Sierra Leone National Revenue Authority 2

Other

EU Technical Assistance Providers 3

DFID Technical Assistance Providers 2

DFID/ODI 2013 and 2015 Evaluators 1

Total 98

48 Embedded and short-term, contracted periodic and peers

Annex 2: Bibliography Andrews, M. (2013) Limits to Institutional Reform: Changing Rules for Realistic Solutions, Cambridge University Press,

Andrews, M. L. Pritchett, S. Samji, and M. Woolcock. (2015). Building capability by delivering results: Putting Problem-Driven Iterative Adaptation (PDIA) principles into practice. Paris: OECD.

Barnes-Robinson, K. and Mansour-Ille, D. (2016) Rapid review: why should it be and how could gender be better integrated into BSI’s work. ODI: London.

Booth, D. (2011) Introduction: Working with the grain? Africa power and politics programme, IDS bulletin, 42(2), 1-10;

Booth, D. (2013) Facilitating development: an arm’s length approach to aid. London: ODI.

Booth, D. and Cammack, D. (2013) Governance for development in Africa. London: Zed Books.

Booth, D. & Unsworth, S. (2014) Politically smart, locally-led development. London: ODI.

Cox, M. and Robson, K. (2015) BSI Evaluation, Agulhas Applied Knowledge

Davies, R. and Dart, J. 2005, The Most Significant Change (MSC) Technique: A guide to its use, Trumpington, Cambridge, United Kingdom, and Hastings, Victoria, Australia.

Hearn, S. (2016) Independent Review of the New Deal for Engagement in Fragile States. New York: New York University Center on International Cooperation for the International Dialogue on Peacebuilding and Statebuilding.

Hymowitz, D. (2016) Too much science not enough art, January 2016, Africa Governance Initiative

Hymowitz, D. (2016) Shoulder to Shoulder, Africa Governance Initiative, November 2016

Jones, H., & Hearn, S. (2009). Outcome mapping: A realistic alternative for planning, monitoring and evaluation. London: ODI.

Pritchett, L. Samji, S. & Hammer, J. S. (2013). It's all about MeE: Using Structured Experiential Learning ('e') to crawl the design space. Cambridge: CGD.

Williamson, T. (2015) Change in challenging contexts: how does it happen? Executive Summary, September 2015, London: ODI.

Annex 3: BSI 2015 evaluation recommendations progress report and rating Recommendation BSI Management Comment External Assessment

1=No progress, 2= some overall progress, 3=Good overall progress

1. In its country programmes, BSI should ensure as far as possible that its technical support to budget reforms makes a strategic contribution to wider economic governance issues. This may entail working with a wider range of counterparts beyond finance ministries.

This remains a key guiding principle of all our work. It’s outworking varies across the programme and the specific nature of a technical intervention. It is most evident in South Sudan with the work on Local services where we engage actively with other Ministries. The focus of our engagement in SL is the Accountant General’s office on cash management but this now embraces the rest of the ministry and other key players such as the Central Bank and the revenue authority. We remain open to working more broadly and do so when it is right to do so. It is always a balance between depth and breadth but we fully endorse the principle in this recommendation.

Rating = 2 BSI work clearly helps to tackle key governance issues relevant for peace and state-building, such as better equitable service delivery in Uganda, better resource mobilisation in Liberia etc. However, on some occasions, BSI seems to lose sight of the ultimate problem to be addressed. In the BTI programme in Uganda for instance, the realisation that CSO would need to be engaged occurred quite late. Better PEA (recommendation 3) may help in this regard.

2. BSI should refocus its g7+ work towards the larger strategic challenge of improving the quality of international support to fragile states, using its research and analytical capacity to support the international dialogue by helping to turn New Deal principles into practical proposals for improving aid management and delivery.

This remains the focus of our engagement with the g7+ although the relationship continues to evolve with a transition away from the more intensive engagement in the period 2013-15 to more of a call down relationship. While we have continued (albeit on an ad-hoc basis) to provide support to the Secretariat in Dili (see secondments of Myra in 2016 and Alexandre in 2017) our main engagement has been in supporting the technical work (at key meetings) and support for engagement with WB and the IMF through Alastair McKechnie and Marcus Manuel alongside research on such things as use of country systems. Lisa Denney who operates as our liaison in this area will be able to amplify this.

Rating = 3 BSI has further moved away from logistical support to the g7+ Secretariat and has provided timely and relevant support to the Secretariat to help it articulate fragile states’ interests in high-level international fora, helping the g7+ to become a credible actor. BSI has been active at the country level to push the New Deal agenda but we recognise that this is difficult without undermining its demand-driven philosophy given the lack of country ownership.

3. BSI should make more explicit use of political economy analysis to identify strategic risks and opportunities in

There are good examples of this in South Sudan with extensive political economy insights provided by key team members and discussed extensively within the team at quarterly stock takes. Comparable engagement in Liberia and Sierra Leone is more informal and less explicit in proportion to the programme size.

Rating = 2 Given the embeddedness of TAs in country and their long-standing relationship with beneficiary government officials, PEA was conducted informally. This


partner countries, with a focus on the long-term goal of reducing fragility and conflict.

We continue to review risks on a regular basis and the programme continues to evolve to pick up on new opportunities such as the capability work in Sierra Leone.

recommendation was mainly applicable to South Sudan, where PEA analysis regarding risks was conducted through the External Advisory Board. PEA was also informally conducted through the Strategic Advice work-stream, allowing BSI advisors to provide key macro-economic advice to the government and donors. Yet, the context limited to some extent the space for BSI to reduce fragility.

4. BSI should explore the use of scenario planning and other risk management tools to help manage its engagement in volatile contexts. It should identify the objectives and activities appropriate to different scenarios, to help it to respond nimbly to changes in context. For high risk work streams, it should identify and monitor the minimum conditions required for successful engagement.

The work in South Sudan exemplifies this recommendation. We have an agreed and extensively discussed a set of triggers (based on the political and security context), these are reviewed and discussed by the team on a quarterly basis. We have also conducted a Conflict sensitivity analysis and have convened an external advisory group. The bid submission for the period (set 2016 to June 2017) specifically used scenarios to manage the complexity and uncertainty of operating there.

Rating = 2 South Sudan exemplifies the effort made in devising risk-management tools. However, BSI should make better use of these tools to make quicker decisions in very volatile contexts.

5. Based on its research programme and lessons learned from its country programmes, BSI should work towards a clearer articulation of the challenges of building capacity in low capacity and volatile

BSI produced a lesson-learning piece in September 2015 to articulate the lessons learned over the previous 5 years. The products included a full report and then three shorter and more accessible briefing notes. These have been widely disseminated and very well received. This directly addresses the first part of this recommendation. BSI has also produced a number of relevant research pieces (including https://www.odi.org/publications/10226-sustaining-public-sector-capability-developing-countries and the more general work on Capabilities of Ministries of

Rating = 2 BSI research has indeed intensified its focus on the challenges of capacity building reflecting on its experience. However, BSI TA conversation with counterparts regarding capacity building has remained limited. It must be underlined however that a demand-led, flexible programme makes training

https://www.odi.org/publications/10226-sustaining-public-sector-capability-developing-countries

https://www.odi.org/publications/10226-sustaining-public-sector-capability-developing-countries


environments and develop explicit capacity building strategies for each of its country programmes. It should conclude agreements with its counterparts as to what forms of capacity building it will provide, including training and mentoring for staff, as appropriate.

Finance see https://www.odi.org/publications/10447-pfm-public-finance-management-capabilities-ministries-financehttps://www.odi.org/publications/10447-pfm-public-finance-management-capabilities-ministries-finance ). There remain challenges in developing a more explicit capacity building agreement with our key partners because of the volatility of many of the contexts in which we work. Key impediments cited by BSI partners as undermining attempts at capacity building include: 1. Underqualified staff 2. Extremely poor management of staff 3. Failure to take on board recommendations 4. Rotation of trained staff 5. Departure of trained staff to undertake further study, 6. Failure to restructure ministry 7. Failure to promote and pay staff based on merit 8. Deference to advisors because everything is left to the last minute and must be done NOW.

plans not necessarily useful in a fast-changing environment. In addition, low pay and high turnover in the bureaucracy limits capacity-building efforts.

6. BSI should work towards a clearer understanding of the role of budget transparency in fragile contexts, including identifying what information needs to be communicated to which stakeholders in order to support budget reforms and better economic governance.

Uganda remains the country where we have made most progress on this but we are explicitly addressing this point in the context of a review of the Transparency Initiative. We remain concerned about the take up/accessing of the website and so the questions of what is communicated and to who are of critical importance. Important to note that the publication of basic financial data in South Sudan in the most adverse conditions possible has certainly improved the accessibility of basic information to different stakeholders.

Rating = 1 The result of BTI on increasing accountability is so far limited. In Uganda, the demand-side of transparency (e.g. CSOs) has been disappointing, showing that the ToC must be refined. In South Sudan, BSI had a role in pushing for publication of government data that has been useful to donors and some few local actors. The extent to which this has had an impact on government functioning is however not clear.

7. BSI and its Advisory Board should focus their attention on developing a sustainable funding model for the programme and how to establish a BSI presence in new countries.

Discussions on this subject have been a regular feature of successive Advisory Boards, as you will see both from the Minutes and the Director’s report. While the recommendation is still valid there are two important points to make. Firstly, the funding environment continues to evolve and the ability to secure core funding for BSI remains elusive (non-renewal of DFID funding and concerns over the Sida extension) This is combined with the type of people on the Advisory Board, Policy-based staff located at the centre) who do not have access to or influence over allocation of large sums of core resources.

Rating = 1 While the topic has indeed been central in BSI Advisory Board discussions, the issue of fragmentation of funding has remained an issue. BSI has not managed to mobilise additional core funding or consolidate its funding sources. Actions might include: a) Strengthen ability to present BSI success stories to multi-laterals, bi-laterals and foundations, perhaps tasking a

https://www.odi.org/publications/10447-pfm-public-finance-management-capabilities-ministries-financehttps:/www.odi.org/publications/10447-pfm-public-finance-management-capabilities-ministries-finance






learning and communications manager. b) Adapt SOC tools and produce more short articles that tell a compelling story. c) Continue to explore the benefits of a strategic partnership model with like-minded organisations (e.g. Crown agents), adding value to conventional PFM reform programming or taking a lead technical role in consortia bids. d) Pilot a case of combining the ODI Fellowship model with BSI advisors in certain strategic contexts where such complementarities are relevant.

8. The Advisory Board should appoint a sub-committee or one of its members to look at its own functioning and develop proposals for how to strengthen BSI governance.

This has been actively discussed (see the November 2016) advisory Board. Our working assumption is to look at an annual advisory board meeting in November where we would hope to secure face to face engagement combined with a May set of meetings which would convene relevant stakeholders for each of the constituent programmes. Further discussion and comment through the Evaluation would be of value.

Rating = 1 So far, no tangible measures have been taken on this aspect.

Annex 4: Annual workstream evaluation and scoring Methodology Evaluators’ assessments rely on assessing results against expectations as outlined in BSI logical frameworks, advisory board reports, outcome mapping plans, annual work stream reports, country reports, and counterparts’ views. Validation involved reviewing the narrative based on stakeholder interviews, including counterparts, advisors and donors. The scoring system draws on DFID annual review system. When our scoring differs from BSI’s this is clearly signalled. Figure 39: Limitations to Annual Review Reporting and Validation

• Managers do not always clearly specify expected outputs in advance. The AWR template

does not ask advisors to report against annual expectations.

• Country advisors with programme logical frameworks seldom engage with logical framework indicators (South Sudan and Uganda) and do not always align their AWR narratives to these indicators. Advisors without programme logical frameworks do not tend to identify and track output or effectiveness indicators. This is not a criticism but it does make a DFID-based validation methodology problematic.

• In our opinion too much rigid adherence to logical framework indicator milestones and targets can just incentivise ‘gaming behaviour’, particularly a desire by donors and their contracted delivery agents to always ensure an A or higher score.

• While advisors can directly contribute to process quality improvements, responsibility for the delivery of actual outputs is often not within their gift, particularly if these actions require any degree of partner attitudinal or behavioural shifts.

• The expectations are mostly those of BSI. It is often difficult to validate whether ‘government expectations’ (or indeed ‘societal expectations’) have been met if they have not agreed the planned expectation. Government officials may not always be aware of BSI’s expectations and not all government officials have had the same expectations. This points to wider issues regarding negotiated terms of references.

• Problems, processes and results are not mutually exclusive to work streams, so results should not be attributed to individual advisors but to the programme more broadly. BSI can only make a plausible contribution to partner results, particularly when other technical assistance is or has been present.

• An unavoidable misalignment of the BSI reporting time frame (April-March) with their partner’s fiscal year (Jan-Dec) means that budget planning outputs may be delivered outside the BSI reporting time frame. It can be hard to reconcile this difference in logical frameworks and if logical frameworks are adhered to, can give the (wrong) impression of under-performance.

• A PDIA approach is problem-led and process-based, rather than focused on the delivery of a tangible output, the approach encourages agility and mid-course correction, particularly if technical capacity or political feasibility changes abruptly. This improves the efficiency and likelihood of effectiveness of the BSI model more broadly but does make it difficult for country teams to predict exactly what will be delivered in advance.

• Advisors are also conscious of the reductive nature of the logical framework. Much of what they achieve can often be outside the rigid confines of logical framework, so advisors are rightly encouraged to report on their achievements broadly. Yet advisors are often so


focused on reporting the details of their work, of process tracing and analysis, that they can overlook unexpected higher level results, for instance, improvements in horizontal accountability (between different government entities) as much as vertical accountability (between state and society), or improvements in service delivery as a result of improved oversight of fiscal transfers to sub-national budget spending agencies.

• Outcome attribution is hard, given the influence of many external variables, including, lack of common ground among budget or revenue stakeholders, unpredictable behaviours (legal ratification, budget policy choice, allocation and execution decisions) made by legislative or executive decision-makers, as well as the influence of shifts the macro-economic or social environment influencing preferences and decisions.

Liberia

Budget Transparency (2015/16) Validated: During 2015/16, BSI contributed to two key outputs, a) better quality budget submitted on time, b) budget and planning stakeholders (e.g. sector ministries, departments) collaborate on budget process and systems improvement. BSI met these output expectations by ensuring better communications and closer working relations between budget and planning divisions and line ministries, as a result budget proposals were better aligned to sector strategies. Aid and debt management information was also better integrated to budget. The finance ministry delivered the budget book on time and published it soon after approval. Contributing to improved budget transparency and better resource utilisation. Rating: A (met expectation)

Department of Budget and Development Planning (2016/17) Validated: During 2016/17, BSI built on the previous year’s gains and exceeded expectations, particularly in terms of ensuring quality improvements: gender responsive budgeting, fiscal decentralisation priorities and SDGs were incorporated, not just in the budget but factored into the AFT 11 roadmap. Importantly DBDP have also been better able to track budget support disbursement triggers. These contributed to better resource utilisation, external revenue mobilisation and budget transparency. Rating: A+ (moderately exceeded expectation)

Tax Policy (2015/16) Validated: During 2015/16, BSI primarily through a mix of skill focused capacity building and brokering among key stakeholders contributed to the delivery of a number of process driven outputs, a) Liberia Revenue Code review process (LRC), b) Better collaboration between Liberia Revenue Authority and Revenue and Tax Policy Division, c) Clarification of RTPD structures, functions and operational plans, d) informing budget process with more credible and timely revenue reports, d) enhanced capacities in revenue modeling and tax policy analysis. Rating: A+ (moderately exceeded expectation)

Natural Resource Tax Administration and Audit (2015-16)


Validated: During 2015/16, BSI guided NRTU staff to conduct their first tax risk assessment in the agriculture and mining sectors, to undertake analysis of mineral pricing data, supported the drafting of Liberia Revenue Code regulations, and fostered mining sector stakeholder collaborations. Enhanced NRTU capacities, particularly in tax auditing, and sector stakeholder engagement went some way to ensure better sector compliance to revised mineral tax law. Rating: A+ (moderately exceeded expectation)

Natural Resource Tax Administration and Audit (2016-17) Validated: During 2016/17, tax administration and audit work stream continued to capacitate the new NRTU particularly with a view to a) undertaking live audit risk assessment and tax audits, including transfer price audits, b) monthly meetings between MLME, MFDP and LRA, and c) tax policies and quarterly tax payer meetings in order to improve compliance in the sector, particularly in terms of validating royalty payments, bill issuance to later filers. BSI helped the NRA access the OECD funded Tax Inspector’s Without Borders Programme. These initiatives generated $3M in potential tax revenue and enhanced sector compliance to tax law. Rating: A+ (moderately exceeded expectation)

Concessions Negotiations (2016-17) Validated: During 2016-17, BSI provided the MFDP and IMCC with specialist training and advice on multi-lateral negotiations and financial modeling of projects. As a result, the IMCC was able to negotiate better deals for Liberia, which once ratified will help protect Liberia’s tax revenue base and employment levels in the agricultural sector. As a result of greater dialogue and reporting between the IMCC and the legislature, sector accountability also improved, an unforeseen benefit. Rating: A++ (substantially exceeded expectation)

Sierra Leone

Support to cash management committee (2015-2016) Validated: During 2015/16 BSI directly contributed to several related outputs concerned with the functionality of a cash forecasting mechanism: a) to improve the credibility of a cash forecasting data available to members of the cash management committee, b) better collaboration among members of the wider CMC constituency, c) expanding the authorizing environment to embed the process, a weekly forum convened to assess cash availability. Rating: A+ (moderately exceeded expectation).

Cash management and forecasting (2016-17) Validated: During 2016/17 BSI continued to strengthen the underlying data systems and collaborative processes by which key revenue, debt and expenditure data is gathered in order to support clarity of decision-making. Trying to support expenditure control through presentation of clearer and better-quality cash information. There are positive results, while some of the data remains unreliable and demand is still weak, debt-forecasting accuracy has greatly improved, directly contributing to better utilization of scarce resources. Stronger network data flows have also improved revenue authority


accountability to the cash management committee, an unexpected result, improving ties between the Ministry of Finance and the NRA. Rating: A++ (substantially exceeded expectation). Mark raised from A.

South Sudan

Aid (2015-16) Validated: BSI expectation to improve government coordination of development partners was modest, limited to facilitating better production of aid data and incorporation of aid information in budget reporting. As a result of a mix of capacity substitution (depleted aid coordination department) and capacity development, particularly to enhance functionality of excel and software based aid data collection reporting systems (AIMS, IFMIS), the Aid Coordination Department was able to exceed expectations, better capturing aid data not just in the budget book, but in quarterly expenditure reporting, and debt management reports, improving financial transparency and potential for better resource utilization. This was achieved despite donor reluctance to return to pre-crisis levels of coordination and engagement with government financial systems more broadly. The result highlights the importance of BSI advisor’s collaborative brokering skills, particularly in the context of crisis and even chaos. Rating: A+ (moderately exceeded expectation).

Aid (2016 -17) Validated: Through a mix of capacity substitution and capacity development, BSI sought to re-establish (and embed) some of the key functions of the Aid Coordination Department, particularly the collection, analysis and reporting of aid data in the budget and in-year expenditure reporting. During the year, BSI followed a similar ‘systems maintenance’ brief to 2015/16, with the added challenge of working remotely from Uganda. As a result, the ACD ensured aid data was presented to line ministries during the budget process and aid tables were included in the budget book as well as quarterly budget expenditure reports and macro-fiscal reports. Difficulties with donor engagement and limited ACD staff with excel capacity meant BSI filled many of the gaps, sourcing donor data through remote systems access. Given the context, it seems likely that even these basic results will not be sustained without a high level of ongoing capacity substitution by BSI advisors and there will be little chance of embedding such data reporting habits in the near future. Rating: B (did not meet expectation). Downgraded from A.

Budget (2015-16) Validated: the AWR did not report against the four relevant logical framework indicators for the period. 1) Number of policy documents prepared and published on MOFEP workshop (7/8), 2) number of sectors/clusters supported in coordinating and aligning aid flows (2), 3) frequency of support to national and state level oversight institutions support, 4) number of budget execution reports produced and published. That said, the budget was submitted on time, estimates were more realistic, helping to ensure funding for salaries, (timely salary payment being a key outcome). Quality and timeliness of quarterly budget execution reports improved, demonstrating increased commitment to fiscal accountability, a sound basis for ongoing IMF talks. Despite a very difficult operating environment (departmental leadership change) and departure of a large US technical assistance provider (which necessitated a shift in focus towards data tracking and quality assurance), it is our opinion that expectations were met but not exceeded.


Rating: A (expectation met). Downgraded from A+.

Budget (2016-17) Validated: modest results were achieved in a challenging and deteriorating operating environment. The escalation of violence early in the fiscal year (July 2016) led to the withdrawal of many BSI staff (many government staff also left the country). As a result of BSI remote support, the government was able to deliver on the constitutional requirement to submit a budget book and speech. Although difficulties in financial reconciliation delayed quarterly reporting and growing budget deficits led to larger central bank borrowing, IMF relations remained on track. BSI effort to ensure budget preparation systems problems can be more easily resolved remotely also helped mitigate the risks to future budget expenditure planning and reporting processes. We agree with BSI that expectations, in terms of core systems maintenance, were not fully met. Rating: B (did not meet expectation). Downgraded from A.

South Sudan – Local Service Support (2015-16) Validated: In 2015/2016 BSI provided continuous support to fiscal transfers for service delivery despite an increasingly challenging situation. It assisted the ministry of finance in the design of transfer mechanisms to service delivery facilities, supporting the creation of allocation criteria. It helped the central government to better monitor local government budget performance and use the IFMIS system. In addition, it strengthened local governments though training, which resulted in the improvement of budget formulation and reporting in many states. Overall the transfer to local governments of non-wage resources for service delivery occurred, although slightly below the expectations as set in the logframe. Overall, the output exceeded, but not in a substantive manner, what was expected. Yet, it must be recognized that BSI worked in a challenging environment with the creation of new states and the withdrawal of donors working on sub-national PFM systems. Rating: A+ (moderately exceeded expectation). Downgraded from A++.

South Sudan – Local Service Support (2016-17) Validated: Given the deepening crisis in South Sudan, BSI work has transformed from improving system to protecting the gains already made. BSI had to stop most of its support to local government system, focusing instead in securing the progress made at the central level. This was done notably through further facilitating the coordination of line ministries on service delivery findings, simplifying different PFM tools at the central level and establishing new systems such as new budget formulation and execution procedures and train the relevant staff in their use. BSI also supported the central level in engaging with and training some local government staff. Despite the volatility of the situation, BSI has in our opinion met expectation by continuously engaging with the ministries, managing to protect some gains previously made and facilitating small changes when possible. Rating: A (met expectation).

South Sudan - Treasury (2015-16) Directorate of Accounts Validated: all 2015-16 results were delivered in a one month period at the end of the fiscal year. The expectation was broad, to maintain core systems in the context of crisis. Particularly in terms of providing timely and accurate revenue and expenditure data, without which Article IV discussions with the IMF would be impaired. In this respect the support was successful. IMF relations were kept


on track largely as a result of a treasury advisor providing a mix of capacity substitution (updated oil revenue management files, developed accounting and revenue reconciliation methodologies, among many others) and capacity development (on the job training) to staff in cash management, revenue recording, bank reconciliation, and BSI played a key role leveraging additional donor technical assistance to improve IFMIS functionality in 2016/17. Certainly, BSI helped maintain treasury functions, particularly IFMIS integrity during the short period. In our opinion as expectations were highly ambitious and they were only moderately exceeded. Rating: A+ (moderately exceeded expectation). Downgraded from A++.

South Sudan - Treasury (2016-17) Directorate of Accounts Validated: During the year BSI continued to provide support to ensure the functionality and integrity of a range of complementary core treasury functions: IFMIS, cash position reporting, salary payment controls, transitional financial arrangements, non-oil revenue remittance reports, among others. IFMIS continued to be used for expenditure transactions and the information could be reconciled with revenue and financing data. The work contributed to improving the integrity and timeliness of government financial documents and the pursuit of an effective fiscal strategy instilling the IMF with greater confidence regarding integrity, validity and credibility of data, despite recognized weaknesses in fiscal discipline and macro-economic instability. In our opinion expectations were met in a challenging operating environment. Rating: A (met expectation)

South Sudan Systems Development (2016-17) Validated: During 2016-17 BSI focused on technical solutions needed to improve budget systems functionality, developing tools that served to improve the quality of data capture in IFMIS as well as national and state budgeting processes (Budget and revenue upload tools, journal and expense voucher import tools). The state level budget preparation system was replaced with a more appropriate solution, implemented in excel rather than Microsoft access. The journal voucher improved credibility of IFMIS data. Innovative technical solutions, coupled with capacity augmentation contributed to improvements in the timeliness and quality of financial information. Broadly the work stream met expectations in a challenging operating environment. Rating: A (met expectation)

South Sudan – Strategic Advice (2015-16) Validated: in 2015/2016, BSI provision of strategic advice consisted in “staying at the top of numbers” to provide timely data analysis in the context of a volatile economic environment. This allowed BSI to support the Ministry of Finance in its analysis of forecast, drawing consequences in terms of cash management and reflecting on the structural reforms needed to stabilize the macro-economic situation. BSI contributed to the government’s significant decision to abandon a fixed exchange rate for a “managed” floating rate. In addition, BSI supported the Ministry of Finance in its budget planning. Another pillar of the strategic advice work-stream has been to support senior officials in their interactions with donors, notably the IMF, ahead of the World Bank/IMF Spring Meetings. Overall, despite the difficult situation where political and security concerns have priority over economic/technical ones, BSI has provided in a timely and flexible manner high-quality advice that contributed to some results, such as the transition towards a non-fixed exchange rate. Rating: A+ (moderately exceeded expectation).

South Sudan – Strategic Advice (2016-17)


Validated: in 2016/2017, BSI continued to provide strategic advice despite the increasing volatility of the context. Advice became more about protecting previous gains than making new headways. BSI for instance supported the Ministry in devising an Economic Stabilization Plan. It also helped to “keep the dialogue going” with the IMF in a context where government credibility has been increasingly eroded. It has continued to provided data analysis, helped the Ministry of Finance to make compelling presentations on key issues to governmental stakeholders, and assist in the budgeting process. We feel that in a very challenging environment, BSI met expectations in the support it provided. Rating: A (met expectation).

South Sudan BSI Advice to DFID (2015-16) Validated: BSI provided a combination of regular and on-demand briefings on economic issues to the DFID-SS office on the macro-economic situation drawing on publicly available data. This allowed DFID, in the absence of an in-house economist, to understand the extremely volatile macro-economic situation of the country and the challenges the government faced. BSI work moderately exceeded expectations in terms of outputs as its advice to DFID went beyond the one brief to DFID per quarter initially agreed on in the DFID logframe. Rating: A+ (Moderately exceeded expectation). Downgraded from A++

Revenue (2015-16) Validated: BSI’s ability to support the Revenue Modernisation Steering Committee to maintain momentum on a comprehensive reform plan agenda following the exit of a large US technical assistance provider is a good example of being demand-led and agile. Hence it did not develop indicators or targets in its programme logical framework to support the new work stream. The support enabled the Directorate of Taxation to progress the bill for the creation of a National Revenue Authority (awaiting ministry approval), tackle staff skill and IT system deficits, improve procedural documentation and craft tax revenue enhancement plans. Relations with IMF were also kept on a strong footing, enabling short-term draw down on the IMF trust fund. The support resulted in the submission of a lot of proposals (to government and donors) many of which have yet to be approved. The causal link between these institutional development initiatives and returns to tax revenue or accountability are also still weak. BSI met expectations in terms of maintaining momentum on reform plans but did not really exceed them. Rating: A (met expectation). Downgraded from A+.

Tax Administration (2016-17) Validated: During the 2016-17 fiscal, BSI continued to ‘steady the ship’, driving forward the RMSC led tax reforms by supporting tax policy, ministry engagement with external development partners, and administrative support and capacity building. Improved tax collection efficiencies were realized as a result of support to commercial bank contracting revisions. The taxation amendment act and revenue authority act were ratified. BSI-led donor coordination contributed to accessing regional capacity building initiatives and stronger IMF and AfDB relations. Funds were secured to support the NRA and the Authority moved closer to being established. While causal links to domestic tax revenue growth and accountability are still weak, % of monthly government expenditure financed by central bank debt was under 80%. Expectations were met. Rating: A (met expectation).

Uganda

Budget Transparency Initiative (2015-16)


Validated: During 2015/16, BSI sought to increase uptake of the budget website and budget hotline introduced by BSI in the previous year. Despite limited funding and technical difficulties (which had constrained handover to government as the website had previously been outside government systems), a credible assessment of website traffic, including sessions and downloads suggests that, with limited funding, BSI increased demand for information, particularly from core users: CSO activists and Parliamentarians. Rating: A (met expectation).

Budget Transparency (2016-17) Validated: Increased injections of funding in 2016/17 let to greater efforts to resolve technical functionality difficulties and to improve government feedback to public requests. While progress in increasing public demand and government feedback was noted, challenges remain. An independent review concluded, that while such supply-side initiatives went some way to improve government budget transparency, the impact on service delivery was marginal. This suggests that results were good but perhaps more modest than reported. Rating: A+ (moderately exceeded expectation). Mark downgraded from A++

Improving Equity and Accountability in Financing of Local Government Service Delivery (2015-16) Validated: BSI exceeded expectations as it not only managed to reach its primary objectives to consolidate the different grants to local governments but also supported the redesign of the formula for fiscal transfers and their acceptance among the relevant governmental stakeholders. It also supported the design and rollout of the OTIMS systems to track resource allocation to local governments. The effort of revamping fiscal transfers was paralleled by the design of an accountability system for local governments on budget formulation and execution. Overall, progress substantially exceeded the expectation of the mere consolidation of grants. Such progress was secured through BSI collegial approach to reforms along the way. Rating: A++ (substantially exceeded expectation).

Improving Equity and Accountability in Financing of Local Government Service Delivery (2016-17) Validated: The progress continued in 2016/2017 through the implementation of the systems designed the previous year. Budget formulation was done according to the new grant systems in all sectors except health and education, pending on the World Bank Intergovernmental Fiscal Transfer Program for Results. To ensure that health and education would also allocate grants according to the new formulae, BSI supported the MoFPED in its negotiation with the World Bank. Regarding the accountability system for local government, the manual for evaluation has been created and relevant officials are being trained. The procurement of an independent evaluation by the Office of the Prime Minister is under way. Overall, we feel that BSI moderately, rather than substantially, exceeded expectation as it has gone moderately beyond the log frame objectives of June 2017. Rating: A+ (moderately exceeded expectation). Mark downgraded from A++

g7+ secretariat

g7+ – support to the g7+ secretariat (2015-16)


Validated: BSI support to the Secretariat has been a mix of strategic advice for international meetings, technical and logistical support, research, and capacity building. Regarding international meetings, BSI provided crucial support in the preparation for the World Bank Spring meetings and in helping the Secretariat to make a case to the IFC on the special circumstances faced by fragile states to attract private investment. It also assisted the Secretariat in the preparation of the Third Financing for Development Conference in Addis Ababa and the IDPS meeting in Stockholm, helping the Secretariat to use data compellingly to promote its views. Support to the day-to-day activities of the secretariat and capacity building was ensured by embedding BSI staff. BSI has also produced good quality research for the secretariat. Finally, BSI had also a catalytic effect in supporting the Secretariat to initiate a SDG indicator tracking system. We feel that ODI has moderately exceeded expectation as it has sometimes gone the extra-mile by being responsive on a very short notice (e.g. while preparing the WB annual meetings). The views of external stakeholders on the quality of BSI support, as well as from the beneficiaries are very positive. Rating: A+ (moderately exceeded expectation).

g7+ – support to the g7+ secretariat (2016-17) Validated: BSI support continued to be a mix of strategic advice, technical support, capacity building, and provision of research outputs as described above. To fill a gap in the Secretariat, BSI provided a Political Desk Officer, who has been instrumental in helping the Secretariat to develop the SDGs tracking systems, and also provide through its work on-the-job, learning-by-doing capacity building. The SDGs tracking system has made progress through the selection of indicators and assessment of the capacity for their collection by member countries. More ad-hoc capacity building initiatives have been undertaken, such as the training of the Secretariat’s Communication Officer in London. We feel that ODI has moderately exceeded expectation as the opinion of the beneficiaries on the support provided continues to be glowing. Rating: A+ (moderately exceeded expectation).

Annex 5: Validation and assessment of ‘Stories of Change’ SOC overview rating table

Year Total Not Significant Significant Not Assessed

2015/16 10 3 6 1

2016/17 11 5 5 1

SOC rating table

Programme Year Story of Change Significance

Liberia 2015/16 Validating rubber export prices in LRA Significant

Tax audit risk assessments in the natural resources sectors

Significant

Revenue Significant

Budget preparation and execution

Not significant

2016/17 The Quest for Efficient Tax Administration in Liberia’s Extractive Industry

Significant

Negotiation support in the agriculture sector Significant

Fiscal table Not Significant

Classification of the functions of government in budget (COFOG). Not Significant

Sierra Leone

2016/17 Cash management and forecasting Significant

South Sudan

2015/16 Exchange Rate Reform Significant

Continued commitment to government-led service delivery amidst insecurity, fiscal and political turbulence (Local Services Support)

NA

2016/17 Working Remotely Not significant

Local Services Support

NA

Taxation amendment act 2016 Significant

Uganda 2015/16 Budget Transparency Initiative Not significant

Improving Adequacy, Equity and Accountability in Financing of Local Government Service Delivery In Uganda

Significant

2016/17 Budget Transparency Initiative Not significant

Improving Adequacy, Equity and Accountability in Financing of Local Government Service Delivery In Uganda

Significant

G7+ 2015/16 Including the fragile states’ perspective in the Addis Financing for Development process

Significant

Emerging: tracking the SDGs in fragile states Not significant

2016/17 Tracking the SDGs in fragile states Not significant


Limitations and observations of the tool

• A total of 21 SOCs were assessed. 10 for 15/16 and 11 for 16/17.

• BSI has produced guidelines on ‘outcome mapping’ but has yet to produce similar guidelines for stories of change processes other than definitions and templates.

• We considered stories to be ‘significant’ when there was a clear link to outcome, not just about process. We recognise that as an incremental approach these stories may be unfolding and revisited each year. We also acknowledge that stories are not always about positive change, but often about systems maintenance, retention of gains or prevention of loss. In some cases change may not be on agenda, so fostering demand for change can also be a story of change. Similarly, a story may just focus on a change of practice, shifting an institutional norm engaging on a collective action problem.

• Yet many of the SOCs currently exhibit the qualities of annual reports. The story of change process has been interpreted as a monitoring tool, as a report template rather than as an evaluative learning process seeking to enhance efficiency, effectiveness, impact and sustainability.

• In our opinion a lack of intervention specific effectiveness indicators (real-time outcome measures) inhibits BSI ability to ensure an outcome focused narrative and thus to ensure the story is built around the desired outcome. Many of the stories do not make the link to higher level results, particularly societal impact in terms of state legitimacy or peace building.

• SOCs are currently internal reporting processes so the feedback loop to partners is weak. Yet if partners can be involved, there is greater potential to expand the authorising environment.

Summaries of Individual SOCs

Liberia 2015/16: Validating Export Rubber Prices in LRA (significant) A BSI advisor, acting on a request from the Commissioner General of the Liberia Revenue Authority, worked alongside the head of the Natural Resource Tax Unit and her team building capacity to review a large concession agreement operating in Liberia’s export rubber industry, a critical source of domestic tax revenue. The team helped to design, and were coached to use, an excel spreadsheet in order to track monthly pricing data, sourcing and validating historical rubber prices against latex prices on the international commodities market. They have since maintained the system for five months, with limited remote support from the advisor. While validating prices for one company for five months may seem like a small step, it is the first time that the unit has validated any related-party transaction in the rubber sector (a sector characterized by such familial transactions) and has the potential to produce returns to revenue. The NRTU is now on a clear path towards institutionalizing a rubber concession reporting and compliance regime that addresses transfer pricing problems in the rubber sector. While this reform has yet to yield increased tax revenues, it has motivated the unit to adapt the model to other sectors, including agriculture and timber. And as a result of increased revenue authority scrutiny, there has been an increase in the number of rubber exporting companies filing financial reports, contributing to improvements in natural resource governance. Result: significant

Liberia 2015/16: Natural Resource Sector Tax Audit Risk Assessment (significant) Embedded in the NRTU, a newly created unit with Liberia’s Revenue Authority, a BSI advisor worked alongside the unit manager to recruit and capacitate a new team to improve tax compliance in the mining and agricultural sectors. As a newly created team, the advisor took small steps, training those with limited knowledge of natural resource tax law, clarifying roles and responsibilities and eventually designing an excel framework that could be used to determine tax risk, assessing transfer pricing, and audit history, among other criteria.


Successful completion of the risk assessments led to the advisor facilitating team field trips to audit artisanal gold mining companies. On the job learning through live cases improved team confidence (as well as identifying more than $2m in tax liability). The team is now ready to try their hand at auditing some of the much larger Chinese and Indian mining conglomerates operating in the country. It is still too early to say, but the NRTU is now confident that it is on the right road, and is in a better position to improve tax compliance in the sector, ensuring domestic taxes begin to constitute a higher proportion of the national budget. Result: significant

Liberia 2015-16: Revenue (significant) As part of a wider review of Liberia’s Revenue Code, BSI advisors, embedded in Liberia’s Tax Policy Division, supported the head of the Direct Taxation Unit to improve the team’s tax policy knowledge and analytic skills. Confidence boosted, and with the benefit of higher-level authority, the team reached outside the finance ministry and formed better relations with key stakeholders (Legislative Budget Office, Liberia Revenue Authority, National Investment Commission and representatives from the small community of large taxpayers) ensuring better information sharing and access to tax revenue data. With help from a suite of BSI-designed excel tools, the team tracked review progress, documented policy discussions and undertook tax policy modelling. Having someone they trusted working alongside them helped. The advisor also afforded access to a wider network of international tax policy experts and donors. Better evidence-backed analysis informed timely policy note submissions to Ministers, enabling tax policy amendments to be factored in to the budget preparation process and even influenced cabinet decision-making. As a result, Liberia’s tax policy framework is more aligned to standard WTO policies and ECOWAS Common Expenditure Tariffs. Wider tax policy revisions discussions have helped surface the thorny issue of fee rather than tax income with some incremental results (fuel storage income now goes to the consolidated account, used for the public good, such as maintaining roads, rather than just state owned enterprises ($2m extra tax generated in the 2015/16 fiscal). The Cabinet also decided against signing a double tax treaty, a classic tax avoidance strategy that would have been a very bad deal for Liberia in the long run. While Liberia continues to rely heavily on ODA, tax as a % of GDP is rising. Result: significant

Liberia 2015/16: Budget Preparation (not significant) BSI advisors, deeply knowledgeable of Liberia’s budget context, have worked alongside, ‘as one of us’, the Budget Policy and Coordination Unit (BPCU) team ensuring technical and coordination support located in the operational realities of the recently merged Budget and Development Planning Department (BDPD). Prodding and nudging people from different departments and ministries to talk together, facilitating the identification of shared interests and establishing common ground, has helped ensure Budget Working Group buy-in to a Budget Calendar and assured the quality and timeliness of Budget Options Paper (BOP) submissions. Such initiatives enabled the BPCU to tackle milestone bottlenecks during the strategic phase of the budget, and keep the whole process on track. Timely provision of credible evidence-based advice also helped policy-makers assess the options for closing budget funding gaps and identify public investment and non-discretionary expenditure priorities in the context of heightened budget austerity, precipitated by the Ebola outbreak, the UNMIL drawdown and the global downturn in commodity prices, particularly falling demand for iron ore from China. Result: not significant

Liberia 2016/17: The quest for efficient tax administration in Liberia’s extractive industry (significant) Sustained capacity development support by BSI advisors embedded in the Natural Resource Tax Unit of Liberia’s revenue authority enabled the team to complete two audits of mineral dealers with a tax liability of $3m. The audits were not without consequences, resulting in appeal and push back, not just by mineral dealers and brokers but also government interest groups. The catalyst for government push back was not just the


audit but also the introduction of a binding enforcement mechanisms by the LRA that sought to ensure compliance to the 2% advance payment of corporate income tax payable at the point of export, potentially challenging the policy dominance of the powerful Ministry of Mines Land and Energy. Following successful negotiations, letters of commitment have been signed by some mineral dealers that may constitute a small step, moving away from an (informal) fee based system towards a more institutionalized and transparent corporate tax regime. Helping ensure that all Liberians benefit from the exploitation of natural resources. Result: significant

Liberia 2016/17: Negotiation support in the agriculture sector (significant) Capacity development support to the Revenue and Tax Policy Division on financial modeling led to a request for BSI to tactically expand the focus of its international tax advice, in the mining sector (part of g7 CONNEX programme) to help the RTPD tackle a pressing problem in the agriculture sector. A large multi-national enterprise had sought to renegotiate the terms of its agricultural concession, citing rising costs and falling commodity prices (in the aftermath of Ebola and a 50% reduction global prices for palm oil) and was putting the Inter-Ministerial Concessions Committee negotiation team under pressure to support their requests and conclude negotiations quickly. BSI worked with the negotiation team to source the required financial data and to undertake detailed financial modeling, collaborating with international lawyers to ensure the negotiating team benefitted from timely robust analysis. The advisory team played a key role in helping to resolve conflicting interests among different IMCC members, finding common ground, agreeing red lines and adopting a collective position throughout the protracted negotiation process. Advisors also played a significant role in ensuring a fair negotiation process. The negotiating team ensured the fiscal relief on offer mitigated the high risk of a further decline in domestic tax receipts from agriculture sector (currently only 5% of domestic tax revenues) and avoided setting precedents that could have an adverse effect on the country, resulting in a race to the bottom. A requirement to submit a report to the legislature for ratification, setting out the benefits and costs of the deal (saving more than £40M in potential government costs), went some way to help improve transparency and accountability in the sector (such details are often shrouded in secrecy). The company will continue to invest and operate in Liberia, protecting around 8,000 jobs and the provision of housing, healthcare and education. Result: significant

Liberia 2016/17: Fiscal Table (not significant) BSI advisors, excel-skilled in data flow problem analysis, helped the budget preparation team design and broker sufficient buy-in to populate a fiscal table. The tool enabled the team to better track and assess implications for the allocation of resources to expenditure items with highest perceived need and to ensure the impact of policy decisions were factored into budget expenditure plans. This initiative contributed to more evidence-based analysis of financing options and helped the budget directorate improve budget policy decisions. While the data has yet to inform budget execution decisions, the tool has created a platform for better resource utilisation. This contributed to improved donor confidence in budget data credibility, improved efficiency of aid allocation and ensured the government kept the stabilisation and recovery programme on track. Result: not significant

Liberia 2016/17: Classification of the functions of government in budget (COFOG) (not significant) BSI technical and soft skills, in budget systems diagnostics, problem analysis and facilitation all helped the budget directorate team to adapt the design, clean-up and improve the data quality of the ministries budget coding system.


There are now fewer data inconsistencies in budget classification and chart of accounts, which is now more aligned to international public expenditure classification standards. Improved alignment, consistency and credibility of finance data contributed to better budget and expenditure decisions making by technocrats and policy-makers. Allocation efficiency improved. Risk of donor attrition was also mitigated and Liberia is better placed to finance budget deficit from external sources. Result: not significant

Sierra Leone 2016/17: cash management and forecasting (significant) BSI advisors, skilled in catalysing simple real-time technical solutions to local cash management problems, coached a small team inside the Accountant General’s Department (AGD) to design an excel tool that could forecast revenue, expenditure and debt on a weekly basis. The team adopted a trial and error approach, adapting the forecasting model as they progressed, ensuring user and senior management support along the way, engaging constructively with a network of data providers from within the Ministry of Finance and outside it, from the revenue authority and the central bank. The team is now able to produce the cash forecast on a weekly basis. It collates reasonably credible data, collected (often cajoled) from the Debt Management Unit, the Central Bank and the Revenue Authority. The team do not always present the data at the weekly Cash Management Committee meeting (as the meeting is not always convened if the Chair is busy). Policy makers still favour a more informal approach to deficit-financing decision-making (usually in a ‘closed circle’). The tool has however gone some way to reassure the IMF and the wider external financing community that Sierra Leone remains committed to trying to support expenditure control through institutionalising a mechanism for regular presentation of clearer and better quality cash information. Result: significant

South Sudan 2015/16: Exchange rate reform (significant) Since independence, South Sudan was under a fixed rate regime. Yet as soon as 2012, the fixed exchange rate between the South Sudanese pound (SSP) and the US dollar became a factor of macro-economic instability. As tensions with Sudan created a volatile oil production for South Sudan, and the global oil prices dwindled, the SSP depreciated on the black market. This contributed to worsen the balance of payment, fuelled inflation, and deepened fiscal deficit. In addition, for the few actors having access to currency exchange at the official rates, this was a source of enrichment while further depleting currency reserves. BSI supported the reform of the exchange rate in different manners. First, it facilitated the interaction of the Government with donors, and notably the IMF, to obtain some fresh money to alleviate the effect of the crisis. Second, it assisted the Ministry of Finance in making a compelling case to other South Sudanese stakeholders for a floating rate regime. Third, it provided advice and analysis to senior officials, especially the Minister and the Adviser on Presidential Affairs. As a result, the exchange rate was abandoned in favour of a “managed” floating rate (a rate officially set but frequently revised depending on the market), which contributed to alleviate fiscal and inflationist pressure. Result: significant

South Sudan 2015/16: Continued commitment to government-led service delivery amidst insecurity, fiscal and political turbulence (unable to make a fully informed assessment) BSI support aimed for funds for service delivery to be well managed, accounted for and to reach in a transparent manner the intended beneficiaries in time and in full. To do so, it set to improving local service delivery funding by tackling four main issues. First, the roles of different governmental institutions had to be clarified. Second, systems were to be created or streamlined. Third, inter-governmental discussion needed to be facilitated. Fourth, capacity gaps needed to be identified and addressed. Different activities were undertaken including supporting fiscal transfers to counties and service delivery units, and general system strengthening at both central and local level. As a result, operating expenses were transferred for service delivery, which was hardly the case before. System for budget formulation and execution were slowly put in place in local governments.


The nature of the light touch review of South Sudan Programme means that we were not able to travel there and assess to what extent the BSI new system were actually used by staff. As a result, we are unable to make a fully informed assessment. Result: N/A.

South Sudan 2016/17: Working remotely (not significant) Following the departure of international TA as a result of the escalating violence in July 2016, BSI advisors supported government counterparts to produce a budget remotely. Working from Kampala, in collaboration with ODI Fellows, BSI Advisors were successful in helping to ensure the annual budget process remained on track and was passed by Parliament on time. Remote support to the finance ministry also ensured a successful IMF article IV mission accompanied by bold fiscal measures that could contribute to macroeconomic stability and stronger PFM. There is some evidence that the fiscal deficit is under control as a result of improvement in the money supply, vesting the international financing community with greater confidence in the South Sudan government commitment to make required fiscal governance reforms. As BSI acknowledges, it is inherently difficult to achieve things remotely and BSI clearly made an important contribution to stopping the budget process from being derailed. Although in our opinion it is more of an output rather than an outcome story, one of systems maintenance rather than systems change and therefore, while a good result, was not a significant change. Result: not significant

South Sudan 2016-17: Local Service support (unable to make a fully informed assessment) BSI support for funding local service delivery morphed significantly with the increased instability and violence in South Sudan, the withdrawal of donors operating on PFM, the continuous creation of new sub-national entities, and the collapse of much budgeting process at the national and sub-national levels. BSI intervention sought to re-focus support on the central level through the simplification of PFM tools and training of officials. As a result, despite the very challenging context, the erosion of budgeting systems was limited and BSI continued to engage with central-level officials for capacity building. Non-wage recurrent funding for service delivery was maintained amidst a decline of fund transfer to the local level. The role of BSI of limiting the erosion of gains previously made can hardly be assessed through a light touch review (i.e. with no field visit). It would require talking to beneficiaries, and gaining a better sense of the counterfactuals (what would have happened without BSI support). Result: N/A.

South Sudan 2016/17: ’Taxation amendment act 2016’ (significant) BSI supported the Ministry of Finance and Economic Planning to undertake a major legislative change that could have a significant impact on annual tax revenue. Working closely with the IMF, BSI facilitated the introduction of a high-level revenue modernisation steering committee, developed a tax reform strategy, cabinet papers on tax reform and fiscal stabilisation, and supporting drafting tax amendment act. The tax amendment was subsequently approved and signed by the president. The act has helped reduce the power of ministries other than the ministry of finance to grant tax exemptions and consequently could increase domestic revenue significantly, if successfully implemented. Challenges remain as the conflict and lack of technical support may constrain government capacity to implement the act. A potential example of good form but poor functionality. Result: significant


Uganda 2015/16: Budget Transparency Initiative (not significant) Initially a trusted BSI advisor was instrumental in catalysing the idea for a budget website in 2013 and a toll-free budget hotline in 2014. The timing was right. The Head of the Budget Policy and Evaluation Department championed the idea internally. The Finance Ministry was keen to distance itself from a corruption scandal that had engulfed other spending ministries. The Budget Transparency Initiative was considered one way to tackle public finance information asymmetries between government and citizens (parliamentarians, civil society, media and the public) and restore public faith as well as international donor trust in the Ministry. Finance ministry commitment to regularly post the annual budget online continues to ensure Uganda maintains its high position in international governance rankings for transparency and accountability. The action often contributes to securing external resource flows from the international aid community and is common in other countries. The Budget Transparency Initiative is however a very complex undertaking. While strategically relevant, in terms of encouraging more open budget reporting by government, including fiscal transfers to local government, it remains complex and costly. Technical functionality, initially poor, has now improved. There is evidence of periodic usage by parliamentarians and civil society, though unsurprisingly little ownership by line ministries and local government limiting potential to improve service delivery. Public demand is also low, constrained by cultural norms, access and government responsiveness to information requests. Result: not significant

Uganda 2015/2016: Improving Adequacy, Equity and Accountability in Financing of Local Government Service Delivery in Uganda (significant) BSI advisors were instrumental in raising awareness in the Ugandan Ministry of Finance on the inadequacy of budget transfers to local government, hindering equitable and efficient local service delivery and investments. As a result, BSI supported the establishment a rationalised grant system to local government that ensured that funds are geographically distributed according to objective criteria of equity and performance. In parallel, the intervention strengthened the accountability of local governments in the use of these funds. The role of BSI has been pivotal in the early success of this difficult reform. It facilitated the negotiation of the new World Bank Programme-for-Results (P4R). The programme opened a window of opportunity as its future funding could be used to ensure that the reform would create no losers, and hence bringing veto-players on board. In addition, besides providing technical support, BSI has played a role of broker within government to ensure that all concerned ministries understood the rationale and implications of the reform, which mitigated the risks for spoilers to emerge. The reform appears set to be successful. The fiscal transfers grant system has been redesigned and the new accountability system created. The changes reported are significant as they pave the way to a significant reform towards improving the equity of resources for service delivery in Uganda and there is now a realisation in different Ministries of the need to allocate funding differently Result: significant

Uganda 2016/17: Budget Transparency Initiative (not significant) During the year the BTI benefitted from an injection of additional donor funding that enabled the team to address some of the technical and design problems that had undermined systems functionality, particularly the complex web-site design, the low citizen uptake and poor government feedback. Technical inputs were successful in simplifying the website to ensure improved functionality and outreach was successful in terms of raising awareness. Yet despite additional funding it is beyond the gift of the BTI team to resolve many of the constraints, particularly the low levels of government responsiveness. Moreover the supply-side solution to citizen outreach conducted by apex and locally contracted CSOs is taking BSI into areas that are less strategically relevant. BSI was not established as a ‘voice and accountability‘ initiative. This kind of demand-side initiative that is better suited to CSO-led accountability programmes (E.g. GAPP). Result: not significant


Uganda 2016/2017: Improving Adequacy, Equity and Accountability in Financing of Local Government Service Delivery in Uganda (significant) BSI has managed to secure the gains previously made (see above) and support further the reforms. The systems designed the previous year have been implemented. In 2016/2017, new formulae have been used to allocate resources in most service delivery sectors, with the notable exception of health and to a lesser extent education. BSI also made progress in ensuring that the new allocation criteria will be applied to these sectors as well in the coming year. This is especially made possible by the support BSI provided to the MoFED in the negotiation of the new World Bank Programme-for-Results (P4R). As an additional source of funds, the P4R will indeed provide the political space to rollout the formulae in the health and education sectors by ensuring that the new allocation criteria do not create losers that may resist change in these sectors. On the front of government accountability, it made headways in completing the procedure manuals and training the evaluator, opening the way for an evaluation of local governments, handling of budget formulation and implementation in 2017/18. As a result, BSI made further progress towards the establishment a rationalised grant system to local government that ensured that funds are geographically distributed according to objective criteria of equity and performance. Laudable progress when similarly complex and sensitive reforms have failed in the past in Uganda. If the accountability system is effective, then local citizens may see dividends in terms of better service delivery in health, education and agriculture. Result: significant

g7+ 2015/16: The fragile states’ perspective in the Addis Financing for Development Agreement (significant) The Addis Ababa Third Financing for Development (FfD) conference aimed at finding ways of paying for the ambitious SDGs. The challenge for the g7+ members was to ensure that their special needs in terms of financing were being recognised. BSI managed to both raise awareness in the secretariat on the importance of advocating for fragile states in this international forum and support it in this task. This materialized through different side-events on the challenges of funding development for fragile states, with papers on domestic resource mobilisation, natural resources, and the role of the private sector, the circulation of a concept note and the constant push for reference to fragile to be included in the final declaration. The final declaration explicitly takes “note of the principles set out in the New Deal by the Group of Seven Plus, countries that are, or have been, affected by conflict.” It is difficult to attribute precisely the final result to BSI without clear counterfactuals (e.g. what would have without BSI). In addition, to what extent such reference will have an impact for Fragile States remains to be demonstrated. Yet, the support of BSI was key in terms of providing the relevant research and helping the secretariat to articulate its points clearly in a compelling manner, sensitising the Secretariat on the importance of lobbying ahead of the conference. This was needed in an environment where the group of fragile states was competing with other groups such as the g77 countries or the Small Island Developing States for attention to their needs. Result: significant

g7+ 2015/16 Emerging: tracking the SDGs in fragile states (not significant) The g7+ wanted to set a system to track the SDGs, and especially SDG 16 it was instrumental to get adopted. The rationale of a g7+ system was that tracking all 169 indicators might prove impossible, especially given the weak capacity of g7+ members’ statistical offices, and would make the tracking of fragile state progress elusive. In addition, tracking certain SDGs goals, such as insecurity, in fragile state, might require a different set of indicators depending on country context. BSI support through a series of g7+ meetings had a strong catalytic effect in this process. By sharing the examples of the ODI SDG scorecard, it made the g7+ aware of the kind of simplified tracking option available. BSI was instrumental in helping the g7+ stay focused on identifying potential indicators for tracking SDGs, the availability of data, and, in the end, what indicators could be selected. This intervention has not (yet) produced tangible change in behaviours or use of system. Result: not significant.


g7+ 2016/17: Tracking the SDGs in fragile states (not significant) The progress made since the previous story of change includes the agreement on 20 indicators in November 2016. Their choice was mainly based on BSI research and presentation during a technical meeting in Nairobi. BSI then assisted in the creation of a template for country-member reporting and provided a challenging function in the discussion on the selection of indicators. Since the selection of the indicators, BSI has supported the development of the first g7+ SDG baseline report, engagement with focal points on how reporting would take place and analysed how SDGs have been included in their national plans. The story is potentially a transformative one as, if the SDGs monitoring system is implemented, it will not only allow fragile states to track their progress about SDGs but also demonstrate to the international community their credibility in influencing the international agenda for development, further raising their status on the international scene. Yet for now, progress is real yet very slow, the ownership of the process by g7+ members unclear. Consequently, this does not constitute a transformative change as yet. Result: not significant.

Annex 6: Logframe validation

Narrative Indicator Milestone 2016 Target end March 2017 Achievement

Impact

Improved fiscal governance in partner countries affected by conflict and fragility, leading to more efficient investment and improved service delivery, weakening economic drivers of conflict

(1) Allocative efficiency: Increased government spending on social priorities in countries where BSI has operated continuously, contributing to improved service delivery

No deterioration in partner countries where BSI has continuously operated.

Improvement in one partner country and no deterioration in the other two.

Not achieved: Indicator of spending on priorities deteriorated in both Liberia and South Sudan

(2) A clear budget process with political involvement: An improved budget process engaging line ministries and other stakeholders (including politicians).



Unable to formulate an assessment: No PEFA evaluation existed for South Sudan and we cannot corroborate ODI estimation of a PEFA score. The PEFA evaluation for Liberia decreased from B to D+ on indicator 11 between the 2014 and the 2016 evaluation.

(3) Fiscal discipline: Improved budget planning and execution (decreased variance between approved budget and executed expenditure) in countries where BSI has operated continuously, contributing to government credibility.


Improvement (decrease in variance) in one partner country and no deterioration in the other two.

Unable to formulate an assessment: No PEFA evaluation existed for South Sudan and we cannot corroborate ODI estimation of a PEFA score. The PEFA evaluation for Liberia stayed unchanged to D+ on indicator PI-2 between the 2014 and the 2016 evaluation.


(4) Better international engagement in Fragile states, enabling progress towards the Sustainable Development Goals.

Some progress reported in the New Deal review (that is expected end 2015/early 2016).

Another source of evidence in addition to the one of 2016 relating to: greater global focus on fragile states, great proportion of resources going to fragile states, greater focus on fragile states in specific initiatives, greater alignment of support to the New Deal principles.

Achieved: Evidence include redefinition of Fragile States in the 2016 OECD report, 2016 IDA replenishment favourable to fragile states, adoption of the Stockholm declaration.

Outcome

More effective, transparent and accountable budget policies, processes and systems in partner countries

(1) Number of significant changes or prevention of losses in the broader area of fiscal governance in BSI’s partner countries

3 significant changes in partner countries

3 significant changes in partner countries

Over-achieved: Respectively 6 and 5 significant changes in partner countries were validated for 2015/16 and 2016/17.

(2) Number of significant changes in the area of global engagement in fragile states

0 significant change 1 significant change due to BSI support

Not achieved: No significant change was identified in 2016/17

(3) Lessons learnt: (1) Number of structured occasions for reflection, adaptation and learning within the programmes or across BSI as a whole. (2) number of external lesson-sharing events.

(1) 2 (2) 2

(1) 2 (2) 2

Over-achieved: (1): 4 and 7 occasions were verified for 2015/16 and 2016/17 respectively. (2): 5 and at least 12 events occurred, as verified by us for 2015/16 and 2016/17 respectively.

Outputs

(1) Improved budget policy, budget transparency, process capacity and financial management systems in partner Ministries of Finance

(1.1) Good quality and appropriateness of BSI technical assistance.

Target: 2 AWRs rated A or higher.

4 (cumulative) AWRs rated A or higher.

Over-achieved: 10 AWRs rated A or higher

(1.2) Good quality and responsiveness of BSI technical assistance as assessed by beneficiaries.

All programmes rate us "good" or "very good".


Achieved

(2) Improved management of domestic revenue

(2.1) Good quality and appropriateness of

2 AWRs rated A or higher.


Over-achieved: 6 AWRs rated A or higher


and external resources by partner Ministries of Finance.

BSI technical assistance.

(2.2) Good quality and responsiveness of BSI technical assistance as assessed by beneficiaries.



Achieved

(3) Financial management systems facilitating government led local service delivery

(3.1) Good quality and appropriateness of BSI technical assistance as measured by BSI reporting and externally verified.



Achieved: 4 AWRs rated A or higher

(3.2) Good quality and responsiveness of BSI technical assistance. As assessed by beneficiaries.



Achieved

(4) More effective international engagement of fragile states being facilitated through the g7+ Group of Fragile States

(4.1) Good quality and appropriateness of BSI technical assistance. As measured by BSI reporting and externally verified.



Achieved: 2 AWRs rated A or higher

(4.2) Good quality and responsiveness of BSI technical assistance. As assessed by beneficiaries.



Achieved

(5) Research products on the challenges and successes of strengthening fiscal governance in fragile states inform the implementation of reform initiatives.

(5.1) Number of substantive research papers produced (cumulative).

"2 substantial papers published: Basic enough Budgets, sustaining capability A paper is prepared on the possible contributions of good fiscal governance to peacebuilding and statebuilding. Progress is made on the 5 planned practical Practice Notes

"1 Paper (on possible contributions of good fiscal governance to peacebuilding and statebuilding) is finalised and disseminated. 1 Paper (on “Capability and Competition in Technical Assistance”) is published and disseminated. 5 Practice notes are completed and disseminated.

Achieved: ND: not all papers are on the topic initially decided in the logframe. For the list of papers, see the relevance section.

(5.2) Number of (i) downloads of BSI publications from the ODI website, and (ii) visitors to the beyondbudgets blog site, demonstrating

(i) BSI Content Downloads from the ODI website: 12,500 per year (ii) Beyond Budgets Blog Page views: 6,500 per year

(i) BSI Content Downloads from the ODI website: 12,500 per year (ii) Beyond Budgets Blog Page views: 6,500 per year

Not achieved In 2015/16 and 2016/17: (i) BSI Content Downloads from the ODI website: 1,978 and 3,907 respectively.


dissemination of research.

(ii) Beyond budgets Blog views: 8,205 (achieving the target) and 2,452 respectively.

Annex 8: ToR

Terms of Reference for an Evaluation

of the Budget Strengthening Initiative (BSI)

Introduction

The Budget Strengthening Initiative49 (BSI) provides advice and support to partner Governments in

fragile states on a demand led basis to assist with the development of financial and budgetary

systems, and to promote more effective international engagement in these areas through the g7+

process. It also aims to support the management of domestic revenue and external aid and to

promote budget transparency. The aim of BSI is to be responsive and flexible to meet urgent needs

and to overcome bottlenecks to development. It is anticipated that inputs provided by BSI will

support existing efforts by Governments and donors and will help to promote changes in attitudes,

behaviour, policy or the systems used by organisations.

Objectives

This evaluation will examine the achievements of the BSI programme between April 2015 and March

2017, building on previous reviews and will independently validate the internal assessments carried

out so far (Stories of Change and Annual Workstream Reports) in the period covered. There is a

specific requirement to assess progress against the objectives set out in the agreed log frame (Annex

2).

The evaluator will therefore examine the inputs, outputs and outcomes of the programme to date.

The revised theory of change (TOC) underlying the programme will also be tested to determine

whether the assumptions about cause and effect underlying the project actually hold in practice.

(S)he will recommend revisions to the programme design and implementation in future phases of

the programme; and will make a judgement on the value of the implementation model even as it

continues to adapt to the demands and requirements in fragile states.

The results of the evaluation and the review of the BSI model will add to the growing literature on

PFM reform and the design of initiatives to support them.

Scope

The evaluator will be expected to review the progress of the programme as a whole and in relation

to specific programmes of support to individual countries and the g7+ group of fragile states, and

the research programme.

The evaluation is designed to assess the efficiency and effectiveness of the BSI programme (between

2015 and 2017) including whether inputs were provided cost effectively and outputs were delivered.

49 See www.budgetstrengthening.org for more details

http://www.budgetstrengthening.org/


The evaluation should test the BSI theory of change in relation to other approaches by other donors

and programmes to public financial management reform and capacity building. This should include a

synthesis of evidence from recent evaluations and reviews (e.g. from the Joint Evaluation of PFM led

by the AfDB50) and interviews with donors operating in the countries where BSI currently works. The

comparison will enable the evaluator to assess the level of innovation in the BSI approach and

whether these innovations help to overcome shortcomings in traditional donor programming.

Key Aspects of the Evaluation

The evaluation should focus firstly on validating the evidence of achievement against the outputs

and indicators set out in the individual partner monitoring frameworks and the overall log frame for

BSI as a whole, in the period April 2015 to March 2017.

Secondly, it will extend the analysis of programme performance using the latest evidence from BSI

monitoring and reporting systems through in-depth interviews at the country and g7+ programme

level. Thirdly, on the basis of the evidence produced, the evaluation will consider the implications for

the revised theory of change (TOC) underlying the BSI programme. This is attached as Annex 1. Any

lessons and areas for improvement in the approach or the TOC itself will also be identified.

As part of the third component the evaluator will focus on the following key questions derived from

the OECD DAC evaluation criteria51 and discuss with the BSI management team how best to gather

the evidence to make an assessment.

50 https://www.afdb.org/en/news-and-events/article/joint-evaluation-of-public-finance-management-reforms-importance-of-policies-design-and-coordination-9703/ 51 http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm

1. Is BSI providing high quality, relevant and cost effective advice?

2. How well has the programme managed uncertainty and risk in fragile states?

How efficiently and cost-effectively has BSI provided its inputs?

3. Are inputs provided by BSI contributing to identifiable institutional changes?

4. Is BSI having an effect on Government strategy and policy on PFM reforms?

5. How demand-led has BSI been and is the quality of its relationships with counterparts

conductive to effective assistance?

6. Has BSI had a catalytic effect?

7. Is BSI having a sustainable impact on budget, policies, processes and systems capacity?

8. Has BSI increased uptake of good practice and promoted innovation?

9. Has BSI increased partner Government capacity and capability sustainably?

https://www.afdb.org/en/news-and-events/article/joint-evaluation-of-public-finance-management-reforms-importance-of-policies-design-and-coordination-9703/

https://www.afdb.org/en/news-and-events/article/joint-evaluation-of-public-finance-management-reforms-importance-of-policies-design-and-coordination-9703/

http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm


The evaluator will provide:

an overall assessment of BSI, including its overall strategy and approach, management arrangements

and funds utilisation;

an assessment of BSI’s relevance and contribution to peacebuilding and state-building goals under

the New Deal.

a review of the Sierra Leone and Liberia country programmes, including their achievements against

their objectives and monitoring frameworks, testing the main linkages in the BSI theory of

change;

a light review of the g7+ programme against the logframe and theory of change and an assessment

of its contribution to strengthening the g7+ secretariat and capacity for engaging with

international partners and to strengthening the implementation of the New Deal at country

level;

a light review the Research & dissemination component, including its relevance, quality and uptake.

a light review of the South Sudan and Uganda country programmes’s achievements against their

logframes and theory of change.

The main requirement is to assess BSI’s overall effectiveness and value for money in achieving its

planned results. It is also expected that any general lessons will be drawn out and specific

recommendations for improvement in programme design or delivery will be made.

Finally, the evaluation should put forward recommendations for future phases of BSI.

Method

The evaluator will initially examine project reports, Annual reviews and Annual Workstream reports,

stories of change and self-assessments by project staff as well as feedback from partner

Governments. This will be done through a desk-based review of BSI programme documentation held

in London and meetings with BSI staff in London.

As part of this initial process, the evaluator will examine any trends in the data collected against

milestone indicators in the BSI log frame. The range and relevance of the indicators for assessing

programme achievements and impact and the time frames for data collection will also be

considered. The evaluator will also select examples of specific interventions by BSI for further

investigation which will cover the programmes in Liberia and Sierra Leone. The aim should be to

conduct a deeper examination of these cases and to explore the key evaluation questions with

individual contacts identified with the help of BSI programme managers. Methods for this will

include:

• Telephone interviews

• E mail exchanges

• Questionnaire surveys

10. In what ways is the support provided by BSI distinguishable from other providers of TA

support?


Arrangements will be made for the evaluator to visit Liberia and Sierra Leone to enable face to face

structured interviews with partner Governments, other donors and stakeholders. The review of the

g7 + work, of the Research & Dissemination component and of activities in South Sudan and Uganda

will be based around desk reviews and remote interviews.

In undertaking the country programme analysis, the evaluator should conduct interviews with the

other donors involved in budget strengthening and PFM programmes and develop a good

understanding of the alternative approaches being used in that country context. BSI staff will be able

to support the setting up of meetings with development partners in Liberia and Sierra Leone. The

evaluator will consider whether these interviews need to be complemented by a questionnaire sent

to a wider group of interested stakeholders.

Finally, the evaluator should examine the extent and manner in which the programme has taken on

board the recommendations of the 2015 Evaluation. The overall evaluation will be undertaken in

accordance with the OECD-DAC evaluation criteria and meet the associated quality standards.

Inputs, Timing and Budget

Overall the evaluation is expected to require up to the equivalent of 64 person days including 2 visits

to Liberia and Sierra Leone.

The evaluation will take place in the February - June 2017 period and will be expected to take place

over a period of six weeks of elapsed time. A draft report must be available by end May 2017 and

the final report must be available by end June 2017 to inform the closure of the Sida core grant to

BSI.

The financial proposal for this assignment must not exceed £45,000 (lump sum contract).

Reporting

The evaluation will be commissioned by the BSI Advisory Board who will oversee and review the

evaluation, and will approve the final report. It is anticipated that the evaluation will be of interest

to a wider PFM community involving partner Governments, PFM practitioners and others. Further

dissemination of the report will be considered by ODI PFI.

Deliverables

The evaluator will provide a report of 30 pages with an executive summary and supported by

annexes as required. This will be presented in an agreed format and in draft to the BSI Programme

Director and then completed after comments are received. The report will be formally submitted to

the Advisory Board and will be published online by ODI/BSI.

A list of recommendations for BSI and for its Advisory Board will be proposed to increase efficiency,

effectiveness and impact of the programme amongst other things. The evaluators may also be

required to make a presentation or to provide a concise statement about the evaluation for

communication purposes internally and externally.

REPORT ENDS

Documents

BSI evaluation report 2017 · MoU Memorandum of Understanding MTEF ... BSI also has a Research and Dissemination programme, ... Hext Consulting Ltd BSI Evaluation Report 2017 July