TCR Validation Report December 2020

Regional: Action on Climate Change in South Asia

Reference Number: TCRV-2020-026 Project Number: 46470-001 TA Number: 8572

ABBREVIATIONS

ADB – Asian Development Bank APEC – Asia–Pacific Economic Cooperation CCA – climate change assessment CPS – country partnership strategy DMC – developing member country DMF – design and monitoring framework GHG – greenhouse gas IED – Independent Evaluation Department RFI – results framework indicator RS-GIS – remote sensing—geographic information system SAARC – South Asia Association for Regional Cooperation SARD – South Asia Department TA – technical assistance TAR – technical assistance report TCR – technical assistance completion report TCRV – technical assistance completion report validation

NOTE

In this report, “$” refers to United States dollars.

Director General Deputy Director General Director Evaluator

Marvin Taylor-Dormond, Independent Evaluation Department (IED) Véronique Salze-Lozac’h, IED Nathan Subramaniam, Sector and Project Division (IESP) Alfredo Baño-Leal, Evaluation Specialist, IESP

The guidelines formally adopted by the Independent Evaluation Department (IED) on avoiding conflict of interest in its independent evaluations were observed in the preparation of this report. To the knowledge of IED management, there were no conflicts of interest of the persons preparing, reviewing, or approving this report. The final ratings are the ratings of IED and may or may not coincide with those originally proposed by the consultants engaged for this report. In preparing any evaluation report, or by making any designation of or reference to a particular territory or geographic area in this document, IED does not intend to make any judgments as to the legal or other status of any territory or area.

TECHNICAL ASSISTANCE COMPLETION REPORT VALIDATION REPORT1 1. PROJECT DATA TA No. 8572

TA Name Action on Climate Change in South Asia

Approval Date

13 Dec 2013 Approved ($)

1,325,000

Signing Date – Revised ($) 3,375,000 Country Regional Planned

Completion Date

12 Dec 2016 Disbursed ($)

2,788,635

Actual Completion Date

14 Mar 2019 Undisbursed ($)

586,365

Department South Asia Department

TA Type

TRTA ( ) KSTA ( ) PATA ( ) CDTA () RDTA ( ) PPTA ( ) RETA ( )

Sources of funding

TASF-V, TASF-VI, TASF-Other sources

Sector and Subsector

Energy Executing agency

Asian Development Bank

2 . DESIGN AND MONITORING FRAMEWORK AND RESULTS

Objective The technical assistance (TA) aimed to help selected developing member countries (DMCs)2 in South Asia to successfully transition to a low-carbon and climate-resilient development path by integrating climate change mitigation and adaptation in project designs and sector development plans. As envisaged under the TA, this would be done through improved screening of investment projects against climate risks and capacity building activities.

TA Rationale South Asia is projected to experience extreme weather conditions and climate change. This is expected to result in significant economic losses in key sectors, such as agriculture, coastal and marine, energy, health, tourism, transport, and water, rendering growth targets harder to achieve. The regional capacity development TA proposed to help South Asian DMCs mainstream climate change mitigation and adaptation through better understanding of climate change issues, and integrate climate adaptation and mitigation in project designs and sector development plans through enhanced institutional capacity.

Results Levels Indicators3 IED Comment on Evaluability of Indicators Outcome South Asian DMCs have increased

a. South Asia Department’s (SARD) approvals representing both adaptation and mitigation projects achieve the 60% result

Evaluable. Outcome indicators were updated during the four TA amendment requests, of which, there were two extension requests of

1 Team members: C. Ramos-Galacgac (Quality Reviewer), P. Deeks (Initial Reviewer), and D. Menezes (Validator

Consultant). 2 The TA report originally envisaged working in six South Asian DMCs: Bangladesh, Bhutan, India, Maldives, Nepal,

and Sri Lanka. However, during TA 8572’s implementation, India and Nepal approved country-specific TA projects with capacity development activities similar to those planned under the regional TA. Upon consultation with these country TA projects, TA 8572 limited its scope of activities in India and Nepal to coordination and knowledge sharing activities (e.g., participation in regional workshops or conferences). Asian Development Bank (ADB). 2013. Technical Assistance for Action on Climate Change in South Asia. Manila.

3 The indicators reflected here are based from the TA completion report’s (TCR) design and monitoring framework (DMF) reflecting approval of revisions in the DMF as contained in various memoranda dated 17 September 2014, 14 December 2015, 10 December 2016, and 14 November 2017. ADB. 2020. Technical Assistance Completion Report: Action on Climate Cange in South Asia. Manila.

2

Results Levels Indicators3 IED Comment on Evaluability of Indicators investment and institutional capacity to manage the impacts of climate change

framework indicator (RFI) target by 2019 (Baseline: 52% in 2012)

b. Government officials in priority

sectors have improved climate change project development and management capacity based on pre- and post-assessment exercises by 2019 (Baseline: Initial information available)

c. Relevant government agencies adopt the guidelines for climate risk screening and emission tracking system (Baseline: Pilot adoption by 2019)

implementation period in the December 2015 and November 2017 memos. In the December 2015 memo, the outcome indicator 1a was changed to “SARD approvals representing both adaptation and mitigation projects stay above the 45% RFI target by 2016 (baseline: 52% in 2012).” However, this indicator was subsequently returned to its original formulation in the November 2016 memo. There was no clear explanation for this change. The target end date for indicator 1b was amended from 2016 to 2019, and a new indicator 1c was added. All outcome-level indicators had baselines. The indicators clearly set out the expected results with measurable targets; and there was a clear hierarchy of results from outputs to outcomes. The third outcome-level indicator related to the adoption of the outputs by government, relates to both the existing and expanded scope of work from the extensions. In themselves, the indicators were realistic and achievable in the context of what the TA intended to deliver; and were verifiable, largely deriving from project progress reporting and supporting documentation. Indicator 1a was clearly articulated and the final version remained unchanged from that originally articulated in the DMF. However, as noted above, the indicator was changed twice. The target was changed from 60% to 45% (second amendment, December 2015); but later on returned this to 60% with a final delivey date from 2016 to 2018 in the succeeding memo (third amendment, November 2016). The final delivery date was once again changed to 2019 (fourth amendment, November 2017). The target date for indicator 1b was revised to 2019, following the extension approvals. This indicator was considered limiting as it focused on immediate pre- and post-training self-assessments of individuals only, and at a given time period, and did not adequately reflect the TA’s contribution to strengthening institutional capacity (Outcome statement) at the end of project.

3

Results Levels Indicators3 IED Comment on Evaluability of Indicators During the first amendment, a third indicator (c) was added to the original DMF. This was a good addition to the results framework and showed the ambition of what the TA intended to achieve with the time and cost extensions.

Output 1 Screening of investment projects against climate risks and for mitigation potential strengthened

1a. General framework for climate screening (risk and potentials) finalized and pilot-tested by 2017 (Baseline: Not applicable [NA])

1b. Set of climate change

indicators for all key sectors (in support of SARD RFIs) identified and compiled by 2018 (Baseline: NA)

1c. 100% of infrastructure projects

undergo risk assessment by 2018 (Baseline: NA)

1d. Framework for low-carbon

projects developed and tested by 2018 (Baseline: NA)

The output-level indicators were specific, measurable, realistic, timebound, and attributable. During the amendment requests, the addition of a new indicator (1d) was relevant to the expanded TA scope. The target dates for the existing output indicators were also amended to reflect the project timeframe extension.

Output 2 Capacity of DMCs in developing and/or implementing climate change strategies and action plans strengthened

2a. Supplementary institutional capacity and training needs assessments (TNAs) conducted for each participating DMC by 2017 (Baseline: Initial information available)

2b. Six capacity building

workshops conducted by 2018 (Note: Number of workshops per country may vary depending on TNAs and capacity mapping results; minimum of one per DMC; Baseline: Initial information available)

2c. Three country climate

assessment and sector-specific road maps for climate resilience and low-carbon growth prepared by 2018 (Baseline: NA)

As above, the indicators were specific, measurable, realistic, timebound, and attributable to the TA. During the amendments, two changes were made to the indicators as follows: Indicator 2a—the DMF noted six TNAs to be conducted. However, the revised indicator did not provide any disaggregated number by DMC, although footnote 1 of the TA completion report (TCR) noted that capacity measures for India and Nepal were dropped as similar activities were planned under another regional TA, and the indicator required a minimum of one per DMC. All target dates were also amended from earlier dates in the original DMF to reflect the project extension.

4

3. PERFORMANCE ASSESSMENT Relevance

Item Highly Relevant Relevant

Less than Relevant Irrelevant

TCR Rating TCRV Rating IED Rationale This validation finds the TA to be highly relevant in terms of its objective, as well as

through stages of identification, design, and implementation. While at completion, the TCR did not clearly identify any contextual shifts or changes in partner priorities that might have affected the adoption of these outputs, recent updates show that the four DMCs have taken steps toward integrating TA outputs in their country systems. The TA is assessed as adding transformational value as it rapidly and efficiently carried out climate change risk assessments through an innovative and automated geographic information system (GIS) web-based tool for preparing assessment reports, which SARD exclusively uses but could be expanded to the other regional departments. The TA has also developed climate change risk atlas for Bangladesh, Maldives, and Sri Lanka, which are being integrated into their country systems of climate change and disaster risk screening for project appraisal and public investment management systems. The TA further triggered some important knowledge partnerships with the Asia-Pacific Economic Cooperation (APEC) Climate Center and the World Climate Research Programme’s Coordinated Regional Climate Downscaling (CORDEX) South Asia. There were four budget increases and two timeline extension requests over the TA’s lifetime, largely arising (i) from priorities identified during national consultations and workshops at inception (first amendment); (ii) in response to emerging requests during implementation from Bangladesh, Bhutan, Maldives, and Sri Lanka (second and third amendments); 4 and (iii) to support Asian Development Bank’s (ADB) mainstreaming of climate adaptation and mitigation across its portfolio (fourth amendment). Each of the amendment requests provided clear justification and noted that these were “minor changes in TA scope, implementation arrangements, and extension of TA completion date.” As the scope of work was expanded under existing outputs, the justifications noted that these would not substantially affect the TA’s outcome, components, benefits, and other implementation arrangements. The SARD Director General approved the TA’s first extension request for completion date to 31 July 2018 on 14 December 2015. The Vice President for Operations 1 approved changes in TA scope and implementation arrangements on 14 October 2014, 21 December 2015, 11 November 2016, and 1 December 2017. Details of the TA amendments are as follows: (i) The first amendment requested additional financing of $750,000 from Technical

Assistance Special Fund (TASF)-V for the expansion of output 1 activities related to developing a climate change database platform and risk screening system, and the design of a greenhouse gas (GHG) mitigation framework for key economic sectors. It also proposed to expand its climate change and disaster risk screening activities for SARD’s pipeline projects. Furthermore, output 2 capacity enhancement activities were proposed to be expanded after priority

4 The TCR noted that the regional TA 8572 was to be originally implemented in the six South Asia DMCs. However,

during implementation, India and Nepal approved country-specific TA projects with capacity development activities similar to those planned under the regional TA. Following consultation with these country TA projects, TA 8572 limited its geographic coverage of activities in India and Nepal to coordination and knowledge sharing activities (e.g., participation in regional workshops or conferences).

5

training needs were identified during inception. The TA also proposed to design capacity building activities in parallel with output 1 activities.

(ii) The second amendment accommodated requests from Bhutan and Maldives governments for priority activities that involved utilization of uncommitted funds, and made an additional request to increase the TA budget by $300,0000 from other TASF resources and a 19-month time extension. In terms of value added, the work was also envisaged to help increase support to ADB in identifying, mainstreaming adaptation, and promoting cleaner and more energy-efficient technologies in all sectors, thus, adding value to its existing efforts. In addition to its existing approach to informing climate issues at the project conceptualization stages, the extension proposed to extend its support to strategically incorporate the climate agenda through reviewing and informing the country partnership strategies (CPSs), and the country-specific climate change assessments (CCAs) were seen to provide the entry points.

(iii) The third amendment ($500,000 from TASF-V) requested expanding the scope to carry out work in response to requests from the Bangladesh government and to continue support to CPSs.

(iv) The fourth and final amendment noted that existing commitments stood at 98.8% of the budget and requested time extension to December 2018, and a $500,000 increase in TA amount from TASF-VI to finalize the SARD remote sensing (RS)-GIS and collect data for Bhutan, India, and Nepal; support CPSs; and finalize the South Asia Association for Regional Cooperation (SAARC)–supported regional study.

The TA was relevant to and aligned with the ADB’s commitment to double its annual climate finance from $3 billion to $6 billion by 2020 and to support identifying projects, mainstreaming adaptation, and promoting cleaner and more energy-efficient technologies in all sectors. This TA was linked to ADB’s strategic objectives in South Asia and was, therefore, deemed an important mechanism in contributing to these broader objectives. The TA was not specifically linked to the delivery of any one particular ADB investment, but its activities have focused on strengthening recipient capacity in select priority areas and have informed the design of several pipeline investments. The TA design and implementation were responsive to the developmental needs of the four DMCs5 and to the region, and was aligned with ADB’s CPSs for each of the four DMCs. The TA also undertook early in-depth consultations with each of the DMCs to identify their priority areas that were confirmed in TA implementation follow-up meetings, and the design was, therefore, considered to be relevant and responsive to stakeholder needs. The revised DMF maintained the link between the outputs and outcomes, and largely reflected in the scope of work. Although the ensuing increase in budget was substantial, this did not substantially change the focus of the results chain or indicators. The most significant DMF update was in December 2014 (first amendment) that included the third outcome indicator related to the governments’ adoption of guidelines for climate risk screening and emission tracking system. Subsequent annual revisions to the DMF were minor and reflected the revised end of project dates.

5 The regional workshop undertook initial stakeholder consultations with the six DMCs during which the following areas

of support were identified: (i) climate change risk assessment, (ii) technology transfer and demonstration projects, (iii) capacity building, and (iv) knowledge development and sharing. Held in Manila in March 2013, this workshop was conducted under SARD’s earlier program Climate Action South Asia (and not under TA-8572-REG).

6

The validation finds, however, that the outcome-level indicator on assessing improved capacity through pre- and post-workshop assessment questionnaires was insufficient to assess increased capacity or how that led to changed practice in the medium term. The DMF revisions did not include output-level indicators on regional relevance, even though funds were allocated for certain regional activities (e.g. SAARC study). However, these revisions are reflected in the DMF’s updated Activities section. The TCR noted that flexibility was built into the design of output 2 so that the TA could respond (within its scope) to emerging requests from South Asian DMCs for support on climate change–related priority capacity building activities. Design also took account of management arrangements, including the establishment of a TA coordination unit in Manila, with national consultants based in-country to facilitate better coordination. The TA’s operational modality was appropriate as this was intended to provide technical and capacity building services to DMCs, drawing on both international and national expertise. It also provided an important function of liaising with national partner agencies.

Effectiveness

Item Highly Effective Effective Less than Effective Ineffective

TCR Rating TCRV Rating Evidence of Outputs Achieved

Output 1 delivered well against its intended objective of strengthening the screening of investment projects against climate risks. Most notably, the TA developed project screening tools and methodologies, and helped incorporate climate adaptation and mitigation into project design. Some notable deliverables as per the TCR included (i) 195 pipeline projects were screened between 2014 and 2018 for climate risks

and another 65 proposed initiatives reviewed for climate change; (ii) Preliminary climate risk and vulnerability assessments of project design brought

valuable information that helped to identify climate risks that could affect project life span and efficacy, and recommended potential adaptation options to address significant climate risks;

(iii) A framework for estimating potential GHG emissions at the project level was drafted, including guidelines for identifying appropriate climate change mitigation strategies, but was yet to be piloted.

(iv) Decision support tools like the SARD’s Climate and Disaster Risk Resilience Decision Support System, a first of its kind RS-GIS was developed.6 This tool automates the preparation of CCA reports for projects, making the process faster, cost-effective, and more efficient. The system includes relevant national and subnational data collected from Bangladesh, Maldives, and Sri Lanka.

Output 2: The TA made good progress in delivering targeted activities that were responsive to the capacity needs of the DMCs. Some examples included (i) in-country and regional consultation workshops were conducted to identify and

prioritize capacity needs, and discuss design issues related to the project activities (inception) through final workshops.

(ii) 10 capacity building events were delivered (exceeding DMF targets of six) based on training needs assessments, with pre- and post-training questionnaires. However, there seem to be no capacity building activity for Maldives (refer

6 SARD’s Climate and Disaster Risk Resilience Decision Support System. https://sardtracking.adb.org.

7

Appendix 1 of the TCR) whereas the indicator states a requirement of a minimum of one per DMC;

(iii) Climate assessments and resilience plans were prepared for Bangladesh, Maldives, and Sri Lanka for strengthening each country’s climate change strategies and action plans; and a vehicle emission reduction road map and medium-term strategy was developed for Bhutan.

(iv) The TA prepared and finalized 21 concept notes and technical reports for Bangladesh (5), Bhutan (7), Maldives (5), and Sri Lanka (4). The DMCs endorsed these through inception workshops.

TA also supported the finalization of a SAARC study, “Climate Risks in the SAARC Region: Ways to address the Social, Economic and Environmental Challenges,” listed as an activity in the DMF.

Evidence of Outcomes Achieved

The TCR noted that the outcome was mostly achieved—with two of three indicators met—and the validation is in broad agreement with this assessment. The first outcome indicator (a) was exceeded as 74% of SARD projects supporting climate actions were approved in 2019 (per TCR reference to SARD annual report). The second outcome indicator (b) was substantially achieved. In 2017 and 2018, government officials attended capacity building workshops conducted in the participating South Asian DMCs on key topics relevant to the regional TA’s in-country sector activities. The third outcome indicator (c) was mostly achieved. By end of 2018, the TCR noted that Bangladesh, Maldives, and Sri Lanka governments were planning to integrate a climate change and disaster risk screening system;and Bhutan government, an emission tracking system into project appraisal and public investment management systems. Recent updates show that Bangladesh is already implementing an online disaster risk information platform, Maldives has incorporated the TA’s developed risk atlas into the country’s national GIS, and Sri Lanka has undertaken steps toward mainstreaming climate change objectives in policy and planning, although progress has stalled due to management changes at the Ministry of Mahaweli Development and Environment after national elections.

IED Rationale This TA was well targeted to address wider developmental constraints in the four participating DMCs, particularly in terms of developing specialized technical and policy-relevant knowledge and tools intended to strengthen DMC capacity on climate change, train individuals to build their capacity, and develop measures to improve ADB project screening for climate risks and strengthening project design for climate adaptation and mitigation. The TCR and supporting documents highlighted evidence of the TA project’s effectiveness. The TA substantially achieved its results at the output levels that in turn contributed to the results at the outcome level. Both outputs were delivered within the TA project’s final revised budget and time frame, following the four TA budget increases and time extensions that were approved. This validation finds the TA to be effective. For outcome indicator 1a, the TCR did not provide a sector breakdown of this project information. For outcome indicator 1b, the TCR referred to attendance and pre- and post-training self-reporting questionnaires to indicate substantial achievement on improved government capacity. However, there is limited evidence of how the capacity building workshops are helping to shift behavior or leading to change in the medium term (i.e., beyond the immediate delivery of the workshops). An important change in the results framework related to the inclusion of outcome indicator 1c, “Relevant government agencies adopt the guidelines for climate risk screening and emission tracking system (Baseline: Pilot adoption by 2019).” This was a valuable addition to the original DMF as it showed the potential TA’s utility and

8

uptake for partners. The TCR noted intent of the governments to adopt climate risk screening systems through this TA’s support provided, but there was no evidence that this had progressed at the time of drafting the TCR, nor was there an explanation of why this did not happen across any of the four DMCs, or any forward recommendation to track this beyond the TA’s life. This was an important part of the results chain and a prerequisite to achieving the outcome of increasing investment and institutional capacity of DMCs to manage the long-term impacts of climate change. Therefore, the extent to which the adoption of these tools would potentially help to increase institutional capacity of DMCs could not be fully established under this validation exercise. The TCR also had limited discussion of what went well and what constraints hindered progress of the indicators. It would have been helpful for the TCR to quantify some of the outputs, such as total numbers of people trained by country and across the main sectoral priorities identified in the TA report (TAR),7 to provide a more comprehensive picture of the scale of effort and levels of partner participation. It is positive to see reference in the TCR of the dissemination of findings in international events to raise awareness. The main challenge going forward is for ADB teams to track the four DMCs’ uptake and use of the knowledge products and systems, and their continued use in investment pipelines.

Efficiency

Item Highly Efficient Efficient Less than Efficient Inefficient

TCR Rating TCRV Rating IED Rationale

This validation supports the TCR rating of efficient. The TA’s socioeconomic value had been demonstrated through the capacity building and knowledge support provided in response to the DMCs’ needs of strengthening capacity for climate-resilient development. For instance, a summary assessment of Bhutan’s motor vehicle training that drew on pre- and post-training questionnaires, noted that the training was useful, gender inclusive, and well targeted to the specific needs of the country. This validation notes that the TA sought four amendmends including four budget increases and two-time extensions resulting in a total budget increased more than 2.5 times. The TCR could have clearly justified these substantial increases as there is a risk of extensions being perceived as due to cost overruns or budget underestimation at appraisal. However, drawing on information from other supporting documents, this validation understands that, instead of going for a higher budget at design stage, the TA adopted an incremental approach of seeking additional funding as country’s opportunities arose, which is a more conservative approach. Climate change is complex, and output 2 was designed to be flexible and responsive to emerging stated demand as these priorities surfaced during consultations. The TA responded to government requests accordingly leading to four requests for additional funding. This validation also notes that for each request, output 1 was opportunistic as it not only expanded the numbers of SARD pipeline projects screened, but was responsive to ADB's new corporate target in 2015 (i.e., which arose during project

7 ADB. 2013. Technical Assistance for Action on Climate Change in South Asia. Manila.

9

implementation) to provide support to CPSs and conduct country-specific assessments.8 TCR’s Annex 2 contained information on budgets and spending across years. For each extension request, the TA adjusted its budget—some subheads were reduced, and some were increased—suggesting that the TA attempted to reallocate its resources efficiently. TCR noted that savings resulted due to a scale back of substantial activities planned for India and Nepal as these were covered under another regional TA, and a slight delay in finalizing consultant technical reports such that these could not be published, leaving unintended savings under miscellaneous TA administration. Most of the outputs and results were achieved within the final revised budget (83% utilization rate), following the four increases (to a total budget of 255% the original approved amount) and agreed expansion in the TA’s scope. This validation notes that despite substantial budgetary increases over the TA’s lifetime, the outputs and outcomes were marginally adjusted. Outcome indicator 3c and output 1d were added for the first amendment. For subsequent amendments, all additional activities remained within the ambit of the existing outputs and in line with the overall outcome statement of strengthening participating South Asian DMC capacities. The project could have included additional country-specific indicators, particularly for output 2, in the course of these revisions. These could have drawn from each of the country-level activities that had concept notes that were developed with government oversight and agreement. The TCR noted that consultants were engaged in accordance with ADB’s use of consultants guidelines, disbursements under the TA followed ADB’s TA disbursement handbook; and, all equipment were procured according to ADB’s procurement guidelines. This validation positively notes that the governments of the participating South Asia DMCs provided in-kind contribution, such as office space, use of facilities, counterpart staff, related administrative expenses, relevant reports, data and information, and other in-kind contributions. This is consistent with what was originally planned during design and specified in the TAR’s Appendix 2. This validation also notes the efficiency gains demonstrated through the SARD RS-GIS-based tool that enabled preparation of the initial CCA in less than 1 hour, compared to the manual version that normally took up to 3 days. Through the successful pilots for screening of projects, the TA showed that such tools could potentially make CCA processes faster, cost-effective, and more efficient, bringing efficiency gains into ADB operations. There were no reported significant delays or process inefficiencies in implementation. The TCR’s rationale for the overall increase in TA budget and for the two-time extensions (24 months) was that this enabled the TA to accommodate country-specific approval processes and small implementation-related delays without significantly altering the scope of work. The additional funds came from other TA budgets. This validation also positively notes that this TA built on a screening framework previously developed under regional TA for Regional Climate Projects

8 In September 2015, ADB announced that the corporate target for climate financing would be doubled to $6 billion by

2020. Of this, $4 billion would be dedicated to mitigation through scaling up support to renewable energy, energy efficiency, sustainable transport, and building smart cities; and $2 billion would be for adaptation through more resilient infrastructure, climate-smart agriculture, and better preparation for climate-related disasters.

10

Consortium and Data Facility in Asia and the Pacific,9 thus showing efficiency gains rather than duplicating efforts. A good example of efficiency is that the India and Nepal components, originally planned under this TA, were reduced in scope as these were taken forward under other country-level TA work thus avoiding duplication of effort. In terms of management efficiency, SARD was the TA executing agency with support from a TA coordination unit of mainly Manila-based consultants, with national consultants based locally in countries. The TA report also noted that provision was made for counterpart contributions from the governments of the participating South Asian DMCs in the form of office space, use of facilities, counterpart staff, related administration expenses, and other in-kind contributions. Project documents noted that the TA coordination unit would be expected to liaise with other relevant ADB units and/or offices, particularly the resident missions in Bangladesh, India, Nepal, and Sri Lanka; South Asia Regional Cooperation and Operations Coordination Division (SAOD) of SARD; and relevant divisions in ADB’s Regional and Sustainable Development Department, Economics and Research Department, and Office of Cofinancing Operations. This arrangement was effective in practice; and involved procuring and managing over 127 person-months of international expertise and 245 person-months of national consultants. The TCR positively noted that SARD liaised with national focal government agencies to build ownership.

Criterion Weight Rating Value Weighted Rating Relevance 0.35 3 1.05 Effectiveness 0.35 2 0.70 Efficiency 0.30 2 0.60 Overall Assessment (weighted average of above criteria)10

2.35

Overall Rating

Item Highly Successful Successful Less than

Successful Unsuccessful TCR Rating TCRV Rating IED Rationale

This validation assesses the TA overall performance, with a weighted average score of 2.35, highly successful. The TA was aligned with ADB’s and DMCs’ strategies, and its results chain was appropriate. The TA was transformational as it efficiently carried out climate change risk assessments through SARD’s exclusive and innovative GIS web-based tool for preparing assessment reports, which could be expanded to the other regional departments. The TA has also developed climate change risk atlas for some DMCs and triggered some important knowledge partnerships with the APEC Climate Center and the World Climate Research Programme. The required four budget additions and two-time extensions expanded the achievement of the TA objectives, as the TA design expected added activities based on DMCs’ requests. This validation recognizes that changes to the TA were justified and that the additional sets of activities that arose following requests from

9 ADB. 2013. Technical Assistance for Regional Climate Projections Consortium and Data Facility in Asia and the

Pacific. Manila. 10 Each sub-rating is assigned the numerical value assigned to each rating: highly relevant = 3, relevant = 2, less than

relevant = 1, and irrelevant = 0. The compound criteria for performance ratings: highly successful (overall weighted average greater than 2.30), successful (overall weighted average greater than or equal to 1.65 and less than or equal to 2.30), less than successful (overall weighted average greater than or equal to 0.75 and less than 1.65), and unsuccessful (overall weighted average is less than 0.75).

11

DMCs for climate-change related capacity building activities remained within the scope of both outputs, and in line with the project’s outcome and the ADB’s strategic priorities. The TA was well delivered, and there was evidence that the TA has contributed to strengthening ADB’s pipeline investments and internal systems. This validation recognizes that there are several notable results achieved and potential future effectiveness subject to DMCs’ full uptake of TA outputs. The choice of TA was appropriate, and the TA achieved two of three outcome indicators and six of seven output indicators. By preparing frameworks for climate risk screening, and developing a SARD RS-GIS decision support system, the TA increased the effectiveness of risk assessments of pipeline projects. The TCR noted that the governments were planning to integrate a climate change and disaster risk screening system into project appraisal and public investment management systems in 2018. Recent updates from 2020 show that further progress has been achieved and risk atlas maps have been integrated in government systems of Bangladesh, Maldives, and Sri Lanka. The TCR noted that the execution modality of providing a venue to interact and share knowledge and experience in each participating DMC strengthened strategic partnerships among concerned national government agencies, private sector, development partners, and other relevant stakeholders. As this is a regional TA, the TCR could have included some aspects of the regionality. It could have, for instance, discussed the effectiveness of the partnerships established through this TA with SAARC and the APEC Climate Centre, and indirect spill-overs from regional knowledge exchange events that could potentially have generated value and cross-learning for regional participants, including those from India and Nepal.

4. SUSTAINABILITY

Item Highly Likely Likely Less Likely Unlikely NA TCR Rating TCRV Rating IED Rationale This validation is in broad agreement with the TCR findings that the TA is likely

sustainable, with some additional observations. Several TA elements are likely to be sustainable particularly those related to streamlining of mandatory climate risk assessment processes for approval of SARD pipeline projects. For example, the GIS-based Climate and Disaster Risk Resilience Decision Support System—a first of its kind—is now operational and being maintained by SARD. The TCR positively noted the potential replicability and upscaling of this system to other DMCs and regions, and the potential benefits of an automated system in improving efficiency and timeliness in preparing climate risk screening (CCAs). However, this will largely depend on the robustness and quality of future data collection processes which will need to be financed beyond the TA. Another TA for Sustaining Climate and Disaster Risk Resilient and Low Carbon Development in South Asia—supports the addition of relevant data and information from Bhutan, India, and Nepal (up to 31 October 2020). However, this system should not rely on grant funds to continue operating in the long term as this may be a risk to its sustainability. The TCR acknowledged that the knowledge partnership with the APEC Climate Center helped provide reliable data and assistance to improve national and/or regional databases for the SARD RS-GIS based tool. There is no further information on the specific value that this partnership brings, and whether it will keep operating beyond the TA. This is an internal tool only for SARD.

12

The TCR suggested that as the technical outputs were developed in response to country priorities, DMCs were likely to considere these their planning and operations, and especially where these are linked to ADB-financed operations. There is no obvious shift in context or changes in political priorities that could limit or disincentivize progress already achieved through the TA.

Lessons Learned (1–3 implementation, 4–7development results, 8 others)

Criteria TCR Self-Assessment Comment 1. Design and/or

planning The TA enjoyed the benefits of engaging with national strategic partners and stakeholders to strengthen mainstreaming climate change concerns in national development planning. In particular, the TA’s output 2, which was designed to be open and flexible, helped accommodate the South Asian DMCs’ request for support for climate change–related priority capacity building activities that can be implemented within the scope of the regional TA.

The TCR identified two findings, not lessons: firstl, engagement with strategic national partners was beneficial, and second, flexibility in output 2’s design enabled the TA to respond to national stakeholder priorities during implementation through consultative workshops with national stakeholders. Ensuring that design of the TA aligns with the ADB’s strategic regional efforts is important in helping to contribute to longer-term sustainable results. The DMFs could cross-reference these areas of alignment. In finalizing this validation report, SAOD elaborated on an open and flexible design in project implementation. It is a suitable solution to accommodating DMCs’ requests for climate change–related capacity building activities in line with project objectives.

2. Implementation and/or delivery

The in-country activities’ being identified through national consultations, anchored on existing collaboration mechanisms, building upon similar past and current initiatives, and close coordination with concerned national government agencies contributed to the TA’s successful implementation of activities and delivery of quality outputs.

The TCR noted that the approach of aligning with existing collaborative partnerships, and of building on similar initiatives, contributed to successful implementation. In fact, this is standard practice for ADB’s engagement with DMCs due to its official bearings. Instead, the TCR could have drawn out a lesson on how this was specific to a regional TA. In finalizing this validation report, SAOD revised the lesson in the TCR and highlighted that considering country context in designing activities is important. Identifying in-country activities requires close collaboration with concerned national government agencies, and building upon similar past and current initiatives or mechanisms.

13

Criteria TCR Self-Assessment Comment Additionally, this validation notes that feasible technical solutions that support national needs are critical to getting buy-in and ownership. For instance, some of the final reports for Bangladesh and Bhutan were produced in collaboration with national partners. It is missing from the TCR a description of the issues that led to including the DMF’s third outcome indicator in the DMF, or to managing lessons arising from a substantial increase in budget mid-way through implementation.

3. Management (staffing, including consultants)

Despite a few changes in ADB staff managing and supervising it, the TA benefited from engaging a core set of consultants who provided the institutional memory and worked closely together to undertake specialized yet interrelated assignments, ensuring work consistency, and timely delivery of quality outputs. In the participating South Asia DMCs, the TA accommodated the in-country project management protocols and, thus, enjoyed full support from the respective partner national focal agencies, including inputs from expert-level working groups and guidance from project steering committees. The working group and steering committee provided the national consultant team valuable advice regarding relevant data and information, facilitated ready access to these inputs, monitored the consultants’ performance and reviewed their outputs, and ensured the country activities’ synergy with similar ongoing initiatives. The TA also benefited from the regular communication between ADB staff, and core consultants, and the national TA coordinators from the partner government agencies and national consultant teams.

The TCR noted that consultants’ performance was a key factor in achieving delivery. Engaging quality consultants is important, and that in the case of high ADB staff turnover, consultants can play a role in keeping the project going. Core consultants and national experts provided institutional memory and, by working together, they were able to deliver quality outputs in a timely manner. SAOD has confirmed this when finalizing this validation report. Changes in ADB personnel did not, therefore, significantly affect progress. The TCR could have commented on what lessons can be derived from the TA coordination unit model, which is a unit located in Manila that engages with national consultants based in countries. The TCR noted that gaining the full cooperation of national agencies was important. The in-country governance arrangements, including the working groups and steering committee, played an important role in providing a steer, facilitating access, and reviewing outputs. SAOD added the importance of recognizing country processes in project management. Recognizing in-country protocols is crucial to working together successfully with partner national agencies.

14

Criteria TCR Self-Assessment Comment 4. Knowledge

building The findings/results and lessons learned from TA activities were disseminated within ADB via in-house seminars and more widely via knowledge sharing events (trainings, workshops, seminars) in the participating South Asia DMCs. The project field visits, baseline and vulnerability assessments, and other on-the-ground activities under the country initiatives provided the South Asia DMC national partners first-hand experience and exposure to realities at the community level, and provided local knowledge crucial for designing and developing site- and/or sector-specific climate change adaptation or mitigation measures and policies. The knowledge and skills gained by the national counterparts in all these knowledge building activities are very much applicable to other participatory planning processes.

() Awareness () Technical product () Adoption or uptake () Building institutional or

system capacity () National or sector practice

(guidelines) ( ) Policy, legal standards ( ) Academic literature

Participatory approaches which engage national stakeholders, including through activities such as field visits, vulnerability, and data assessments help to deepen understanding of the local context so that TA design can be appropriately strengthened and learning can be shared with resident missions and other country programs. Participatory approaches can also help to build stakeholder understanding, knowledge, and skills, and are more likely to result in deeper institutional ownership. SAOD added that exposure to local knowledge is important. First-hand experience and exposure to realities at the local or community level is vital in promoting knowledge in designing climate change adaptation and mitigation measures or policies. Measuring participant satisfaction and understanding immediately following training events are limited aspects of institutional learning. However, assessing emerging shifts in behaviors and practice could help to interpret the strength of effectiveness of TA knowledge interventions—evidence on these shifts over time could have been collected during project completion reporting. The use of different dissemination pathways (such as trainings, workshops, seminars) is critical for sharing and transferring knowledge to external partners and can help to stimulate wider interest as these are placed in the public domain (downloads are trackable, and can serve as a proxy for wider interest). Within the ADB sharing tools and knowledge, particularly when they relate across sectors, these can help promote uptake of more efficient models that can contribute to institutional efficiency. Demonstration of the benefits of the tools (such as the time and efficiency improvements from the SARD RS-GIS) can usefully encourage their wider uptake.

15

Criteria TCR Self-Assessment Comment 6. Partnership

(and cofinancing)

The knowledge partnership with the APEC Climate Center (Busan, Republic of Korea) contributed to the incorporation of updated and highly reliable historical and projected climate data and information, and climate scenarios into the SARD RS-GIS based Climate and Disaster Risk Resilience Decision Support System. It also contributed practical strategies to mitigate the impacts and consequences of climate disasters and natural hazards, and to formulate the most appropriate climate change adaptation/mitigation measures for South Asia DMCs.

( ) Internal to ADB () External to ADB (may also

include ADB)

This has not been presented clearly as a lesson. The TCR noted that partnerships with other regional technical institutions, such as the APEC Climate Center and SAARC, but did not elaborate on how this was used in building a wider collaborative effort in the region. This validation notes that consideration should be given as to whether and how these institutional partnerships will be sustained beyond the TA. SAOD highlighted further the importance of developing strategic partnerships in advancing climate change solutions. Leveraging knowledge partnerships with key centers of excellence (e.g., APEC Climate Center) will be a crucial feature in the continuous enhancement of climate data and information for the SARD RS-GIS web-based Climate and Disaster Risk Resilience Decision Support System.

7. Replication and scaling up

With capacity building as an excellent entry point for mainstreaming climate change in national development planning in South Asia DMCs, time-based relevant capacity building plans with appropriate institutional arrangements and streamlined coordination mechanisms will be important in achieving development objectives and sustaining outcomes.

( ) Replication () Scaling up

Institutional capacity building and successful demonstration of pilots provide a strong foundation for scaling up efforts to mainstream climate adaptation and/or mitigation into national policies and programs. This also requires supportive national institutional leadership and streamlined coordination to sustain outcomes.

8. Post-TA financial resources

( ) ADB ( ) Government ( ) Private sector ( ) Other

Not applicable

9. Others Not applicable

16

TCR Quality Assessment (Reviewer’s Assessment)

TCR Quality TCRV

IED Comment HS S LS US Coherence of TCR (25%)

The TCR provided details on the outcomes and objectives, revised DMF, and financial reporting. It also provided information on the governance and management arrangements.

Quality of Data (25%)

Quality of data is assessed satisfactory, although this validation remarks the following shortcomings: (i) The TCR mainly used quantitative data in its

assessment against the DMF indicators and could have provided some further qualitative data with reference to the sources.

(ii) Under output 1, the TCR could have explained ADB’s plans to institutionalize the decision support system beyond this pilot phase and what factors might affect this (e.g., funding constraints, data availability etc.).

(iii) While it appears that RS-GIS system became operational toward the end of the TA, this validation cannot gain access to the site (security error reported).

(iv) The TA report also proposed to strengthen governance and institutional capacity to access and absorb climate finance. However, this was not reflected in the DMF or explained in the TCR.

Quality of Lessons Learned (50%)

The TCR should have offered lessons that could be applicable to similar regional programs in the future, to the ADB’s regional cooperation and integratinon strategy and to country resident missions. This is an essential part of a self-evaluation exercise and the TCR missed the opportunity of offering useful lessons for other future TAs. SAOD provided reformulated lessons in finalizing this validation.

Overall TCR Quality (weighted as per performance)11

The TCR quality is rated satisfactory.

Further IED Action (e.g., in-depth evaluation)

Y

Reason: It will be useful for ADB to ascertain whether national agencies have taken any follow-up action on the proposed climate change and disaster risk systems as intended and what difference this is making. The ADB SARD could also consider an internal stock-take on the use and potential replicability of the RS-GIS tool in other regioal departments. It is further suggested that staff preparing TCRs get familiarized with how lessons are presented, as it is a missed opportunity to just relate

11 Each sub-rating is assigned the numerical value assigned to each rating: highly satisfactory = 3, satisfactory = 2,

less than satisfactory = 1, and unsatisfactory = 0. The compound criteria for TCR quality rating was highly satisfactory (overall weighted average greater than 2.30), satisfactory (overall weighted average greater than or equal to 1.65 and less than or equal to 2.30), less than satisfactory (overall weighted average greater than or equal to 0.75 and less than 1.65), unsatisfactory (overall weighted average is less than 0.75).

17

some facts from the report without extracting lessons to be used in the future. IED can support regional departments to improve this.

Other Remarks The TA has demonstrated that there is a need to strengthen DMC capacity on climate change, and that the development of knowledge, tools, and delivery of capacity building events are important pathways in strengthening capacity. The TA has also shown through the successful pilots for screening of projects that such tools can potentially bring efficiency gains into ADB operations by making CCA processes faster, cost-effective, and more efficient. (The SARD-RS-GIS based tool can prepare the initial CCA in less than 1 hour, compared to the manual version which normally took up to 3 days).

18 Attachments

Attachment 1: Description of the Technical Assistance The technical assistance is described in the technical assistance completion report.1 Attachment 2: Design and Monitoring Framework The design and monitoring framework is in the technical assistance report.2

Planned and Actual Achievements of the Technical Assistance

Performance Indicators Planned Actual

Reasons for

Variance Outcome South AsiaN DMCs’ investment and institutional capacity to manage the impacts of climate change increased

a. SARD approvals

representing both adaptation and mitigation projects achieve the 60% RFI target by 2019 (Baseline: 52% in 2012)

b. Government officials in

priority sectors have improved climate change project development and management capacity based on pre- and post-assessment exercises by 2019 (Baseline: Initial information available)

c. Relevant government

agencies adopt the guidelines for climate risk screening and emission tracking system (Baseline: Pilot adoption by 2019)

a. Achieved. 74% of SARD

projects approved in 2019 supports climate change adaptation and/or mitigation.

b. Substantially achieved. In

2017 and 2018, capacity building workshops government officials attended were conducted in the participating South Asian DMCs, on key topics relevant to the regional TAs in-country sector activities.

c. Partially achieved. By end

2018, governments of Bangladesh and Maldives planned to integrate a climate change and disaster risk screening system, and the government of Bhutan an emission tracking system, into project appraisal and public investment management systems.

Not specified in the TCR.

1 Asian Development Bank. 2020. Technical Assistance Completion Report: Action on Climate Change in South Asia.

Manila. https://www.adb.org/projects/documents/reg-46470-001-tcr. 2 Asian Development Bank. 2013. Technical Assistance for Action on Climate Change in South Asia. Manila.

https://www.adb.org/projects/documents/action-climate-change-south-asia-tar.

Attachments 19

Outputs

Output 1 Screening of investment projects against climate risks and for mitigation potential strengthened

1a. General framework for

climate screening (risk and potentials) finalized and pilot-tested by 2017 (Baseline: NA)

1b. Set of climate change

indicators for all key sectors (in support of SARD RFIs) identified and compiled by 2018 (Baseline: NA)

1c. 100% of infrastructure

projects undergo risk assessment by 2018 (Baseline: NA)

1d. Framework for low-

carbon projects developed and tested by 2018 (Baseline: NA)

1a. Achieved. By June 2017,

general framework for climate and disaster risk screening in SARD had been updated and applied on pipeline projects thereafter.

1b. Substantially achieved.

The SARD RS-GIS based climate risk screening decision support system includes climate change indicators for key sectors and subsectors.

1c. Achieved. All SARD

pipeline projects underwent initial climate and disaster risk assessment at the draft project concept paper stage.

1d. Partially achieved. In

2016, the TA drafted a framework for estimating potential GHG emissions at the project level.

1d. Not specified in the TCR

Output 2 Capacity of South Asian DMCs in developing and/or implementing climate change strategies and action plans strengthened

2a. Supplementary

institutional capacity and training needs assessments conducted for each participating DMC by 2017 (Baseline: Initial information available)

2b. Six capacity building

workshops conducted by 2018 (Note: Number of workshops per country may vary depending on training needs assessment and capacity mapping results; minimum of one per DMC) (Baseline: Initial information available)

2c. Three country climate

assessments and

2a. Achieved. By the time of

in-country inception workshops in 2016 and 2017, one-on-one consultations with relevant government institutions regarding capacity development needs were conducted.

2b. Achieved. A total of 10

capacity building workshops were conducted in the participating South Asian DMCs in 2017 and 2018, on key topics relevant to the regional TA’s in-country activities.

20 Attachments

sector-specific road maps for climate-resilience and low-carbon growth prepared by 2018 (Baseline: NA)

2c. Achieved. By end 2018, climate assessments and resilience plans were prepared for Bangladesh (agriculture and water resources sector), Maldives (coastal and marine resources), and Sri Lanka (five sectors). A vehicle emission control strategy and reduction road map were prepared for Bhutan in July 2017.

DMC = developing member country, GHG = greenhouse gas, RFI = results framework indicator, RS-GIS = remote sensing – geographic information system, SARD = South Asia Department. TA = technical assistance. Sources: Asian Development Bank. 2013. Technical Assistance for Action on Climate Change in South Asia. Manila; and Asian Development Bank. 2020. Technical Assistance Completion Report: Action on Climate Change in South Asia. Manila.