Annual Evaluation Overview Report for 2010RETIREMENT EVALUATION OVERVIEW REPORT FOR 2011 ii 7. CHIEF EVALUATOR’S ASSESSMENT 51 7.1 Performance of EBRD activities based on evaluation

Annual Evaluation Overview Report for 2010

June 2010

ab0cdEvaluation Department (EvD)

RETIREMENT EVALUATION OVERVIEW REPORT FOR 2011

i

TABLE OF CONTENTS

Page LIST OF ABBREVIATIONS iii EXECUTIVE SUMMARY v 1. PERFORMANCE OF INVESTMENT OPERATIONS ASSESSED AGAINST THE EBRD'S MANDATE 1996-2009 1 1.1 Introduction 1 1.2 Evaluation Performance Indicators 1 1.3 Evaluation System 3 1.4 Overall Performance 5 1.5 Transition Impact 7 1.6 Additionality 9 1.7 Environmental and Social Performance and Change 10 1.8 Early Transition Countries 12 2. MONITORING AND EVALUATION OF TRANSITION IMPACT 15

2.1 Introduction 15 2.2 Comparison of TIMS outcomes with EvD evaluations 16 2.3 Transition Impact Retrospective 3: Synopsis of EvD Evaluations 21

3. FURTHER ANALYSIS ON PERFORMANCE OF EVALUATED PROJECTS 23

3.1 Factors affecting performance in evaluation reports 23 3.2 Comparision of project performance by size 33

4. EVALUATION OF TECHNICAL COOPERATON OPERATIONS 37 4.1 TC Evaluation Coverage 37 4.2 Performance evaluation of TC operations 39 4.3 TC-Related Evaluation Work in 2009 40

5. VALIDATION BY EVD OF PERFORMANCE RATINGS ASSIGNED DURING SELF-EVALUATION 42 5.1 The self-evaluation process and validation of ratings by EvD 42 5.2 Comparing the ratings from the self-evaluation and the independent evaluation process 42 5.3 Development of the observed differences over time 45 5.4 Comparison by sector 46 5.5 Major conclusions 48 6. ROLE OF THE BOARD’S AUDIT COMMITTEE IN OVERSEEING THE EVALUATION FUNCTION 49

6.1 Introduction 49 6.2 Review of Evaluation Reports by the Audit Committee 49 6.3 Observations on Evaluation Findings and Lessons Learned as presented in the Minutes of the Meetings of the Audit Committee 50

RETIREMENT EVALUATION OVERVIEW REPORT FOR 2011

ii

7. CHIEF EVALUATOR’S ASSESSMENT 51 7.1 Performance of EBRD activities based on evaluation findings 51 7.2 Review of the independence of the Evaluation Department 52 7.3 Assessment of environmental and social issues 54 7.4 Assessment of technical cooperation related issues 56 7.5 Process review of the system of follow-up of evaluation recommendations 2010 56

LIST OF APPENDICES

EVALUATION DEPARTMENT

ABBREVIATIONS AND DEFINED TERMS ADB Asian Development Bank AEOR Annual Evaluation Overview Report BGD Banking Group Director BP British Petroleum CA Central Asia CEALS Central Europe Agency Lines CEB Countries of central and eastern Europe and the Baltic states CEO Chief Executive Officer CIS Commonwealth of Independent States CLE Country Level Evaluation CSE Country Strategy Evaluation CSU Consultancy Services Unit EAP Environmental Action Plan EBRD European Bank for Reconstruction and Development EC European Commission ECG Evaluation Cooperation Group EEC Eastern Europe & Caucus EIB European Investment Bank EP Environmental Performance ESAP Environmental and Social Action Plan ESD Environmental and Sustainability Department ESI Environmental Social Impact ETC Early transition countries EU European Union EUR Euro EvD EBRD’s Evaluation Department FDI Foreign Direct Investment FI (1) Financial institutions business group (2) Financial intermediary FIRR Financial internal rates of return FOPC Financial and Operations Policies Committee FSOP Financial Sector Operations Policy GPS Good practice standards for private sector evaluation of the ECG IFC International Finance Corporation IFI International financial institution LLD Lessons Learned Database MC Management comments MDB Multi lateral Development Bank MEI Municipal and Environmental Infrastructure MR Monitoring report MSE Micro and small enterprise MSME Micro small and medium sized enterprises OCE Office of the Chief Economist OCU Official Cofinancing Unit OGC Office of the General Counsel OL Operation leader

iii

EVALUATION DEPARTMENT OM Operations manual OPER Operation Performance Evaluation Review OPSCOM Operations Committee OT Operation team PCR Project Completion Report PEP Politically Exposed Person PIU Project Implementation Unit PMM Portfolio Monitoring Module RO Resident office (of EBRD in a country of operation) SEE (1) Southern and eastern Europe (2) Countries of southern and eastern Europe SME Small and medium-sized enterprises TC Technical Cooperation TC Com Technical Cooperation Committee TCFP Technical Cooperation Funds Programme TI Transition impact TIMS Transition Impact Monitoring System XMR Expanded Monitoring Report (investment operations) XMRA XMR Assessment ex-ante before project signing at project appraisal ex-post after project signing at post-evaluation

iv

v

EXECUTIVE SUMMARY OF THE ANNUAL EVALUATION OVERVIEW REPORT FOR 2010

The Evaluation Department (EvD) operates fully independent from management and reports to the Board of Directors exclusively. In the Annual Evaluation Overview Report (AEOR) EvD synthesises the findings of the evaluation process regarding the Bank’s mandate performance, helping the Bank to fulfil its accountability obligations towards the Board of Directors. EvD also helps preserve the corporate memory of the Bank by collecting “lessons learned” through project evaluation and the preparation of Special Studies. 1. INVESTMENT PERFORMANCE JUDGED AGAINST THE BANK’S MANDATE Overall assessment of Bank performance based on evaluated projects. Based on the aggregated evaluation results, EvD concludes that the EBRD has been successful in operating according to its mandate. In total 57 per cent of the evaluated projects in 1996-2009 achieved Successful–Highly successful overall performance ratings. When weighted by volume of investment, this figure rises to 69 per cent. Of evaluated projects in 1996-2009 a total of 79 per cent scored positively on transition impact – 86 per cent when weighted by volume. Of the projects that were evaluated in 2009, 51 per cent achieved a Successful or Highly successful overall performance rating, the lowest level since 2002, while 75 per cent of projects were rated Satisfactory to Good for transition impact, down from 86 per cent in 2008. In 2009, evaluated projects began to show the effects of the current economic turmoil, which was seen particularly in financial performance ratings. Performance ratings across most indicators continue to decline from the high levels seen in 2004. In the current economic situation, ratings for financial performance have suffered more than ratings for transition impact and fulfilment of objectives. Although overall performance levels in respect of additionality showed continued high scores, this indicator fell sharply in 2009 in respect of the Verified in all respects rating category, and this appears to be caused by a gradual decline that is visible when ratings are presented in terms of year of project approval. Regional analysis reveals that the countries of Central Asia and eastern Europe and the Caucasus still achieve lower ratings for overall performance and transition impact than projects in other regions. In the ETCs, very low ratings have been reduced but the recent increase in the proportion of Successful projects appears to have stalled. The proportion of projects rated Good or Excellent for environmental performance has been falling steadily since 2004. 2. MONITORING AND EVALUATION OF TRANSITION IMPACT [QUERY: CAN’T CHANGE

CAPITALISATION] Review of TIMS process and methodologies. In the AEOR for 2007, EvD recommended that the Office of the Chief Economist (OCE) should sharpen its definitions of objectives and benchmarks in order to give a better basis for the assessment of transition impact. OCE has been working over recent years to standardise indicators in each industry sector and

vi

harmonise the rating structure across sectors in an attempt to refocus the project reviews on key transition priorities. The new approach has been developed and is being tested on a pilot basis during the first half of 2010. OCE is continuing to work on improving the consistency of its data records and ensuring that each completed project receives a final assessment and rating in TIMS. Because of differences in focus, purpose and timing between credit monitoring and transition impact monitoring, OCE has taken steps to ensure that it is able to track and monitor projects in parallel to the credit monitoring process where necessary. Comparison of TIMS outcomes with EvD evaluations. The report makes a comparison between the outcomes on 255 operations that have been evaluated by EvD and monitored in TIMS. For the 168 active projects in the sample, EvD’s ratings for transition impact tend to be higher than OCE's transition rank. This may be because an increase in risk (which has been seen in 2009) leads directly to a downgrade of the OCE rank. EvD's evaluation reviews the transition impact of a project through all the seven transition indicators of the EBRD guidelines, and does not have such a rigid relationship between risk and overall transition impact rating. Among the 87 completed operations in the sample, where OCE has removed the risk adjustment and applied a final rating for transition impact, there is a very close correspondence between the results from EvD and OCE. 3. FURTHER ANALYSIS ON PERFORMANCE OF EVALUATED PROJECTS This chapter presents the results of two studies that have been performed during the year on factors affecting performance of evaluated projects. Based on a comprehensive database of projects evaluated by EvD in 1996-2009, the first part of the chapter analysed factors contributing to project success/failure using simple statistics and econometric modelling (ordered logit method). The main conclusions broadly concur with those of previous studies (AEOR 2004 and 2008) for the overall project performance:

• The main factors contributing to the project being successful are both internal (market analysis and financial analysis) and external (government behaviour).

• The same factors together with management skills appear to be important negative factors pushing the overall project performance down.

• Bank handling is relatively less important as a negative factor than as a positive one.

With regards to transition impact, explanatory power of factors affecting performance is weaker than that for the overall performance:

• Bank handling stands our as being particularly an important positive factor for project success, however, it is insignificant as a negative factor.

• Sponsor commitment has a weaker but still significant effect as a positive factor, but again becomes insignificant as a negative factor.

• Market analysis, management skills and government behaviour are both important negative and positive factors.

All the above factors appear in most of the EvD reports and lessons learned, underscoring importance of areas requiring particular attention from the Bank:

• In government-sensitive sectors, projects should be pursued with full agreement and continuous active policy dialogue with country authorities at all levels.

• Good governance remains an important source of long-term project success.

• Enhancing competitive environments supports project success.

vii

The second half of the chapter revisits the issue of project size and the finding in the AEOR for 2009 that larger projects tend to be rated higher than smaller projects for a number of key indicators. The Evaluation Department prepared a paper during 2009 that investigated the findings in more depth, and has updated the main analysis to incorporate the 61 projects evaluated in 2009. The Evaluation Department does not find any reason to revise the conclusions of the original paper based on this update. Although the results from 2009 show a number of larger projects with rather low performance ratings, the numbers (only nine larger projects) are too small to draw firm conclusions. To date, the cumulative results confirm that larger projects are generally rated more highly than small projects. Further analysis to find the main reasons for this conclusion will be carried out. 4. EVALUATION OF TECHNICAL COOPERATION OPERATIONS In compliance with its fiduciary responsibility towards the contributors to its Technical Cooperation (TC) Funds Programme, the Bank puts emphasis on the evaluation of TC projects. Accordingly, TC projects are subject to a mandatory self-evaluation process, in the form of Project Completion Reports (PCRs), and to an independent evaluation process on a sample of the PCRs. Chapter 4 explains that since 1993, when EvD started TC evaluation work, it has conducted 82 Operation Performance Evaluation Reviews (OPERs) and 30 Special Studies on sectors and themes, covering numerous TC operations. Overall, 27.2 per cent of completed TC operations had been evaluated through an OPER report, PCR Assessment or PCR Review by the end of 2009. When TC operations evaluated through Special Studies are added, the cumulative coverage ratio for the period 1991-2009 is 64.6 per cent. The report reviews key TC evaluations in 2009. 5. VALIDATION BY THE EVALUATION DEPARTMENT OF PERFORMANCE RATINGS ASSIGNED

DURING SELF-EVALUATION The Banking Department prepares a self-evaluation report in the form of an expanded monitoring report (XMR) on each project ready for evaluation. EvD’s evaluation may result in different performance ratings than assigned by the operation team in the respective XMRs. As described in Chapter 5, in the last five years XMR ratings were validated by EvD without changing the ratings in 60 per cent of cases. Six per cent of XMR ratings were upgraded by EvD and 34 per cent downgraded. Projects subject to evaluation through OPERs or special studies were much more likely to have ratings adjusted than those evaluated through XMR Assessments. The gap between XMR and evaluation ratings appears to be increasing. Environmental performance was the indicator most likely to be rated lower (46 per cent) by evaluators, followed by transition impact at 40 per cent. Some teams showed much higher levels of downgrades than others. The Evaluation Department currently provides training for new bankers drafting XMRs for the first time, and uses such opportunities to warn against over-optimistic ratings. Ongoing communication with senior Banking staff continues, mostly through the process of discussing draft OPER reports, and this issue will continue to be raised in that context. EvD has discussed the environmental performance ratings with the Environment and Sustainability Department and is organising discussions with the sector teams showing the most substantial differences in relation to transition impact in particular. 6. ROLE OF THE BOARD'S AUDIT COMMITTEE IN OVERSEEING THE EVALUATION FUNCTION Chapter 6 highlights how the Audit Committee has reacted to important evaluation findings and lessons learned. EvD lists the 27 evaluation reports that have been discussed in the Audit Committee during 2009: 15 OPER reports on investment operations, two OPER reports on technical cooperation operations, three special studies, the Annual Evaluation Overview

viii

Report for 2009, three reports on EvD's work programme, two special reports and one report on evaluation recommendations. The chapter highlights a number of key issues on which the Committee commented during the year, including country-level evaluation, the use of the log-frame approach, joint evaluation with other MDBs and realistic transition targets in municipal and environmental infrastructure projects. 7. CHIEF EVALUATOR’S ASSESSMENT Performance of EBRD activities based on evaluation findings. Chapter 7 starts with a presentation of the overall performance of the Bank based on evaluated projects since 1996 as mentioned under section 1 above. Based on these findings whereby transition impact shows continued positive results and the lower overall performance ratings demonstrate that the Bank operates in difficult environments, EvD concludes that the EBRD has been successful in operating according to its mandate. However, the downward trend since 2004 in the higher rating categories on environmental performance gives an indication that this is an area that should be watched. It is important to note, that if performance outcomes are weighted by volume, higher scores are obtained, as is demonstrated in parts of the document. Review of the independence of the Evaluation Department. The Chief Evaluator highlights the Evaluation Department as an important accountability tool for the Board of Directors. The Chief Evaluator is of the view that the independence of the evaluation function continues to be very well secured in the recently Board-approved Update of the Evaluation Policy of the EBRD (2010) and that in the oversight role of the Board through its Audit Committee, extensive attention is given to the evaluation findings and recommendations so that EvD’s accountability and lessons learned focus is fully supported. The Evaluation Policy established in 2005 has in the mean time been updated and appears on the Bank’s website as “Update of the Evaluation Policy of the EBRD”. The Chief Evaluator lists the issues taken into account in this policy update: Assessment of environmental/social issues. During 2009-2010, EBRD’s Evaluation Department (EvD) and Environmental and Sustainability Department (ESD) conducted a dialogue on the Bank’s environmental and social categorisation process, summarised in Appendix 11. Based on this dialogue EvD then developed a number of recommendations as steps that could be undertaken and management/ESD responded. These recommendations and responses are summarised in this section of the report. Process review of the system of follow-up of evaluation recommendations 2010. Since the Evaluation Department became fully independent from management in June 2005, management has had the opportunity to provide formal management’s comments (MCs) to evaluation reports. In 2006, the Board of Directors confirmed a new system on follow-up of evaluation recommendations by management, as proposed by EvD in the AEOR of that year. The Chief Evaluator is of the view that the process of “Follow-up of Evaluation Recommendations 2010”, whereby management reports on the follow-up of evaluation recommendations presented in evaluation reports and whereby a review is made by EvD on management’s accomplishments respectively, worked well and that many lessons have been learned.

1. PERFORMANCE OF INVESTMENT OPERATIONS ASSESSED AGAINST THE EBRD'S MANDATE 1996-2009

1.1 INTRODUCTION

The EBRD’s Evaluation Department (EvD) operates fully independent from management and reports to the Board of Directors exclusively. EvD helps preserve the corporate memory of the Bank by evaluating projects, strategies and policies, and by carrying out Special Studies on sectors, programmes and special themes, including country-level evaluation. The way in which the Department carries out evaluations is presented in the Bank’s Evaluation Policy.1 EvD synthesises its overall findings, including the Bank’s performance on its mandate, in this Annual Evaluation Overview Report (AEOR), thereby complying with its accountability obligations towards the Board of Directors. To ensure an optimal lessons learned orientation, EvD assists the Banking teams and others during the early stages of project preparation to use relevant lessons. This process ensures that this experience is applied to the selection and design of future projects. The experience gained from the Bank’s past performance and the generic and specific lessons and recommendations presented in this and other evaluation reports are available for the Bank’s future strategic orientation. Management’s comments to this report are presented to the Board of Directors in a separate communication in parallel to this document. This chapter presents the Bank’s overall performance over the period 1996-2009, based on results from evaluated projects. 1.2 EVALUATION PERFORMANCE INDICATORS

The evaluation performance indicators, which allow EvD to assign the overall performance rating, are based in part on the Bank's mandate to foster transition in its countries of operations. The relevant indicators consist of the following:

FIGURE 1.1: EVALUATION PERFORMANCE INDICATORS2

Mandate-related

indicators

Sound banking principle-related

indicators Bank effectiveness-related indicators

Performance on transition impact

Environmental performance and

change

The Bank's additionality

Project and company financial

performance

Fulfilment of project

objectives

The Bank's investment

performance

Bank handling

Overall performance rating

1 Evaluation Policy of the EBRD as presented on the Bank’s Website: http://www.ebrd.com/projects/eval/ showcase/core.htm. An Update to the Evaluation Policy of the EBRD (BDS10-024) was approved by the Board of Directors on 23 March 2010. 2 Details on the EBRD’s Operation Performance Rating System at Post-Evaluation, with details on the benchmarks for each of the rating criteria are presented in Appendix 1 of the Evaluation Policy. See also Appendix 13.

2

In Figure 1.1, the indicator boxes presented in blue make up the indicators that define “results on the ground” and as such make up the “transition outcome” rating.3 The figure shows that the overall performance rating of an evaluated operation builds on several underlying performance ratings, derived from the Bank's mandate. Transition impact is the overriding individual rating for all operations. Environmental performance and change are significant indicators for projects with high environmental risks. The following broad performance dimensions are addressed: a. Transition impact

- transition impact is defined as the effects of a Bank project on businesses, markets or institutions that contribute to the transformation from a centrally planned economy to a well-functioning market economy.

The evaluation of transition impact focuses on the broader effects that the project has on the sector and economy at large. Three key areas covering seven transition impact indicators, as used by the Bank during the screening and approval of projects, are applied when evaluating transition impact in Bank projects: A. PROJECT CONTRIBUTIONS TO THE STRUCTURE AND EXTENT OF MARKETS

• greater competitive pressures (1) • market expansion via linkages to suppliers and customers (2)

B. PROJECT CONTRIBUTIONS TO MARKET ORGANISATIONS, INSTITUTIONS AND POLICIES THAT

SUPPORT MARKETS • increased private sector participation (3) • institutions, laws, regulations and policies that promote market functions and

efficiency (4) C. PROJECT CONTRIBUTIONS TO BUSINESS BEHAVIOUR

• transfer and dispersion of skills (5) • demonstration effects and innovation (6) • higher standards of corporate governance and business conduct (7)

EvD assigns a rating to the short-term verified transition impact of a project that can be checked at the post-evaluation stage, as well as to the longer term transition impact potential that can still be realised. EvD then reviews the risk4 that the project may not realise its full transition potential and assigns a rating of Low, Medium, High or Excessive risk. Appendix 6 presents the list of transition objectives that is used by EvD when evaluating ex-post (after project signing at post-evaluation) and the Office of the Chief Economist (OCE) when assessing transition impact ex-ante (before project signing). The transition matrices presented in Appendix 7 for projects evaluated in 2009 illustrate how EvD deals with measuring ex-post. Appendix 8 gives further details than presented in this chapter (Sections 1.4-1.8) on the overall performance scores and shows how the seven underlying performance rating categories behave for all evaluated projects. In order to further document EvD benchmarks for each performance category, Appendix 13 presents detailed descriptions of the benchmarks applied to the individual performance evaluation categories.

3 Presenting evaluation findings based on “results on the ground”, that is “transition outcome”, makes the findings more comparable with other multilateral development banks (MDBs). See further details in Appendix 8, section 1. 4 As EvD evaluates projects at least 18 months after last disbursement of a loan and two years after disbursement of equity, it is experienced that considerable transition impact potential still remains to be realised in the years after the evaluation.

3

b. The environment - environmental and social performance measures how well the environmental

objectives of the project (institutional, emissions control, regulatory compliance, social issues and public participation) were identified and have been met, including potential environmental and social benefits

- environmental change is measure as the difference between the environmental performance before the project started and its performance at the time of evaluation.

c. Additionality

- the Bank's additionality in terms of whether the Bank provides financing that could not be mobilised on the same terms by markets and/or whether the Bank can influence the design and functioning of a project to secure transition impact.

d. Sound banking principles

- project and company (financial) performance provide the sustainability element to allow transition impact to enfold beyond the project/company

- fulfilment of project objectives concerns the extent of verified and expected risk weighted fulfilment potential of the operation’s “process” and “project” objectives (“efficacy”) upon validation of their relevance.

e. The Bank's investment performance - the Bank’s investment performance measures the extent to which the gross

contribution of a project is expected to be sufficient to cover its full average transaction cost and contribute during its life to the Bank’s net profit. Unlike the other dimensions, however, it does not represent any impact of the project “on the ground” in the country.

f. Bank handling

- Bank handling assesses the due diligence, structuring and monitoring of the project, as undertaken by all departments and units involved in the operation process, and the Bank as a whole. A judgement is made on the quality of the work and on how effectively the Bank carried out its work during the life of the project. Positive and negative lessons are generated. In case operations are evaluated that are handled by the Corporate Recovery Unit, Bank handling will also take into account problem recognition, remedial action and recovery efforts.

1.3 EVALUATION SYSTEM

1.3.1 Functioning of the evaluation system The evaluated operations referred to in this AEOR are based (a) on the post-evaluation of a sample of evaluated projects undergoing an operation performance evaluation review (OPER) and (b) on the assessment of expanded monitoring reports (XMRs), the self-evaluation reports prepared by operational staff. With the existing evaluation system EvD fulfils the objective of evaluating a sufficient number of operations and fully complies with its accountability objective. This is done through covering all of the Bank’s ready operations – thereby looking at all self-evaluation reports that are produced by operation staff during the year – with different degrees of evaluation intensity. The quality management objective is fulfilled adequately through gathering lessons when preparing OPER reports and maintaining a lessons-learned dissemination process whereby evaluation staff provide lessons through the lessons learned database (LLD) to operation staff so that this important material is available early on during the project appraisal and preparation process. EvD staff check on the use of lessons through reviewing the quality of the sections on “Lessons learned from past experience” in operation reports before Board approval.

4

EvD is of the view that the self-evaluation system in the EBRD, whereby operational staff prepare XMRs and evaluation staff provide bankers with the necessary assistance during the preparation of the self-evaluation documents, works well and generates valuable lessons learned. However, as suggested in Chapter 5, there is room for improvement in respect of assigning performance ratings by operational staff during the self-evaluation process. As in previous years EvD conducted an analysis comparing the ratings assigned to projects by bankers with the ratings assigned by EvD during the validation process of performance ratings in the self-evaluation XMR reports. The overall level of downgrades by EvD over the past five years is 34 per cent, with environmental performance the indicator most often downgraded by EvD, that is in 46 per cent of the cases. The regular training sessions for bankers on the preparation of their XMRs, which are conducted by EvD, aim to further improve quality of the self-evaluation process. The process of evaluation and self-evaluation of investment operations is presented in Diagram 1.1.

DIAGRAM 1.1: THE EVALUATION PROCESS

LESSONS LEARNED DATABASE (LLD)

Self-evaluation of investment operations

(by Banking Department) *******************

Direct evaluation of investment operations

(by EvD)

XMR Assessment

(about 30 from crop of ready operations)

Selection: Random

sampling

In-depth OPER &OPER reports

(23 reports from crop of ready operations)

Selection: lessons learned potential

Special studies &

Mid-Term Reviews

(about 6 reports)

Sample coverage

XMR Review

(those not covered by an OPER report or XMR Assessment)

FEEDBACK MECHANISM

• Distribution of reports • Workshops at HQ &

Resident Offices • Direct contacts with

bankers • XMR training

XMR preparation by Operation

Leader in Banking Department on all projects ready for

evaluation

Review of XMRs by Team Leader, and all

other departments involved in the

operation process (signing-off process)

Review of all XMRs by EvD on quality. Upgrade

required in case of low quality

XMRs are filed in operation file and

integrated in XMR Assessments and OPER reports

100% coverage

EVALUATION OF INVESTMENT OPERATIONS THE PROCESS

XMR = Expanded Monitoring Report OPER = Operation Performance Evaluation Review

5

1.3.2 Project selection for evaluation Applying the Bank’s Evaluation Policy, evaluations are normally undertaken by EvD after the investment has been made, that is 18 months after last disbursement of a loan and two years after last disbursement of equity. In addition, at least one year of commercial operation must have occurred, evidenced by one year of audited financial accounts. Evaluation results from 2009 are taken from 61 randomly sampled operations, 115 of which were ready for evaluation. Of the 61 operations selected, 24 were evaluated through OPER reports, five through Special Studies and 32 through XMR Assessments. Evaluation, therefore, covered a total of 53 per cent of projects ready for evaluation in 2009. Section 1.3 of Appendix 8 presents the selection methodology of projects for evaluation in more detail. Between 1996 and 2009, 934 investment operations of the Bank’s total cumulative portfolio of 1,830 standalone operations had reached a stage where they were ready for evaluation. The conclusions of this chapter are based on a sample of 679 evaluated projects: 284 evaluated through OPERs, 19 through Special Studies and 376 through XMR Assessments. The evaluations were conducted by EvD in 1996-2009. The overall coverage ratio is therefore 73%. A well-balanced sector and country coverage in the sample of evaluated projects has secured a broad representation of the overall portfolio of the Bank. Appendix 6 provides further details on the size and representation of the sample of evaluated projects. 1.4 OVERALL PERFORMANCE

During 1996-2009, 57 per cent of evaluated operations were given Successful or Highly Successful rating. Weighting the results by volume of investment increases the proportion rated Successful or better to 69 per cent.

57 per cent of evaluated operations were given Successful or Highly Successful ratings on Overall Performance for the period 1996-2009.

In 2004 the Successful and Highly Successful rated projects reached 73 per cent (Chart 1.1). Since then they have fallen and in 2009 they totalled 51 per cent, the lowest level since 2002. The proportion of projects rated Highly Successful rose from none in 2001 to 19 per cent in 2006, but dropped back to three per cent in 2009. At the same time, the number or projects with an Unsuccessful rating has remained steadily below 10 per cent since 2003.

6

Chart 1.1: Overall performance, percentage distribution of assigned ratings (679 investment operations evaluated 1996-2009)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1996-99(169)

2000-03(153)

2004 (52) 2005 (52) 2006 (52) 2007 (54) 2008 (44) 2009 (61)

Year of evaluation (number of evaluations)

% o

f eva

luat

ed p

roje

cts

Highly Successful Successful Partly Successful Unsuccessful

The decline in ratings each year since 2004 was investigated in the AEOR for 2009, in which it was suggested that during the expansionary period of 2003-06, when there were large inflows of foreign direct investment (FDI) into the EBRD's countries of operations. The EBRD may have felt the need to take on more challenging projects in order to remain additional. The lower outcomes in some cases reflect the greater risk. The outcomes for additionality in 2009 may be seen to support this argument (see Section 1.7 below). In 2009, evaluated projects began to show the effects of the current economic turmoil, which was seen particularly in financial performance ratings. Chart 1.2 below shows the breakdown of overall performance ratings by country groups, for all investment operations evaluated since 1996. South-eastern Europe has the highest ratings overall, and this has continued in recent years. Results for Central Asia and Russia, which had been improving strongly, have weakened somewhat in the last couple of years. The Early Transition Countries are distributed between the EEC and CA groups. They are discussed separately in Section 10 of Appendix 8.

7

Chart 1.2: Overall performance, percentage distribution of assigned ratings by country groups (627 investment operations evaluated 1996-2009)5

4% 10% 3%14%

40%

51%

43%44%

50%

40%

25%38%

29%

30%

16% 14% 16% 15%6%

11%0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

CA (57) CEB (226) EEC (90) RUS (136) SEE (118)

Region (number of evaluations)

% o

f eva

luat

ed p

roje

cts


Note: 52 regional projects are omitted

5 Central Asia (CA): Kazakhstan, Kyrgyz Republic, Tajikistan, Turkmenistan, Uzbekistan Central Europe and Baltics (CEB): Croatia, Czech Republic, Estonia, Hungary, Latvia, Lithuania, Poland, Slovak Republic, Slovenia Eastern Europe and Caucasus (EEC): Armenia, Azerbaijan, Belarus, Georgia, Moldova, Ukraine Russia (RUS) South-eastern Europe (SEE): Albania, Bosnia & Herzegovina, Bulgaria, FYR Macedonia, Montenegro, Romania, Serbia

8

1.5 TRANSITION IMPACT

Chart 1.3 below presents the performance ratings on transition impact, applying the six-point rating scale that was introduced in 1999. Of a total of 679 projects evaluated in 1996-2009, 79 per cent achieved Satisfactory–Excellent ratings.

79 per cent of evaluated operations obtained Satisfactory–Excellent ratings on transition impact for the period 1996-2009.

This score is a very important accomplishment that confirms the Bank’s mandate compliance. However, 21 per cent of the evaluated projects obtained a rating of Negative–Marginal, which shows that the Bank operates in difficult environments where many obstacles to transition remain. It can be seen in Chart 1.3 that the proportion of projects rated Good or better has fallen since 2004. Nevertheless, the results for 2009 are still better than in the years before 2004. In 2009, a slightly increased proportion of projects achieved an Excellent or Good rating for transition impact, although there was also an increase in the proportion of projects rated less than Satisfactory.

Chart 1.3: Transition impact, percentage distribution of assigned ratings (679 investment operations evaluated 1996-2009)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1996-99(169)

2000-03(153)

2004 (52) 2005 (52) 2006 (52) 2007 (54) 2008 (44) 2009 (61)


% o

f eva

luat

ed p

roje

cts

Excellent Good Satisfactory Marginal Unsatisfactory Negative

Chart 1.4 presents the transition impact (TI) rating distribution by country groups of the 627 projects evaluated in 1996-2009, after 52 regional projects have been removed. The best performance is found in southern and eastern Europe (SEE). Transition impact ratings have improved in recent years in most regions. In Eastern Europe and Caucasus (EEC), the proportion of projects rated Excellent to Good has fallen in recent years and the proportion rated Excellent to Satisfactory stayed at the same level. In the period 2005-09, the worst performing region was Central Asia, with only 43 per cent of projects rated Excellent or Good and a further 25 per cent rated Satisfactory.

9

Chart 1.4: Percentage distribution of transition impact ratings on 627 post-evaluated investment operations in 1996-2009 by country groups

5% 10% 3%18%

33%

43%47%

46%

49%

26%

26%23%

21%

22%26%12% 19% 13%

9%6% 9%

10%

5%2%

6%2%4% 3% 2%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

CA (57) CEB (226) EEC (90) RUS (136) SEE (118)


% o

f eva

luat

ed p

roje

cts


Note: 52 regional operations are omitted. Chart 1.5 shows assigned TI ratings by sector. Infrastructure has slightly lower outcomes than other sectors but the differences are small. All sectors have improved at the level of Satisfactory or better ratings, compared with the 2000-04 period (see Appendix 8, Section 3.5).

Chart 1.5: Percentage distribution of transition impact ratings on 679 post-evaluated investment operations in 1996-2009 by sector groups6

46% 44% 46%43%

22% 23% 26%29%

14% 12%6%

8%13%9% 4%

15% 16%

1% 5%7% 3%3%1%2%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Financial Institutions(231)

Corporate (272) Energy (72) Infrastructure (104)

Sector (number of evaluations)

% o

f eva

luat

ed p

roje

cts


6 Corporate – agribusiness, general industry, property/tourism and telecommunications Energy – natural resources, and power and energy Infrastructure – municipal/environment and transport

10

1.6 ADDITIONALITY

Chart 1.6 shows the performance of projects on additionality, over the period 1996-2009.

Chart 1.6: Additionality, percentage distribution of assigned ratings (679 investment operations evaluated 1996-2009)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1996-99(169)

2000-03(153)

2004 (52) 2005 (52) 2006 (52) 2007 (54) 2008 (44) 2009 (61)


% o

f eva

luat

ed p

roje

cts

Verified in All Respects Verified at Large Verified in Part Not Verified

Although a very high proportion of projects continue to be rated Verified at Large or better, there has been a gradual deterioration in performance at the Verified in All Respects level, and this has culminated in a strong fall in 2009. Additionality is strongly related to conditions at the time of project approval. Chart 1.7 shows the proportion of evaluated projects rated Verified in All Respects for additionality by their year of Board approval, and also juxtaposes the level of total Board approvals in the same years, in terms of EUR million.

Chart 1.7: Additionality, percentage of projects Verified in All Respects by year of Board approval (679 investment operations approved 1991-2007 and evaluated 1996-2009)

and total annual volume of Board approved projects

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007

Year of Board approval

% o

f eva

luat

ed p

roje

cts

0

1,000

2,000

3,000

4,000

5,000

6,000

EU

R m

illio

n

% of projects Verified in All Respects (left-hand axis) EBRD's annual Board approved volume (right-hand axis)

11

The decline in ratings achieved for additionality is clearer when shown by year of approval than by year of evaluation. The majority of projects evaluated in 2009 (around 75 per cent) were approved in 2005-07, when only around 40 per cent of approved projects subsequently achieved Verified in All Respects, the highest rating for additionality. The sudden fall in this performance category in projects evaluated in 2009 is actually the result of a more gradual decline. The decline is seen across a number of regions but particularly SEE and Central Asia. Much of the fall took place after 2002. At this time there was an economic boom in the EBRD's countries of operations. Foreign direct investment (FDI) across the region quadrupled between 2003 and 2007. Liquidity in the market was also increasing, which would tend to threaten the financial additionality of the EBRD's projects. The Bank had the experience of some of its longer standing or better established clients prepaying loans and choosing to obtain finance from elsewhere. Nonetheless, volumes of new EBRD project approvals increased strongly during the period. Given the liquidity in the market, it is worth mentioning that the EBRD found itself additional in so many projects rated Verified in All Respect or Verified at Large. The Evaluation Department poses the question whether the fall in ratings was entirely because of the greater availability of finance during those years, or whether the EBRD's strategy of strong growth itself led it to accept projects in which it was less additional. In answering this question it should be mentioned that the relatively high scores (89 per cent) on the two top additionality ratings shows that although at times the financial additionality of a project at Board approval might have been on the low side, this must have been compensated by a higher design and functioning additionality, pointing to positive transition impact at the corporate level. 1.7 ENVIRONMENTAL AND SOCIAL PERFORMANCE AND CHANGE

1.7.1 Measuring environmental performance and environmental change The environmental evaluation data from 1996-2009 covers two environmental dimensions. The first dimension concerns environmental and social performance7 of the Sponsor, for example the preparation and implementation of environmental action plans, compliance with contractual environmental conditions, performance against national and EU statutory regulations, and so on. The second dimension is the extent of environmental change (positive or negative) brought about by the evaluated operation. Under Bank handling, EvD also considers environmental Bank handling with respect to categorisation, environmental due diligence and project monitoring. Environmental performance8 is included in the ex-post assessments of all projects. As presented in Chart 1.8, cumulative environmental performance in 1996-2009 was rated Good or Excellent in 54 per cent of cases and Satisfactory in a further 32 per cent. Over the period 1996-2009, only three per cent of the projects evaluated have been rated Unsatisfactory in respect of environmental performance and none Highly Unsatisfactory.

7 It is important to note that from 2003 onwards, the social elements were incorporated in the new environmental policy. From that time onwards the rating category in fact covers environmental as well as social performance. 8 Environmental performance of projects is measured by accumulating the environmental and health and safety performance indicators: environment being the status of the environment in the project vicinity; health and safety: the way in which health and safety and respective risk assessment systems are effectively applied and the extent of compliance in this respect; pollution loads and energy efficiency: the extent to which the emissions are significantly lower that the regulatory limits; environmental management: the level of compliance with the agreed environmental action plan; public consultation and participation: whether the public consultation and participation has been carefully planned and organised with a responsible person in charge.

12

Cumulatively, 86 per cent of evaluated operations obtained ratings of Satisfactory to Excellent on environmental performance for the period 1996-2009.

This cumulative rating of 86 per cent shows an overall positive development and reflects the dedication of the Bank in handling environmental issues. However, as presented in Chart 1.9 below, following a peak in environmental performance in 2004, the average year-on-year Excellent or Good rating has declined from 77 per cent in 2004 to 42 per cent in 2009, when no projects were rated Excellent for this indicator. Some of the projects evaluated in 2008-09 had already experienced delays in environmental investments as a result of the crisis. When companies are under financial stress, investments in environmental quality may be delayed resulting in a lower compliance of environmental conditionality. During the financial crisis it is important that greater emphasis is placed on environmental performance and meeting the project specific Environmental and Social Action Plan (ESAP) conditions to stop and possibly reverse these declines.

Environmental performance ratings have been declining from a high of 77 per cent Good or Excellent in 2004 to 42 per cent in 2009.

In respect of cumulative environmental change,9 cumulatively 24 per cent of the evaluated projects were rated Substantial or Outstanding, while 53 per cent achieved Some environmental change. However, in 2009 no projects attained an Outstanding rating, which is cause for some concern. The proportion of projects achieving Some change has been growing at the expense of other ratings, especially higher ones, in recent years.

Chart 1.8: Environmental and social performance, percentage distribution of assigned ratings (664 investment operations evaluated 1996-2009)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1996(34)

1997(36)

1998(49)

1999(50)

2000(42)

2001(49)

2002(49)

2003(53)

2004(52)

2005(52)

2006(51)

2007(50)

2008(42)

2009(55)


% o

f eva

luat

ed p

roje

cts

Excellent Good Satisfactory Marginal Unsatisfactory Highly Unsatisfactory

9 The extent of environmental change (environmental impact) is measured as the difference between the environmental performance before the project started and its performance at the time of evaluation.

13

Chart 1.9: Extent of environmental change, percentage distribution of assigned ratings

(6634 investment operations evaluated 1996-2009)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1996(34)

1997(36)

1998(49)

1999(50)

2000(42)

2001(49)

2002(49)

2003(53)

2004(52)

2005(52)

2006(51)

2007(50)

2008(42)

2009(54)


% o

f eva

luat

ed p

roje

cts

Outstanding Substantial Some None/Negative

1.7.2 Environmental and social impact As part of the EvD 2007 Special Study on Achieving the Bank’s Environmental and Social Mandate through Direct Investments, EvD introduced and proposed a new measure for environmental social impact (ESI) and pilot tested this during 2008 and 2009. The new index was designed to parallel the Bank’s assessment of transition impact and to combine the current environmental performance (EP) and environmental change (EC) ratings into one rating. EvD has found that for most projects the ratings for EP and ESI are identical. There were a few projects for which the ESI rating was higher, and few were the ESI rating was lower, but overall the rating for ESI was equal to or higher than EP (see Appendix 8). EvD believes that the new rating gives a more accurate assessment of the Bank’s environmental and social impact. 1.8 EARLY TRANSITION COUNTRIES

The Early Transition Countries Initiative (ETCI) was launched in 2004 to increase the Bank’s activities in its poorest countries of operations: Armenia, Azerbaijan, Georgia, Kyrgyz Republic, Moldova, Tajikistan and Uzbekistan. Mongolia was added when it became a country of operations in 2007. The countries form a sub-set of the Central Asia and eastern Europe and Caucasus regions. The performance of evaluated projects in the early transition countries has been and remains lower than in other regions. Charts 1.10 and 1.11 show the development over time of overall performance and transition impact ratings in early transition countries. The proportion of projects rated Unsuccessful has fallen to nine per cent, which is a major achievement. However, improvements in the proportion rated Successful or better appear to have stalled. This may be an effect of the current economic situation. A similar result is seen for transition impact, where very few projects are now rated Unsatisfactory or Negative but there is still a relatively large number rated Marginal.

14

Chart 1.10: Overall performance in early Chart 1.11: Transition impact in early transition countries: development of ratings transition countries: development of ratings over time (83 investment projects evaluated over time (83 investment projects evaluated 1996-2009) 1996-2009)

Overall performance

27%42% 40%

40%

33%49%

33%

3%

9%24%

0%

20%

40%

60%

80%

100%

1996-1999 (15) 2000-2004 (33) 2005-2009 (35)

Year (number) of evaluation(s)

% o

f eva

luat

ed p

roje

cts


Transition impact

9%20%

48%43%

53%18% 17%

13%15%

15%29%

3%13%

0%

20%

40%

60%

80%

100%

1996-1999 (15) 2000-2004 (33) 2005-2009 (35)


% o

f eva

luat

ed p

roje

cts


1.9 FINDINGS AND CONCLUSIONS BASED ON PERFORMANCE OF EVALUATED

OPERATIONS IN 2008

• In total 57 per cent of the evaluated projects in 1996-2009 achieved Successful–Highly Successful overall performance ratings. When weighted by volume of investment, this figure rises to 69 per cent. Of evaluated projects in 1996-2009 a total of 79 per cent scored positively on transition impact – 86 per cent when weighted by volume. These positive outcomes lead EvD to the conclusion that the EBRD has been successful in operating according to its mandate.

• Performance ratings across most indicators continue to decline from the high levels

seen in 2004. In the current economic situation, ratings for financial performance have suffered more than ratings for transition impact and fulfilment of objectives.

• Additionality fell sharply in 2009 in respect of the Verified in All Respects rating

category, and this appears to be a result of a gradual decline that is visible when ratings are presented in terms of year of project approval. Nevertheless, a high proportion of projects are rated at least Verified at Large for additionality.

• Regional analysis reveals that the countries of Central Asia and eastern Europe and the

Caucasus still achieve lower ratings for overall performance and transition impact than projects in other regions. In the ETCs, very low ratings have been reduced but the recent increase in the proportion of Successful projects appears to have stalled.

• The proportion of projects rated Good or Excellent for environmental performance has

been falling steadily since 2004.

15

2. MONITORING AND EVALUATION OF TRANSITION IMPACT 2.1. INTRODUCTION The Transition Impact Monitoring System (TIMS) was established in 2003 to better monitor the transition impact during project implementation.10 Since then EvD has paid regular attention to the developments of the entire system, from process to findings and methodologies, including recent OCE initiatives to introduce and monitor broader social indicators for ETC. The EvD work programme for this year specified that the 2010 AEOR would again compare evaluation data with data from the transition monitoring system on a cohort of projects rated both by EvD and by OCE.11

Regarding the process of TIMS, the TIMS Review document (formerly the Review Checklist) is the core monitoring report of TIMS that periodically records and rates the project accomplishments and potential in support of the client’s transition progress towards the realisation of a full market economy. In the 2004 and 2006 AEORs, recommendations were made to support the very good start and further improve the processing of the Review Checklist.12 In the AEOR of last year, EvD noted that the TIMS process had become well tuned for information gathering and presentation of results for quarterly reporting.13 In the area of TIMS ratings methodologies it was suggested in the AEOR for 2007 that OCE should sharpen its definitions of objectives and benchmarks in order to give a better basis for the assessment of transition impact. OCE has been working over recent years to standardise indicators in each industry sector and harmonise the rating structure across sectors in an attempt to refocus the project reviews on key transition priorities. The new approach has been developed and is being tested on a pilot basis during the first half of 2010. This initiative is expected not only to further improve the overall quality of TIMS, but it will also allow for aggregation of transition impact within and across sectors. OCE has also introduced a new rating on information quality, in order to try to receive better information from Banking. It is continuing to work on improving the consistency of its data records and ensuring that each completed project receives a final assessment and rating in TIMS. Because of differences in focus, purpose and timing between credit monitoring and transition impact monitoring, OCE has taken steps to ensure that it is able to track and monitor projects in parallel to the credit monitoring process where necessary. This will also ensure that operations that are no longer monitored in Project Monitoring Module (PMM) (for example, sovereign loans that have achieved physical completion) continue to be monitored in TIMS while their transition objectives remain to be achieved. OCE is also working at rationalising transition impact monitoring so that it focuses its resources

10 The transition monitoring system is expected to meet essentially three objectives: (a) improve the structure of the projects by fine tuning the balance between transition target, covenants and risk mitigating factors; (b) address transition impact issues as soon as they arise; and (c) provide a regular assessment of progress in achieving transition impact (see the ExCom Paper of December 2002). A summary description of the TIMS methodology is provided in the 2005 Strategic Portfolio Review, Report BDS06-52 pages 28-29. 11 See “Evaluation Department’s Work Programme Final Report for 2010”, Report BDS10-012, January 2010, Section 10, page 16. 12 See Annual Evaluation Overview Report (AEOR) for 2004 (BDS04-069), Chapter 7, Section 7.3.4, and AEOR 2006 (BDS06-122(Final)), Chapter 2, Section 2.2.4. 13 A benchmarking system is built to assess realised transition and the risk attached to the remaining potential, that is the more is realised, the lower the risk attached to remaining potential. The benchmark approach also facilitates the conversion by OCE of its potential rating and associated risk into a realised TI rating equivalent at the end of the TIMS review, when a project is closed.

16

on those projects with a greater potential role for the OCE to affect their success during implementation and at those times that are most critical to the achievement of the transition objectives of the project. Meanwhile, given the continuously growing number of projects that become subject to TIMS review, it is useful to provide this year a new comparison of TIMS outcomes and EvD evaluations. The common sample of operations that all have both an ex-ante and ex-post transition impact rating, has increased to 255 projects this year, an increase of 60 projects compared with last year. The analysis of the sample is presented in Section 2.2. Further rationalisation of TIMS around standardised transition impact objectives raises the issue of the treatment of social and environmental indicators, which are intertwined with transition but conceptually different. This issue was already identified in the evaluation of TIMS in the AEORs of 2007 to 2009. This issue continues to be an area of discussion in the Bank. OCE is working on ways to broaden its monitoring system to be able to monitor integrated approaches and, to the extent possible, social and environmental issues. 2.2 COMPARISON OF TIMS OUTCOMES WITH EVD EVALUATIONS 2.2.1 What is to be compared The TIMS process generates a series of reports for each project that reflect the history of the transition performance during the life of the project, while EvD evaluations are typically made after project completion for selected projects. Both EvD and TIMS use a rating structure that summarises the performance of a project at a particular time. While there are some methodological differences (see Box 2.1), both approaches have enough in common to allow a comparison of their respective aggregated outcomes, as they rely upon the same fundamental rating categories established in 1999. The comparison places TIMS outcomes in a broader framework of the evaluation cycle that goes beyond project implementation. In this section EvD identifies a common sample of projects that had both TIMS monitoring and EvD ratings, uses the sample to compare the latest TIMS to the EvD ex-post ratings, and derives conclusions on current TIMS rating properties. On that part of the Bank’s portfolio that includes both ex-ante and ex-post ratings, it appears appropriate to compare the TIMS expected transition impact with the EvD overall transition impact, as they both reflect the realised transition, in addition to remaining potential. The TIMS assessment of “expected transition” is rated according to the 1 to 8 ordinal scales defined in the portfolio analysis of the budget for 2005. The EvD ex-post overall transition ratings refer to the 1 to 6 OCE/EvD scale of 1999.

17

Box 2.1: Commonality and variants in the TIMS and EvD rating systems OCE and EvD make the same distinction between the transition impact potential of a project and the risks to transition impact in their respective evaluations. The transition potential and risk are measured along the same ordinal scales. The methodology was presented first to the Financial and Operations Policies Committee (FOPC) in a 1997 memorandum called “Transition impact of projects” (CS/FO/97-3), and confirmed later by the Board following the adoption of the memorandum “Moving transition forward” (See BDS99-24 – Rev1). EvD focuses on ex-post performance and works with three categories of ratings, separating (a) short-term realised transition (b) remaining transition potential and (c) risk attached to the remaining potential, as components entering into an overall transition impact rating. The rating scale used by EvD (and Banking for projects at entry and Expanded Monitoring Reports) includes all the above components at different levels of TI performance, which are:

1 excellent, 2 good, 3 satisfactory, 4 marginal, 5 unsatisfactory, 6 negative (only ex-post)

OCE monitors ongoing implementation and uses only two categories: transition potential and risk. Any change in short-term observed performance in TIMS is registered mostly within changes of original risk: the higher the short-term performance, the lower the risk to the overall potential. In the context of the strategic management of the Bank’s portfolio, OCE has developed a 1 to 8 rating system that combines risk with transition impact potential in order to assess how both flow and stock of projects are achieving their expected impact on transition (see the 2006 Strategic Portfolio Review, BDS07-069, pages 29-41). The OCE combinations of transition impact potential/risk are classified and ranked as: Rank Potential Risk Potential Risk Potential Risk 1 Excellent Negligible 2 Excellent Low Good Negligible 3 Excellent Medium Good Low Satisfactory Negligible 4 Excellent High Good Medium Satisfactory Low 5 Excellent High/Excessive Good High Satisfactory Medium 6 Good High/Excessive Satisfactory High Marginal Low/Negligible 7 Satisfactory High/Excessive Marginal High Marginal Medium 8 Marginal High/Excessive Unsatisfactory <any> <any> Excessive

2.2.2 Features of the common OCE/EvD sample At the end of 2009, the stock of active operations subject to transition impact monitoring in TIMS stood at 1,113. In addition, 411 projects formerly monitored in TIMS were considered "completed" from a transition perspective and were no longer monitored. The common sample now comprises 255 projects,14 which represents only 17 per cent of all projects that have been monitored in TIMS. As shown in Table 2.1, the performance of the sample is quite closely aligned to that of the population of all projects evaluated by EvD during the period 2001-2009. However, this joint sample does not closely mirror the sectoral distribution of projects in the TIMS portfolio as a whole.15 Therefore, the findings could still change substantially over the coming years as the sample changes.

14 Although EvD sometimes groups operations for evaluation purposes, care was taken to include in the sample of 255 projects only the results from evaluation reports that had a clear "lead operation" monitored in TIMS. Evaluations covering several distinct operations, or focusing on a different lead operation from the corresponding TIMS reports, were excluded. 15 About one third of the evaluated projects in the joint sample are in the Financial Institutions sector (31 per cent), 39 per cent in the Corporate sector, 10 per cent in Energy and the remaining 20 per cent in Infrastructure. Among the 1,524 projects currently or previously monitored in TIMS, 44 per cent are in the Financial Institutions sector, 31 per cent in the Corporate sector, 8 per cent in Energy and 17 per cent in Infrastructure.

18

Table 2.1: Distribution of overall transition impact ratings in the 255 project sample compared with larger EvD project groups. Excellent Good Satisfactory Marginal Unsatisfactory Negative

All projects 1996-2009

10% 45% 24% 13% 6% 2%

All projects 2001-2009

10% 48% 23% 13% 5% 1%

Projects in the sample of 255

12% 50% 24% 11% 2% 1%

2.2.3 Results of the comparison for the full sample The results shown in Charts 2.1 and 2.2 show that there is a general similarity in the pattern between EvD and TIMS ratings. Half the projects have been rated Good by EvD, and a majority of the sample has achieved a TIMS ranking between 2 and 4, which equates to a Good potential with Negligible to Medium risk. However, the TIMS ratings do not follow a smooth curve: there is an almost equal proportion of the projects at each of the ranks 2 to 4 and a spike at rank 8. This seems to have been caused by the increasing proportion of completed projects included in the sample. Therefore, for further analysis EvD has separated the active from the complete projects in the TIMS database. Projects continue to be actively monitored in TIMS until the EBRD exits from the project itself (through repayment, prepayment or equity exit), or until OCE concludes that an active project has achieved all the transition impact it is likely to achieve and further monitoring is not worthwhile. At this stage a final rating is applied for transition impact and the risk rating is reduced to Negligible (or occasionally Low). As is clear from the list in Box 2.1, this means that projects for which TIMS monitoring is complete cannot have a rank of 5 or 7 and are unlikely to have a rank of 4. Once the sample includes a large proportion of completed operations, they will start to distort the curve of the overall results. At the end of 2009, the sample of 255 projects consisted of 87 complete and 168 active operations. Chart 2.1: Latest TIMS updated expected Chart 2.2: EvD overall transition impact transition impact ranking of 255 projects rating of 255 projects assessed and evaluated assessed and evaluated in 2000 to 2009 in 2000 to 2009

4%

25% 25%24%

14%

5%

0%

3%

0%

5%

10%

15%

20%

25%

30%

1 2 3 4 5 6 7 8

12%

50%

24%

11%

2% 0%0%

10%

20%

30%

40%

50%

60%

1 2 3 4 5 6

Key: 1=Excellent/Negligible Key: 1=Excellent 2=Excellent/Low – Good/Negligible 2=Good 3=Excellent/Medium – Good/Low – Satisfactory/Negligible 3=Satisfactory 4=Excellent/High – Good/Medium – Satisfactory/Low 4=Marginal 5=Good/High – Satisfactory/Medium 5=Unsatisfactory 6=Satisfactory/High – Marginal/Low or Negligible 6=Negative 7=Marginal, High/Medium 8=Unsatisfactory/any risk, any rating/Excessive.

19

2.2.4 Results of the comparison for the active sample (168 operations) Charts 2.3 and 2.4 show the results for the active projects only. The TIMS rankings show a smoother curve than for the full sample, while the pattern of EvD ratings is virtually unchanged. The patterns are similar for EvD and TIMS, although the TIMS rankings appear to peak at a slightly lower level than the EvD ratings. The striking difference compared with the results in the AEOR for 2009 is that the most common TIMS rank this year is 4 (31 per cent of rated operations, up from 29 per cent at the end of 2008) while last year it was 3 (23 per cent of rated operations this year, down from 35 per cent at the end of 2008). There is also a noticeable increase in the proportion of projects with rank 5 in TIMS (21 per cent, up from 10 per cent last year). Closer analysis shows that this has less to do with operations downgraded in TIMS during the year, than with additions to the common database (that is, projects evaluated during 2009). Of the projects evaluated in 2009 and actively monitored in TIMS, around 17 per cent had rank 3 in TIMS, 23 per cent had rank 4 and 40 per cent had rank 5. It is recognised that TIMS ratings have fallen during 2009. In the Institutional Performance Report: Fourth Quarter 2009 and Year Ended 31 December 2009 (BDS10-022), it was reported that "the largest number of downgrades ever realised in a year" had occurred in 2009. In many cases the downgrade was because of a rise in the risk rating. It was also reported that an unusually high proportion of projects entering the TIMS database had a high risk rating, leading to a lower rank.16 Although such new entrants to the TIMS database will not yet show up in the common database of projects also evaluated by EvD, this illustrates the new environment of increased risk in which the Bank is operating. Chart 2.3: Latest TIMS updated expected Chart 2.4: EvD overall transition impact transition impact ranking of 168 evaluated rating of 168 evaluated projects projects active in TIMS at the end of 2009 active in TIMS at the end of 2009

2%

16%

23%

31%

21%

5%

0%3%

0%

5%

10%

15%

20%

25%

30%

35%

1 2 3 4 5 6 7 8

12%

52%

23%

11%

2%0%

0%

10%

20%

30%

40%

50%

60%

1 2 3 4 5 6

Key: 1=Excellent/Negligible Key: 1=Excellent 2=Excellent/Low – Good/Negligible 2=Good 3=Excellent/Medium – Good/Low – Satisfactory/Negligible 3=Satisfactory 4=Excellent/High – Good/Medium – Satisfactory/Low 4=Marginal 5=Good/High – Satisfactory/Medium 5=Unsatisfactory 6=Satisfactory/High – Marginal/Low or Negligible 6=Negative 7=Marginal, High/Medium 8=Unsatisfactory/any risk, any rating/Excessive

Why is this reduced risk not also seen in the results of projects evaluated by EvD? One 16 Around 70 per cent of projects entering the TIMS database in 2009 were rated as having "Good" potential with "High" or "High, perhaps Excessive" risk (that is, rank 5).

20

explanation may be the way EvD is treating of risk. Although both OCE and EvD take consideration of risk, in the case of TIMS an increase in risk leads directly to a downgrade of the rank. Furthermore, the ratings are linked to the achievement of specific transition objectives within the foreseen timeframe. EvD's evaluation reviews the transition impact of a project through all the seven transition indicators of the EBRD guidelines, and does not have such a rigid relationship between risk and overall transition impact rating. This allows for a broader and more long-term assessment that a project, while at risk of missing some immediate and specific targets, scores on other transition criteria. It is also possible that the project recovers over time and achieves part of the intended transition impact in the long run. Some of the transition objectives tracked in TIMS might be regarded by EvD as more closely related to achievement of objectives, or even environmental and social impact, than to transition impact. 2.2.5 Results of the comparison for the completed sample (87 operations) A comparison is also made on a smaller set of 87 completed operations taken from the joint database, which includes only projects considered "complete" by OCE and no longer rated under TIMS, and the corresponding ratings of the EvD's transition impact. As shown in Charts 2.5 and 2.6, the patterns of TIMS rankings and EvD ratings are very similar, with the largest group of projects obtaining a rank of 2 in TIMS. A rank of 2 represents Good/Negligible or Excellent/Low ratings and corresponds well to the Good rating most commonly applied by EvD. Therefore it appears that the differences between TIMS and EvD results tend to disappear once the risk factor is removed and a final TIMS rating is applied. This analysis and the related conclusion apply to the sample in its entirety. It is not excluded that variances could occur at sector level. Chart 2.5: Latest TIMS updated expected Chart 2.6: EvD overall transition impact transition impact ranking of 87 completed rating of 87 completed projects assessed and projects assessed and evaluated in 2000 to 2009 evaluated in 2000 to 2009

7%

43%

31%

11%

0%

5%

0%3%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

1 2 3 4 5 6 7 8

11%

47%

26%

11%

2% 1%0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

1 2 3 4 5 6

Key: 1=Excellent/Negligible Key: 1=Excellent 2=Excellent/Low – Good/Negligible 2=Good 3=Excellent/Medium – Good/Low – Satisfactory/Negligible 3=Satisfactory 4=Excellent/High – Good/Medium – Satisfactory/Low 4=Marginal 5=Good/High – Satisfactory/Medium 5=Unsatisfactory 6=Satisfactory/High – Marginal/Low or Negligible 6=Negative 7=Marginal, High/Medium 8=Unsatisfactory/any risk, any rating/Excessive

21

2.3 TRANSITION IMPACT RETROSPECTIVE 3: SYNOPSIS OF EVD PROJECT EVALUATIONS

In 2009, EvD prepared a synopsis of EvD project evaluations,17 which covered 202 operations evaluated in 2005-08, the period after the second Transition Impact Retrospective (TIR 2) report. The evaluation of transition impact (TI) performed ex-post by the EvD measures the degree to which a project succeeded in advancing transition, for a given stage of country and sector reform environment.18

The main findings show that the transition impact performance emerging from the EvD project evaluations in 2005-08 has improved compared with that in the TIR 2 period, with 85 percent of 202 evaluated projects being rated Satisfactory and above. However, performance varies quite significantly by sector and country groups. Sector disaggregation shows that in infrastructure, those ratings sum up to 78 per cent, which EvD attributes partly to difficult economic environment in a number of countries and risk of changes on the political front. When disaggregated by country groups, the performance in SEE countries was the best, as the reform environment has been conducive to high gains in transition impact from projects. Cross-regional projects have shown the second-best scores. At the same time, countries at both ends of transition, the most advanced CEB countries and the least advanced Central Asian countries, have the lowest proportion of projects rated Good and above. Russia’s performance was only slightly higher than that in the CEB country group, with performance in infrastructure being worse compared with that in other sectors, which is similar to infrastructure sector performance across the entire region. The analysis of factors affecting performance indicates that commercial and institutional building factors together with the Bank handling appeared the most important ones for a project to be successful. Lessons from unsuccessful projects emphasise once again importance of governance and management skills, integrity and transparency, better initial project design, as well as rigorous due diligence at the initial stage of the project.

17 Evaluation Department, 2009, Transition Impact Retrospective 3: Synopsis of EvD Project Evaluations. 18 The EvD overall transition impact rating for a project is derived from an assessment of the short-term verified impact, the potential for further transition impact, and the risk attached to the realisation of this potential.

22

Chart 2.6: TI ratings of 202 investment operations evaluated, 2005-08

17%7% 6%

14%

50% 42% 35%48% 49%

44%

19%35%

30%21%

27%

11%13%

10%4%

12%20%

23%

4%12%15%

8%4% 6% 2% 5% 1%4% 3%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2005 (52) 2006 (52) 2007 (54) 2008 (44) TIR2 (247) TIR3 (202)

Ye a r o f e v a lua t io n (num be r o f e v a lua t io ns )

% o

f eva

luat

ed p

roje

cts

Excellent Go o d Satis fac to ry Margina l Uns a tis fac to ry Negative

23

3. FURTHER ANALYSIS ON PERFORMANCE OF EVALUATED PROJECTS

This chapter presents the results of two studies that have been performed during the year on factors affecting performance of evaluated projects. Section 3.1 reports on a study analysing the relative importance of different factors in affecting project performance, building on previous studies presented in the AEORs for 2004 and 2008. Section 3.2 revisits the issue of project size and the finding in the AEOR for 2009 that larger projects tend to be rated higher than smaller projects for a number of key indicators. 3.1 FACTORS AFFECTING PERFORMANCE IN EVALUATION REPORTS

This section is an extension of the analysis of factors affecting project performance, conducted in the 2004 and 2008 AEORs. The purpose of this section is twofold: (i) to assess the sample of projects evaluated by EvD19 using the previously created “factors affecting performance” framework and (ii) to perform statistical and econometric analysis into the causality with regards to the factors and the various project performance ratings that are assigned by EvD.

The methodology used to realise the first objective is almost identical to the previous studies. The main difference, apart from the obvious further sample expansion, is that all projects are assessed for the presence of both positive and negative factors, denoted by a plus and minus one respectively in the newly created database. In previous work, Successful and Highly Successful projects were assumed to only have positive factors and Unsuccessful and Partly Successful projects only to have negative factors.

While this change in methodology could potentially introduce a bias when comparing results across the old and new samples or when using econometric techniques across the complete sample, the initial assumption is largely realised across the expanded sample – only two per cent of Successful projects have a negative factor and five per cent of Partly Successful projects have a positive factor. As a result, any such bias is likely to be minimal. In this chapter, for the purpose of the factor analysis, the bar between “successful” and “unsuccessful” was set in the middle of the scale of the four-level rating scale that EvD uses to assess “overall performance”.20

The second objective is the most complex and adds the most to this exercise compared to the previous work. Apart from assembling a simple statistical summary, the econometric analysis was conducted using the “ordered logit” approach, which has its origin in the econometrics in consumer choice and survey data. Fundamentally, this approach asks that given a set of ordered ratings, what is the probability of a project being assigned a particular rating, given a set of explanatory variables. To take into account potential endogeneity issues (econometric jargon that captures the fact that all ratings are part of a larger whole and hence are all correlated with each other) that could bias the results, a restricted set of regressions was run with the set of explanatory variables limited to the 10 factors described below. Nevertheless, the model proved effective and led to some powerful results.

19 Based on OPERs covering the period 1996-2009. 20 EvD’s evaluation of projects systematically focuses on specific areas: project rationale, achievement of objectives, transition impact, additionality environment, and Bank handling. The “overall assessment”, which encompasses all of the above, is rated on a scale of four ratings: Highly Successful, Successful, Partly Successful, and Unsuccessful.

24

3.1.1 Factors affecting performance

Factors affecting performance are not intrinsically part of the OPER reports, and they are not recorded in a systematic way, although these factors are tightly linked to the specific characteristics of each project. However, the factors can be detected from a careful review of each OPER report. They are at times explicitly mentioned and on other occasions more difficult to apprehend, especially for financial and commercial categories. Institutional building factors are more easily found since they often appear in the transition impact section.

Regarding the factors themselves, as in the previous works, the projects were assessed for 10 key factors affecting project performance, which can be grouped into five functional categories (see Table 3.1). Three are internal to projects: the financial, commercial and institutional performance. The two others were external to the project: the effect of business cycle (a catch-all for all macroeconomic shocks) and role of government. Bank handling of the process is added as a determinant factor both in project design and implementation. One further point that should be made is that in the 2008 exercise, it was not necessary to add new separable factors to properly describe the enlarged sample. The earlier set of 10 secondary factors was complete and robust enough to be applied to the updated analysis. This also facilitates comparing the results of this exercise with previous studies in the 2004 and 2008 AEORs.

Table 3.1: Description of factors affecting performance

Main factors

Secondary

factors

Comments

Financial analysis Quality of financial analysis as part of project appraisal 1/

Financial

Cost performance Actual costs versus expected costs at appraisal Sales performance Actual sales of the client company versus projected

sales Market analysis Understanding of demand and competition, as part of

project analysis at appraisal

Commercial

Competition On both quality and price of the product 2/

Sponsor commitment

Local or international sponsor

Management skills

Including senior management skill, experience and entrepreneurship 3/

Institutional

Governance Quality of corporate governance at Board level External Business cycle Extended to include other external shocks such as

financial crises, natural disasters and conflicts (when not directly attributable to the government)

Government behaviour

Positive or negative government interference with client’s implementation of the project

Notes:

Bank handling The EBRD’s management of the project at appraisal

and implementation, including quality of relations with the client 4/

1/ Identified in OPER reports mostly when the project appraisal is clearly off track. 2/ Also increased market share of company in presence of competitors. 3/ Plus effect on improved organisation of company when observed. 4/ Relies upon either on Good (and better) or Marginal (and worse) OPER ratings for Bank handling.

25

3.1.2 Summary statistics

The descriptive statistics for the main performance ratings, assigned to each project in an OPER report, show that on average most projects are overall rated somewhere between Successful and Partly Successful, with Successful being the most common rating (see Table 3.2).21 For most other ratings, projects are on average rated Satisfactory and Good, with Good being the most common rating.22

Table 3.2: Summary statistics of evaluation ratings

Source: OPERs and EvD calculations.

Furthermore, as one would expect, the performance ratings are highly correlated with each other (see Table A.2.2, Appendix 10). The strongest between-ratings correlation is between company and project financial performance. Overall performance is highly correlated with the transition impact and fulfilment of objectives, with readings of 0.85 and 0.86 respectively, although financial performance and Bank performance are also important co-variants. The two environmental ratings, with the exception of their own cross-correlation, are the exception to the rule of strong correlations across ratings.

Bank handling is highly correlated with overall performance, transition impact and fulfilment of objectives, while it is slightly less correlated with net factor balance,23 company financial performance and project financial performance.

Net factor balance is highly correlated with many of the performance ratings; indeed, the correlation with overall performance is almost perfect (see Charts A.2.1 and A.2.2). A histogram on Chart A.2.1 shows a breakdown of net factor balances across the entire sample,

21 In the translation of ratings to numerical values in Table 3.2, a higher number indicates a higher rating. Hence, for overall performance, a “Highly Successful” project scores 4 and an “Unsuccessful” project scores 1. 22 In the analysis the following numbers are associated with the rating scores: Highly Successful=4, Successful=3, Partly Successful=2 and Unsuccessful =1 for overall performance; Outstanding=4, Substantial=3, Some=2 and None/Negative=1 for environmental change; for all other performance categories: Excellent=6, Good=5, Satisfactory=4, Marginal=3, Unsuccessful=2 and Highly Unsuccessful=1. 23 Net factor balance here denotes the sum of all positive and negative factors in a project, thus if a project has three positive factors and two negative factors, it would have a net balance of +1.

Frequencies

Mean Mode Median Standard deviation n/r 1 2 3 4 5 6

Overall performance 2.5 3.0 3.0 0.9 0 44 62 122 26 — —

Transition impact 4.2 5.0 5.0 1.3 0 11 19 37 56 104 27

Environmental performance

4.4 5.0 5.0 1.0 6 0 10 37 72 95 34

Environmental change 2.1 2.0 2.0 0.8 6 55 112 70 11 — —

Company financial performance

4.0 5.0 4.0 1.5 1 23 19 33 71 73 34

Project financial performance

3.9 5.0 4.0 1.5 1 27 20 37 68 69 32

Fulfilment of objectives 4.2 5.0 4.0 1.4 0 15 16 39 58 88 38

Bank handling 4.3 5.0 5.0 1.2 0 11 13 31 62 108 29

Investment performance 3.8 4.0 4.0 1.5 2 26 17 37 97 41 34

26

which confirms the hypothesis of factors contributing to success/failure of projects. For example, one would expect Unsuccessful projects to have a net factor balance of approximately -4. Putting the correlation between overall performance and the net factor balance in a bubble chart24 shows that the upward trend between the net factor balance and project performance is clearly observable. However, the more subtle conclusion is that the net factor balance appears to have an almost symmetrically distribution close to each different project rating and this is supportive of the distributional assumptions necessary to conduct the econometric analysis.

Looking beyond just the sum of the factors, the prevalence of positive individual factors across Successful and Highly Successful projects, broken down by industry groups, shows a similar picture to that of the net factor analysis in the AEOR 2008 (see Table 3.3).

Table 3.3: Breakdown of positive factors affecting performance for Successful and Highly Successful projects

Categories determining outcomes Financial institutions Corporate Energy Infrastructure

Number of projects 41 56 18 33 148Financial analysis 10% 9% 0% 3% 7%Cost performance 2% 18% 11% 27% 15%Sales performance 27% 34% 17% 9% 24%Market analysis 29% 21% 6% 15% 20%Competition 27% 41% 11% 15% 28%Sponsor commitment 24% 64% 39% 21% 41%Management skills 68% 57% 50% 58% 59%Governance 37% 30% 39% 39% 35%Business cycle 2% 7% 0% 0% 3%Government behaviour 10% 11% 28% 15% 14%

Bank handling 73% 45% 61% 55% 57%

Commercial factors

Institutional building factors

External factors

Sector

Average across sectors

Financial factors

Sources: EvD database and calculations.

Noticeably, the concentration of occurrences is on the institutional building factors, especially management skills. Bank handling is also a recurrent factor contributing to the success of the projects with an occurrence rate of 57 per cent of Successful and Highly Successful projects. This rate increases to 73 per cent for successfully implemented financial sector projects. Competition is the most prevalent of the commercial factors with an occurrence rate of 28 per cent. However this jumps to 41 per cent of the projects in the corporate sector.

For Unsuccessful or Partly Successful projects, the prevalence in the negative factors is more evenly distributed than in the case of the successful projects (see Table 3.4).

24 See Chart A2.2 of Appendix 10. The size of the bubbles is proportional to the number of projects – in fact, the chart is a graphical representation of a three-dimensional frequency table.

27

Table 3.4: Breakdown of negative factors affecting performance for Partly Successful and Unsuccessful projects

Categories determining outcomes Financial

institutions Corporate Energy Infrastructure

Number of projects 21 47 20 18 106Financial analysis 24% 26% 10% 33% 24%Cost performance 0% 34% 40% 33% 28%Sales performance 0% 40% 10% 28% 25%Market analysis 14% 28% 20% 17% 22%Competition 10% 19% 0% 0% 10%Sponsor commitment 14% 23% 25% 6% 19%Management skills 62% 45% 45% 28% 45%Governance 57% 34% 20% 17% 33%Business cycle 43% 26% 5% 22% 25%Government behaviour 5% 23% 50% 67% 32%

Bank handling 29% 36% 35% 33% 34%

SectorAverage across sectors

Financial factors

Commercial factors

Institutional building factors External factors


Once again, a concentration of occurrences is on the institutional building factors, especially management skills. Bank handling is also important, with a 34 per cent occurrence rate. The role of government appears also disruptive, with a third of Unsuccessful or Partly Successful projects suffering a negative government behaviour factor. This is particularly common in the energy and infrastructure sectors.

3.1.3 Econometric methodology

Looking at the prevalence of factors is a simple way to understand their basic distribution and to characterise what sort of factors an average, say, Successful infrastructure project has. It is not possible to answer the more nuanced question of what is the marginal effect of a particular factor – that is, which factors most strongly influence a project’s rating – without turning to econometrics for analysis of the data.

The econometric methodology used is known as the “Ordered Logit” model, which can answer questions on the marginal effects of factors.25 First, it is important to understand that the model is based on the fact that there is a set of ordered ratings that describe the variable in question. For example, in this assignment overall performance is described by the ratings Highly Successful through to Unsuccessful. Indeed, all the variables of interest in this assignment are based around an ordered rating of some form, as this is EvD’s method of rating characteristics of a project, which are fundamentally unquantifiable. This is what makes the Ordered Logit approach so appropriate for this analysis.

The model essentially asks, given a set of explanatory variables, what is the probability of a project achieving a certain rating (as mentioned above, project performance ratings are assigned by EvD in evaluation reports). What is the appropriate set of explanatory variables for this assigning a certain rating? The factors themselves are clearly appropriate to try to explain aspects of a project’s performance; however, a problem emerges if other project ratings are included. For example, if the probability of overall project performance was regressed on both the factors and the fulfilment of objectives rating, interpreting the results 25 Appendix 10 discusses the mathematical and statistical underpinnings of the model in more detail as well as the exact method to calculate the marginal effects, but it is worth making some aspects of this model clear from a non-technical standpoint.

28

would be difficult. This is because, as shown in Section 3.3, overall project performance and fulfilment of objectives are highly correlated (and thus there is an endogeneity problem) and it is impossible to say which way the causation runs.

As a result, any idiosyncratic shock to overall performance may also be reflected in fulfilment of objectives, and this means that the results could be biased. In the lexicon of econometrics this is known as an endogeneity problem. There are ways to correct for this, and indeed the construction of an exogenous set of factors is one such approach, but the data set is not rich enough neither in the number observations nor the number of rating categories to use more sophisticated methods to allow the inclusion of clearly endogenous variables. Thus, all of the analysis was conducted with the set of explanatory variables limited to the 10 exogenous factors affecting performance.

Marginal effects can then be calculated by asking how the probability of a project, being assigned a particular ranking, changes when a factor is added or removed. Note, to allow for a more parsimonious model the set of factors has been split between positive and negative factors. Therefore, there is a difference in marginal effects between adding a negative factor and subtracting a positive one.

Finally, the change in the probability of a particular rating by adding a factor depends on what factors a project already has. For example, if a project has a lot of negative factors adding one positive factor has a negligible effect on the probability of a project being successful, while if the project has no other factors, the additional positive factor may be quite important. The form of the model used here provides us with a tool to address this. The Ordered Logit model requires the estimation of “cut-off points” where a project is on a cusp between one rating and another. That is to say, at the cut-off between Partly Successful and Successful, a project will have a 50 per cent chance of being rated Successful or Highly Successful and 50 per cent chance of being rated Partly Successful or Unsuccessful. These cut-off points are useful in the sense that they allow for platforms from which to assess the marginal effects.

Therefore, the question that is considered in the results is: what is the marginal effect of adding/removing a factor at a particular cut-off point?

3.1.4 Econometric results

Here we present the results of econometric analysis for the overall performance ratings, from nine main OPER ratings.26 Overall performance appears by far the most significant. The results for project financial performance and company financial performance are nearly identical. The results for Bank performance are almost entirely down to the Bank handling factor; almost all the remaining factors are insignificant.

3.1.5 Overall performance

The key results for the model for overall project performance are summarised by positive and negative factors (see Charts 3.1 and 3.2). First, we look at what happens to the probability that a project achieves a Successful rating when an additional positive factor is added, and the

26 These are overall performance, transition impact, fulfilment of objectives, project financial performance, company financial performance.

29

project is at the cut-off between a Partly Successful and Successful ratings (see Chart 3.1).27 The first result is that both competition and cost performance do not have a significant impact on the probability of a project being successful. This is in contrast to the market analysis factor, which raises the probability by 41.5 per cent. At this cut-off point, the probability of being a Successful project is 50 per cent anyway, so adding a market analysis factor raises this probability to over 90 per cent. Bank handling, government behaviour and financial analysis have slightly weaker impacts, although they are of the same order of magnitude of around 40 per cent.

Chart 3.1: Change in probability of a successful project by adding a positive factor28

15%

20%

25%

30%

35%

40%

45%

Market

analy

sis

Bank H

andli

ng

Govern

ment b

ehav

iour

Manag

emen

t skil

ls

Financ

ial A

nalys

is

Spons

or Com

mitmen

t

Corpora

te go

verna

nce

Sales P

erform

ance

Compe

tition

Cost P

erform

ance

Cha

nge

in p

roba

bilit

y Estimated Impact

Sigificance Level


We follow the same principle when considering what happens when a negative factor is added as opposed to a positive one (see Chart 3.2). As before, cost performance is the least significant factor, and in this case the estimated effect even has a wrong sign. The remaining factors are all significant, with management skills having the strongest effect. Financial analysis, market analysis and government behaviour also appear to be particularly important as is the case with the results of adding a positive factor. On the other hand, Bank handling has become relatively less important as a negative factor than as a positive one.

In general, there is less heterogeneity across the marginal effects for negative factors compared with that for positive factors. The negative factors mostly have marginal effects of a similar order of magnitude, in the 30-40 per cent range. The standard errors of these estimated marginal effects are roughly 10 per cent, thus most of the small differences between the marginal effects of negative factors are not statistically significant. The exception of this is cost performance. Abstracting from this factor, it is possible to say that when a project acquires a negative factor, it does not matter too much what type of factor it is for the impact on the overall performance. In contrast, the factor type is more important when a positive factor is added.

27 Note that in these charts, the black line represents the critical value of a one-sided five-per cent significant level test on the estimated change in the probability. Hence, for the change to be statistically different from zero, the black line must be contained within the bar. 28 Project at the cut-off between Successful and Partly Successful.

30

Chart 3.2: Change in probability of a successful project by adding a negative factor29

‐50%

‐40%

‐30%

‐20%

‐10%

0%

10%

20%

Manag

emen

t skil

ls

Financ

ial A

nalys

is

Market

analy

sis

Govern

ment b

ehav

iour

Compe

tition

Spons

or Com

mitmen

t

Sales P

erform

ance

Busine

ss C

ycle

Corpora

te go

verna

nce

Bank H

andli

ng

Cost P

erform

ance

chan

ge in

pro

babi

lity

Estimated ImpactSignificance Level


The results for other cut-off points are very similar to those described above. Thus, the same conclusions can largely be drawn.

A final point to recognise is that the cut-off points are far away from each other. Thus, a project that is on a cut-off point between Successful/Partly Successful has almost a zero probability of being Highly Successful or Unsuccessful and, more importantly, adding any one additional factor has a negligible impact on the probability of either a Highly Successful or Unsuccessful ratings. This applies equivalently to other cut-off points.

There are two reasons for this. First, there are only four ratings of project overall performance, hence, by definition, the step between each rating is large and therefore it is very unlikely that a project can easily jump across two ratings with just a small change (that is, a change of just one factor) in its attributes. Second, the factors are powerful explanatory variables that capture the majority of the variance of the overall performance, thus the stochastic element of the rating is much smaller. This means that the probability distribution tends to be more densely distributed about the point estimated by the factors. In other words, if the project has the factors estimated to be consistent with a particular project rating, it is almost certain to have that rating.

Transition impact

The behaviour of transition impact rating in its response to the factors contrasts to that of the overall performance rating. The main difference comes from the fact that transition impact has six rating categories as opposed to four. It appears that the explanatory power of the factors is weaker. Hence, there is much greater variance in the probability distribution of a particular rating. This makes the representation of marginal effects for the transition impact regression more difficult, as there is more variation in what adding a factor does to the probability of a particular rating.

29 Project at the cut-off between Successful and Partly Successful.

31

As in the previous case, first, we look at what happens to the probability of a Satisfactory transition impact when an additional positive factor is added at the cut-off point between Marginal and Unsatisfactory (see Chart 3.3).30 Since the cut-off points are much closer together in this model, adding an additional factor raises the probability of a Satisfactory rating and has insignificant effects on the probability of a Marginal rating. In effect, there is a transfer in the net balance of probabilities away from an Unsatisfactory rating to a Satisfactory one. In terms of individual factors, the explanatory powers of the factors have weakened when compared with those for the overall performance regressions, with five factors having an insignificant effect. Bank handling stands out as being particularly important, with a marginal effect of 25 per cent. Sponsor commitment is slightly weaker at 30 per cent, followed by market analysis at 25 per cent.

Chart 3.3: Change in probability of a satisfactory transition impact given an additional positive factor31

‐10%‐5%0%5%10%15%20%25%30%35%40%

Bank H

andli

ng

Spons

or Com

mitmen

t

Market

analy

sis

Manag

emen

t skil

ls

Govern

ment b

ehav

iour

Sales P

erform

ance

Cost P

erform

ance

Compe

tition

Corpora

te go

verna

nce

Financ

ial Ana

lysis

Cha

nge

in p

roba

bilit

y

EstimatedImpactCritical value


The equivalent results for adding a negative factor show that the overall marginal effects are much weaker, with the largest being eight per cent (see Chart 3.4). This is because the addition of a negative factor at this cut-off point is essentially a transfer of probabilities away from a marginal rating to a negative rating (see Chart 3.5). The probability of being satisfactory is already quite low at this cut-off point, thus adding an additional negative factor does not make a difference.

30 As with the overall performance model, the results are broadly similar, but not identical, when different cut-off points are reviewed as long as we keep the relative positions of the ratings that are being compared the same. 31 Project at the cut-off between Marginal/Unsatisfactory.

32

Chart 3.4: Change in probability of a satisfactory transition impact given an additional negative factor32

‐9%‐8%‐7%‐6%‐5%‐4%‐3%‐2%‐1%0%1%

Financ

ial A

nalys

is

Govern

ment b

ehav

iour

Manag

emen

t skil

ls

Busine

ss C

ycle

Market

analy

sis

Sales P

erform

ance

Bank H

andli

ng

Compe

tition

Cost P

erform

ance

Corpora

te go

verna

nce

Spons

or Com

mitmen

t

Cha

nge

in p

roba

bilit

y

Estimated ImpactCritical value


Chart 3.5: Change in probability of a transition impact given an additional negative factor33

Marginal transition impact Negative transition impact

‐25%

‐20%

‐15%

‐10%

‐5%

0%

5%

Financ

ial A

nalys

is

Govern

ment b

ehav

iour

Manag

emen

t skil

ls

Busine

ss C

ycle

Market

analy

sis

Sales P

erform

ance

Bank H

andli

ng

Compe

tition

Cost P

erform

ance

Corpora

te go

verna

nce

Spons

or Com

mitmen

t

Cha

nge

in p

roba

bilit

y

Estimated Impact

Critical value ‐5%0%5%10%15%20%25%30%

Financ

ial A

nalys

is

Govern

ment b

ehav

iour

Manag

emen

t skil

ls

Busine

ss C

ycle

Market

analy

sis

Sales P

erform

ance

Bank H

andli

ng

Compe

tition

Cost P

erform

ance

Corpora

te go

verna

nce

Spons

or Com

mitmen

t

Cha

nge

in p

roba

bilit

y Estimated Impact

Critical value


There are large differences to the types of negative factors that affect transition impact ratings when compared with the effect of positive factors. Financial analysis has the largest marginal effect (see Chart 3.5), while its addition as a positive factor was insignificant. In contrast, while Bank handling was an important positive factor, it is insignificant as a negative factor, and the same applies to Sponsor commitment. However, management skills, market analysis and government behaviour are important negative and positive factors.

3.1.6 Concluding remarks

Based on a comprehensive database of projects evaluated by EvD in 1996-2009, this chapter analysed factors contributing to project success/failure using simple statistics and econometric modelling (the Ordered Logit method).

32 Project at the cut-off between Marginal/Unsatisfactory. 33 Project at the cut-off between Marginal/Unsatisfactory.

33

The main conclusions of this chapter broadly concur with those of the previous studies (AEOR 2004 and 2008) for the overall project performance:

• The main factors contributing to the project being successful are both internal (market analysis and financial analysis) and external (government behaviour).

• The same factors, together with management skills, appear to be important negative factors pushing the overall project performance down.

• Bank handling is relatively less important as a negative factor than as a positive one.

With regards to transition impact, explanatory power of factors affecting performance becomes weaker compared with that for the overall performance:

• Bank handling stands out as being a particularly important positive factor for project success, however it is insignificant as a negative factor.

• Sponsor commitment has a weaker but still significant effect as a positive factor, but again becomes insignificant as a negative factor.

• Market analysis, management skills and government behaviour are both important negative and positive factors.

All the above factors appear in most of the EvD reports and lessons learned, underscoring the importance of areas requiring particular attention from the Bank:

• In government-sensitive sectors, projects should be pursued with full agreement and continuous active policy dialogue with country authorities at all levels.

• Good governance remains an important source of long-term project success.

• Enhancing competitive environments supports project success.

3.2 COMPARISON OF PROJECT PERFORMANCE BY SIZE

In the AEOR for 2009, the Evaluation Department mentioned that larger projects tend to perform better than small ones in three dimensions: overall performance, transition impact and financial performance. The tendency had been observed over a number of years. This finding provoked interest in the Audit Committee and, in response, the Evaluation Department prepared a paper during 2009 that investigated the findings in more depth. It consisted of an analysis of the performance of large and small projects across performance indicators, geographic regions and industry sectors. It also reported on a more qualitative analysis of the projects in terms of factors of project success or failure, and appended a number of case studies of a cross-section of individual projects. The paper was circulated to the Board of Directors as input to the discussion on the Capital Resources Review (CRR4) and is attached at Appendix 9.

This section of the AEOR for 2010 updates the numerical results to the end of 2009 but does not repeat or update the qualitative analysis and case studies from the later part of the paper. The sample used for the analysis is the same as in Chapter 1 and Appendix 8 of this report: 679 investment projects evaluated in 1996-2010. The sample contains 247 smaller projects, defined as having less than €10 million of EBRD-approved finance; 333 medium projects,

34

with at least €10 million but less than €50 million; and 99 large projects, with at least €50 million of EBRD-approved finance. Investments made through vehicles such as the Direct Investment Facility, Direct Lending Facility and Western Balkans Initiative are not included.

3.2.1 Overall results

Chart 3.6 shows that it is still the case that larger projects have better overall performance ratings: 71per cent of larger projects were rated Highly Successful or Successful compared to 40per cent of smaller projects. Of the 61 evaluations added to the database in 2009, the results are not entirely in line with this pattern. Only nine larger projects were evaluated in 2009 and only three (33per cent) of these were rated Successful or better. However, with such a small number of evaluations the results are decided by a handful of cases and cannot be considered representative. Among the 27 medium projects the figure was 59per cent and among the 25 smaller projects it was 44per cent, closer to the overall results shown in Chart 3.6.

Chart 3.6: Overall performance rating by project size:

679 projects evaluated 1996-2009

12%4%

61% 53%

36%

19% 26%

40%

10% 9%20%

10%0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Large (99) Medium (333) Small (247)

Size

% o

f eva

luat

ed p

roje

cts


The results for transition impact and financial performance are shown in Charts 3.7 and 3.8. The pattern of higher ratings for larger projects has been maintained in 2009. Among the 61 evaluations conducted in 2009, the medium projects were rated more highly than smaller projects for both these indicators. Of the nine larger projects evaluated in 2009, seven were rated Satisfactory to Excellent for financial performance and seven for transition impact. In the case of transition impact, this was the same proportion as among medium projects; in the case of financial performance, the larger projects were the best performing group, although the small numbers mean that firm conclusions cannot be drawn.

35

Chart 3.7: Transition impact Chart 3.8: Project financial performance

13% 5%

53% 48%

38%

21% 23%

26%

9% 10%

20%

9%

11%

4%3% 2%2%3%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


Size

% o

f eva

luat

ed p

roje

cts

Excellent Good SatisfactoryMarginal Unsatisfactory Negative

14% 10%

38%

32%21%

24%

27%

21%

7%17%

25%

6%13%

5% 10%

22%

4% 3%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


Size

% o

f eva

luat

ed p

roje

cts

Excellent GoodSatisfactory MarginalUnsatisfactory Highly Unsatisfactory

3.2.2 Results by country groups

Charts 1.2 and 1.4 in Chapter 1 show that projects in the CEB and SEE regions continue to achieve the highest ratings for overall performance and transition impact. This is also the case for financial performance, as shown in Chart 5.3 of Appendix 8. CA and EEC are the worst performing regions for each of these indicators. Chart 3.9 below shows the distribution of evaluated projects across these regions.

Chart 3.9: Regional distribution of 679 projects evaluated 1996-200934

44%35%

51%

23% 26%42%

42% 53%

43%

44%55%

45%

14% 12% 6%

33%19% 13%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

CA CEB EEC Regional RUS SEE

Region

% o

f eva

luat

ed p

roje

cts

Small Medium Large

We might expect to see a large number of smaller projects in the poorly performing regions and a higher proportion of larger projects in the better performing regions. It is the case that the regions with the highest proportion of smaller projects are the two lowest achieving regions, CA and EEC. However, SEE, the best performing region of all, also has a relatively high proportion of small projects. Excluding Regional projects, which are not considered in the regional analysis, Russia has the lowest proportion of smaller projects and the highest

34 CA: Kazakhstan, Kyrgyz Republic, Tajikistan, Turkmenistan, Uzbekistan. CEB: Croatia, Czech Republic, Estonia, Hungary, Latvia, Lithuania, Poland, Slovak Republic, Slovenia. EEC: Armenia, Azerbaijan, Belarus, Georgia, Moldova, Ukraine. SEE: Albania, Bosnia & Herzegovina, Bulgaria, FYR Macedonia, Montenegro, Romania, Serbia.

36

proportion of larger projects, but this is not reflected in very high ratings. Therefore it cannot be concluded that the difference between performance of larger and smaller projects is directly related to the regional distribution, although there seems to be some relationship in the case of the more poorly performing regions.

3.2.3 Results by sector

Charts 2.4 and 3.5 of Appendix 8 both show that there is little difference among sectors in terms of overall performance and transition impact ratings. There is more difference in project financial performance ratings, as shown in Chart 5.5 of Appendix 8. Projects in the Energy and Infrastructure sectors are more likely to be rated Satisfactory or better than those in the Corporate and Financial sectors. This may be because Energy and Infrastructure projects are more likely to have public sector involvement and be less open to market forces.

Chart 3.10 below shows the distribution of evaluated projects across sectors. It does not show any clear relationship to the distribution of performance ratings by sector.

Chart 3.10: Sectoral distribution of 679 projects evaluated 1996-200935

48%36%

6%

34%

42%51%

63%

50%

11% 13%32%

16%

0%10%20%30%40%50%60%70%80%90%

100%

FinancialInstitutions

Corporate Energy Infrastructure

Sector

% o

f eva

luat

ed p

roje

cts

Small Medium Large

3.2.4 Conclusions

The Evaluation Department does not find any reason to revise the conclusions of the original paper based on this update. Although the results from the 61 projects evaluated in 2009 show a number of larger projects with rather low performance ratings, the numbers (only nine larger projects) are too small to draw firm conclusions. To date, the cumulative results confirm that larger projects are generally rated more highly than small projects. Further analysis to find the main reasons for this conclusion will be carried out.

35 Corporate = agribusiness, general industry, commercial services, property/tourism, and telecommunications. Energy = power and energy, and natural resources. Infrastructure = municipal/environment, and transport.

37

4. EVALUATION OF TECHNICAL COOPERATION OPERATIONS

4.1 TC EVALUATION COVERAGE

4.1.1 Introduction Technical cooperation (TC) activities are primarily used to facilitate the EBRD’s core investment operations and enhance the fulfilment of its transition impact mandate. In compliance with its fiduciary responsibility towards the contributors to its Technical Cooperation Funds Programme (TCFP), the Bank’s main TC funding source, and to those of other funding facilities, the Bank is obliged to exercise the same attention for TC operations as it does for investments funded from the Bank’s own resources. Accordingly, TC operations are subject to a diligent appraisal, monitoring, and self-evaluation process. The results of these process steps are documented in: (a) the Technical Cooperation Request package to the TC Review Committee for the appraisal stage, notably including the TC Project Profile and consultant terms of reference; (b) the Project Progress Reports during monitoring stages; and (c) the Project Completion Report (PCR). The TC evaluation and self-evaluation process is presented in Diagram 1 below.

DIAGRAM4.1: THE PROCESS OF TC EVALUATION [QUERY: CAN’T CHANGE CAPITALISATION IN TITLE OR DIAGRAM]

LESSONS LEARNED DATABASE

TC coverage through self-evaluation

(by Banking Dept. & OCU) ******************* TC coverage through

independent evaluation (by EvD)

PCR

Assessment (20 cases selected

from recent PCRs)

TC OPER reports (6 reports)

Special Studies &

Mid-Term Reviews (6 reports)

Sample coverage

FEEDBACK MECHANISM

PCR Preparation Operation Leader

Banking Department

PCR Central File (OCU)

100% coverage

PCR Acceptance

(OCU)

PCR

Review (OCU)

EVALUATION OF TC OPERATIONS THE PROCESS

TC = Technical Cooperation PCR = Project Completion Report OCU = Official Co-financing Unit

38

In addition to the mandatory self-evaluation process for each TC operation, independent evaluations based on a sample of completed TC operations are carried out by EvD. Independent TC evaluation work falls broadly into three categories:

(a) In-depth evaluations of individual TC operations in the form of an Operation Performance Evaluation Review (OPER). Around six TC OPERs are completed per year, occasionally supported by consultant input.

(b) Special Study, often a Mid-Term Review of a TC fund or programme, which typically involves a field visit and is sometimes supported by a consultant. Since 2002, there have been six such Special Studies prepared per year, not all related to TC.

(c) Desk-study-type review of a group of PCR Assessments (which is also counted as a Special Study) has been conducted annually since 1998 involving around 20 TC projects. Through a review of available files and interviews conducted with the OL, it is attempted to verify the information base provided through the self-evaluation of the OL.

4.1.2 TC evaluation coverage by EvD Since 1993, when EvD started TC evaluation work, 82 OPERs and 30 Special Studies on sectors and themes have been carried out covering many TC operations. In addition, since 1998 a total of 11 PCR Assessment synthesis exercises have been completed. Between 1998 and 2002, EvD also prepared PCR Reviews of 40 assignments per year. From 2003, this role was taken on by OCU. Overall these reports, although very different in scope and evaluation focus, have covered over 1,750 TC-funded consultant assignments, involving approximately €549 million of funding from some 29 individual countries and 32 multilateral funds under the EBRD's TCFP.36 The total volume of evaluated TC operations based on an OPER exercise, as a percentage of the volume of TC operations with a completed PCR (see Table 4.1), has increased from 13.3 per cent in 1997, immediately before the PCR review and assessment work was introduced, to 27.2 per cent in 2009. If groups of TC commitments covered in Special Studies on sectors and themes are included, the coverage ratio rises to 65 per cent.

Table 4.1: Technical cooperation evaluation coverage status in 1991-20098 (€ million)

TC completion and PCR coverage

1991-1999

1991-2000

1991-2001

1991-2002

1991-2003

1991-2004

1991-2005

1991-2006

1991-2007

1991-2008

1991-2009

a. PCRs completed 236.5 302.8 364.8 424.4 491.8 547.4 623.1 694.4 746.2 802.2 849.1

b. TC operations evaluated through OPER reports

32.1 36.6 41.1 49.2 56.5 63.4 73.8 82.1 86.0 93.4 163.1

c. PCR assessments by EvD 8.7 13.8 18.9 23.1 27.9 34.5 40.5 42.7 45.8 49.2 49.2 d. PCR reviews by EvD 6.3 12.7 12.7 19.0 19.0 19.0 19.0 19.0 19.0 19.0 19.0 e. Total TC operations (b+c+d) 47.1 63.2 72.7 91.4 103.5 116.9 133.3 143.8 150.9 161.6 231.3

f. Evaluation coverage (b+c+d)/a (%)

19.9% 20.9% 19.9% 21.5% 21.0% 21.4% 21.4% 20.7% 20.2% 20.1% 27.2%

g. TC operations related to Evaluation Special Studies

93.4 106.4 121.7 160.7 174.5 175.3 239.6 297.9 308.6 314.1 317.6

h. Total TC operations evaluated (b+c+d+g)

140.5 169.6 194.4 252.1 278.0 292.3 373.2 441.8 459.4 475.7 548.9

i. Evaluation coverage (b+c+d+g)/a (%)

59.4% 56.0% 53.3% 59.4% 56.5% 53.4% 59.9% 63.6% 61.6% 59.3% 64.6%

Chart 4.1presents the information from Table 4.1 in graphic form. It shows the growth of TC evaluation since 1995. In addition to direct TC evaluation, EvD provides further important assessments of TC performance through the evaluation of investment operations which have an important TC component (for example, project preparation, implementation, and so on).

36 This represents about 43 per cent of total TCFP funding commitments or 36 per cent of cumulative TCFP funding mobilisation. As at 31 December 2009 the Bank was successful in mobilising €1,532.9 million of TC funding, of which €1,283.7 million was committed.

39Chart 4.1: Evaluation coverage of technical cooperation commitments for 1995-2009 (€million)

0

100

200

300

400

500

600

700

800

900

1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009Year (cumulative)

TC

com

mitm

ents

(€m

illio

n)

Completed TCs not covered by EvDSpecial Studies covered by EvDCovered by EvD (excluding Special Studies)

27.2%

64.6%

100%

When selecting TC operations for evaluation, EvD takes into account TCFP funding sources, sector distribution of evaluation work in general and lessons learned potential of TC operations. Appendix 12 highlights the contributions of donors to TC operations that have been evaluated by EvD through an OPER exercise. It shows that most of the donors with relatively high contributions to the Bank’s TCFP are adequately represented in the Bank’s evaluation activities through TC OPER exercises. 4.2 PERFORMANCE EVALUATION OF TC OPERATIONS

Performance outcomes of the evaluation of TC operations do not lend themselves to aggregation of overall evaluation results in the same way as investment operations, for two reasons. First, given their mainly “facilitating” role as noted earlier, not all TC results are meaningfully assessable in their own right upon TC completion. The full impacts often only come to full fruition in the wake of investment implementation and, hence, can be ascertained only at a later stage. Secondly, EvD does not select TC operations randomly, rather it selects TC operations for which an OPER report will be produced on the basis of size (individual or group of related TC operations exceeding €200,000), lessons learned relevance and potential, and other practical considerations (for example, country, sector, banking unit spread, more recent TC operations where direct beneficiary counterparts are assumed to be still with the TC recipient, and so on). EvD’s TC evaluation experience, nevertheless leads to the conclusion that the Bank has improved the preparation of TC operations in recent years. This can be attributed in part to the TC Review Committee, which reviews and approves all acceptable TC funding requests. An important role is played by the Official Co-financing Unit (OCU), aiming to increase the quality of TC operations through providing guidance to operation leaders and ensuring a systematic and unified reporting regime to the donors. In addition, the assistance provided by the Consultancy Services Unit (CSU) in terms of reviewing the terms of reference and helping with consultant selection in relation to the TC operations helps secure a good quality at entry. EvD enjoys a steady and cooperative dialogue with OCU and CSU for TC work, and regularly discusses individual issues and findings from its reports to find pragmatic solutions and ways of improvement. In addition, EvD experiences that there is a general notion that a more systematic provision of TC-related skills (that is, in project management) is desirable in the Bank. Responding to the recommendations from recent PCR Assessment exercises, OCU readily took the lead in developing a basis for a future training of bankers, including general project management techniques and internal process-related knowledge.

40

EvD has substantially contributed not only to the preparation of relevant training material, such as case studies from its evaluation experience, but also regularly presents lessons learned during the training sessions. In total, four training courses on TC issues have been conducted since June 2009, with some 45 participants. Another EvD recommendation that was taken on board was the elaboration of a more streamlined template for TC project progress and completion reporting. After due discussion of the new format with donors and operating staff, the new templates are introduced as of March 2010. The next PCR Assessment might be a good opportunity to judge on the application of the new template and whether any effects can be observed on the quality of reporting. In addition to these activities, a feedback mechanism was established between EBRD’s Evaluation Department (EvD) and Technical Cooperation Committee (TC Com) by which EvD comments on those project proposals that are linked to assignments previously evaluated, hence reminding the members of the TC Review Committee of related lessons learned. 4.3 TC-RELATED EVALUATION WORK IN 2009

The TC operations that were evaluated in 2009 through TC OPER exercises were funded by donor contributions from France, Italy, the EBRD-Central Asia Institution Building Fund, the Early Transition Countries Fund and the EU/EBRD SME Finance Facility. TC operations evaluated through Special Studies were funded by donor contributions from Belgium, Switzerland, the Early Transition Countries Fund, the Mongolian Cooperation Fund and EC-Tacis. These operations were approved between 1999 and 2008 and cover the following sectors: Central Europe Agency Lines (CEALS), co-financing lines and Regional Venture Funds (RVFs); community/social services; energy; extractive industries; finance; local authority services; manufacturing; and transport and storage. By TC type, they involved advisory services, project implementation and project preparation. 4.3.1 TC OPER reports Under its work programme for 2009, EvD carried out six TC OPER exercises. For the TC OPERs, the following ratings were assigned: • Bosnia and Herzegovina Regional Railway Project: Partly Successful • Dnipropetrovsk Municipal Water Corporate Development Support (Ukraine): Successful • Uzbekistan Telecommunications Regulatory Development Programme: Partly Successful • Tbilisi Public Transport Project Corporate Development Programme (Georgia): Unsuccessful • EU/EBRD SME Finance Facility Special Fund (Regional): Partly Successful • Georgian Gas Transmission Pipeline Rehabilitation: Successful Selected lessons learned from these TC operations are presented in Appendix 4. 4.3.2 Special Study on PCR Assessments. In 2009, the Evaluation Department completed the PCR Assessment exercise from the 2008 work programme. This was substantially delayed to allow collaboration with the Official Co-financing Unit (OCU). The 2008 exercise comprised both the review and assessment of PCRs. The review process, conducted by OCU, considered all PCRs prepared during the period under consideration and examined the quality of the PCR reports themselves. The assessment process, conducted by the Evaluation Department, considered a selection of 20 PCRs prepared during the period and addressed the performance of the underlying TC operation. The final report combined both the review and assessment exercises and therefore involved a significant amount of cooperation between the two departments. The 2009 exercise will be completed in 2010, after the Audit Committee review of the previous report.

41

Each of the PCR Assessments prepared by the Evaluation Department comprises a short desk review during which the responsible operations staff and OCU are consulted and short file reviews are made. The assessments aim at further strengthening the TC monitoring and evaluation system, by focusing on the following quality criteria:

• fulfilment of objectives • client commitment • Bank performance • consultant performance • contribution to the Bank’s investment • transition impact • donor visibility • lessons learned

425 VALIDATION BY EvD OF PERFORMANCE RATINGS ASSIGNED DURING SELF-

EVALUATION

5.1 THE SELF-EVALUATION PROCESS AND VALIDATION OF RATINGS BY EvD

When a project is ready for evaluation, the operation team prepares a self-evaluation document, the Expanded Monitoring Report (XMR). The XMR builds on the basic monitoring report (MR) by adding information requirements that are relevant for a self-evaluation document (for example, relating to achievement of objectives, environmental performance, transition impact, lessons learned and overall assessment). The operation team provides a qualitative description of the performance of the project and assigns a performance rating to each indicator. The evaluation that is conducted by EvD, which starts with a review of the XMR, may result in different performance ratings than assigned by the operation team (OT).

5.2 COMPARING THE RATINGS FROM THE SELF-EVALUATION AND THE INDEPENDENT EVALUATION PROCESS Ratings are provided for nine indicators: overall performance, transition impact, environmental performance, extent of environmental change, additionality, project financial performance, company financial performance, fulfilment of objectives and Bank handling. This analysis covers the 255 projects evaluated in the last five years for which a full XMR was provided.37 For this analysis, it is necessary to have a clear rating assigned both in the XMR and in the OPER and the XMR Assessment reports.38 In contrast to the analyses in previous AEORs, EvD opted to cut off the figures from projects evaluated more than five years ago, so that the results are not overly influenced by very old results. 5.2.1 Proportion of ratings amended by EvD Table 5.1 and Chart 5.1 show that while the majority of XMR ratings remained unchanged, (60 per cent of all the ratings compared), 34 per cent were downgraded. The greatest number of downgrades was on environmental performance (46 per cent), followed by transition impact, company financial performance and Bank handling (all 40 per cent). The overall performance rating was downgraded for 30 per cent of projects. Far fewer ratings were upgraded: only six per cent in total. The highest number of upgrades occurred in relation to environmental change, which also displays the smallest number of downgrades.

Table 5.1: Differences in performance ratings between self-evaluation and independent evaluation, 2005-2009

Indicator Upgraded by EvD

Unchanged by EvD

Downgraded by EvD

Number of comparisons

Overall performance 4% 66% 30% 254 Transition impact 7% 53% 40% 255 Environmental performance 6% 48% 46% 240 Extent of environmental change 12% 69% 20% 235 Additionality 7% 74% 19% 255 Company financial performance 6% 54% 40% 249 Project financial performance 4% 58% 38% 250 Fulfilment of objectives 5% 61% 34% 255 Bank handling 4% 56% 40% 254 All ratings 6% 60% 34% 2247

37 There have been a few occasions where this did not occur, for example if the project was in corporate recovery and an XMR was

not practical, if a single evaluation covered several linked projects with separate XMRs, or if a project was completed and the relevant staff left the Bank before an XMR was completed, in which case a briefer memo from portfolio management staff might be accepted instead.

38 A number of XMRs or evaluation reports have individual performance categories rated "Not applicable" to the project under review and these missing ratings could not be assigned retrospectively by EvD. Therefore the figures presented below clearly indicate the number of comparisons that were possible to make for each indicator, and these vary from one indicator to another.

43

Chart 5.1: Graphic representation of the differences in performance ratings between self-evaluation and independent evaluation, 2005-2009

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Over

all pe

rform

ance

(254

)Tr

ansit

ion I

mpa

ct (2

55)

Envi

ronm

ental

Per

form

ance

(240

)En

viro

nmen

tal C

hang

e (23

5)Ad

ditio

nalit

y (25

5)Co

mpa

ny P

erfo

rman

ce (2

49)

Proj

ect P

erfo

rman

ce (2

51)

Objec

tives

(255

)Ba

nk H

andl

ing (

254)

% o

f eva

luat

ed p

roje

cts

XMR ratingsupgraded by EvD

XMR ratingsdowngraded byEvD

XMR ratings leftunchanged by EvD

It is not unexpected that Bank handling might be downgraded more often than other indicators, since operations staff are asked to judge their own and their colleagues' performance directly. Transition impact has shown up in the past as the indicator that was most frequently marked down. This has been discussed at some length in previous AEORs; one possibility may be simply that this indicator is considered the most influential in determining the overall performance rating and therefore bankers are likely to aim for the best possible rating. More unexpected are the results for environmental performance and company financial performance. The Environmental and Sustainability Department (ESD) signs off XMRs and therefore implicitly confirms the environmental ratings contained in them. In many cases they contribute to this section of the report. Therefore large differences between XMR and evaluation ratings for this indicator indicate differences of opinion between the Evaluation Department and ESD. This issue is being followed up between the two departments. The benchmarks for company financial performance are clearly defined in the Evaluation Policy and leave less room than some others for subjectivity. The performance is compared with the projections made at appraisal and the rating defined by whether the company has out-performed or under-performed those projections. It is the experience of EvD however that in difficult economic circumstances, operation teams tend to report on how well the company has performed in the circumstances – that is, how well it has performed in comparison with its peers. As a result of discussions between EvD and the Banking Department on this issue, a footnote was added to the new version of the Evaluation Policy approved in 201039 allowing evaluators more flexibility to take local conditions into account when rating this indicator. It is assumed therefore that the

39 Footnote 6 to Appendix 1 to the Update of the Evaluation Policy of the EBRD (BDS10-024). Please see Appendix 13 to this report.

44differences of opinion over company financial performance will diminish in coming years, and EvD will watch this. Tables 5.2 and 5.3 show figures separately for OPERs and XMR Assessments. The figures for OPERs include those operations evaluated through Special Studies, as these are evaluated to a comparable depth. It is clear that the evaluation concurs with the self-evaluation much more frequently in the case of XMR Assessments (75 per cent compared with 42 per cent). The more in-depth OPERs and Special Studies are more likely to amend the self-evaluation ratings either to raise or lower them. The majority of the indicators have a greater than 50 per cent chance of being downgraded in the case of an OPER evaluation. In all types of report, environmental performance is one of the most frequently downgraded indicators, although in OPERs Bank handling is slightly more likely to be marked down. In all cases, environmental change, additionality and overall performance are among those indicators least likely to be amended downwards.

Table 5.2: Differences in performance ratings between OPERs and independent evaluation, 2005-2009


Unchanged by EvD

Downgraded by EvD



Table 5.3: Differences in performance ratings between XMR Assessments and independent

evaluation, 2005-2009


Unchanged by EvD

Downgraded by EvD



A difference is to be expected, as the XMR Assessment does not allow an in-depth evaluation of the project. The evaluator is dependent on internal Bank documents and the views of Bank staff for the assessment, while an OPER allows greater opportunity to visit the site and discuss the project with individuals not forming part of the operation team. It may be difficult, for example, for an evaluator to decide that Bank handling was at fault without looking at the issues in more depth, while it may be easier to reach an assessment on an indicator such as fulfilment of objectives, which is based on the achievement of clear outcomes. Another factor explaining the differences is that OPERs are often selected for their lessons learned potential and high profile, and that among them one will find a higher proportion of more challenging projects where management has different opinions than EvD. In addition, when evaluation staff must sign off on an XMR on which they have to prepare an OPER report later, the scrutiny of the XMR will be less, as EvD’s in-depth analysis will take place

45when the OPER exercise is done. When preparing an XMR Assessment, however, evaluators tend to challenge bankers more thoroughly and the outcome is that downgrades are agreed during the assessment process before signing off the XMR. Therefore, the difference between EvD’s OPER ratings and XMR Assessment ratings is not necessarily alarming, but it is important to continue gathering these data and EvD will continue to report respectively in future AEORs. 5.2.2 Extent of changes to ratings Table 5.4 shows the maximum upgrade and downgrade for each rating category. In most cases the maximum upgrade for a given category is less than the maximum downgrade. Not only are ratings more likely to be downgraded than upgraded, but they are likely to be downgraded further.

Table 5.4: Overview of maximum downgrades and upgrades when comparing the differences between the self-evaluation and independent evaluation ratings, 2005-2009

Indicator Maximum upgrade (no. of rating points)

Maximum downgrade (no. of

rating points)

Number of reports with maximum downgrade

Overall performance 1 3 1 Transition impact 2 3 7 Environmental performance 2 4 1 Extent of environmental change 3 3 1 Additionality 2 2 11 Company financial performance 1 3 3 Project financial performance 1 3 4 Fulfilment of objectives 1 3 2 Bank handling 1 4 1 5.3 DEVELOPMENT OF THE OBSERVED DIFFERENCES OVER TIME

A study of the results over time shows the differences between XMR and evaluation ratings tend to become larger in recent years (see Charts 5.2-5.10). Variance has increased in 2009 for overall performance, environmental performance and additionality (although, in the last case, this followed a number of years where the differences were very low). In the case of some other indicators, notably transition impact and Bank handling, although the total proportion of downgraded reports has remained steady or fallen, the proportion adjusted by more than one grade has increased.

Charts 5.2-5.10: Differences over time in evaluation performance ratings assigned by operation staff and independent evaluators, 2005-2009 [QUERY: CAN’T EDIT CHARTS TO EDIT

CAPS]

Overall performance

0%

20%

40%

60%

1998-2000(131)

2001-04

(199)

2005(51)

2006(50)

2007(52)

2008(41)

2009(60)

Transition Impact

0%

20%

40%

60%

1998-2000(129)

2001-04

(198)

2005(51)

2006(50)

2007(52)

2008(41)

2009(61)

46

Additionality

0%

20%

40%

60%

1998-2000(131)

2001-04(198)

2005(51)

2006(50)

2007(52)

2008(41)

2009(61)

Fulfilment of objectives

0%

20%

40%

60%

1998-2000(129)

2001-04

(192)

2005(51)

2006(50)

2007(52)

2008(41)

2009(61)

Environmental Performance

0%

20%

40%

60%

1998-2000(127)

2001-04

(193)

2005(51)

2006(48)

2007(48)

2008(38)

2009(55)

Environmental Change

0%

20%

40%

60%

1998-2000(127)

2001-04(187)

2005(49)

2006(47)

2007(47)

2008(38)

2009(54)

Project Financial Performance

0%

20%

40%

60%

1998-2000(131)

2001-04

(192)

2005(50)

2006(50)

2007(50)

2008(40)

2009(61)

Company Financial Performance

0%

20%

40%

60%

1998-2000(131)

2001-04

(187)

2005(50)

2006(50)

2007(50)

2008(38)

2009(61)

Bank Handling

0%

20%

40%

60%

1998-2000(131)

2001-04

(192)

2005(51)

2006(50)

2007(52)

2008(40)

2009(61)

Key to charts

Downgraded by 1 point Downgraded by 2 pointsDowngraded by 3 points Downgraded by 4 points

The bracketed numbers shown beneath the years indicate the total number of comparisons made for that year, including XMRs upgraded or unchanged. The chart shows downgrades as a percentage of the total number of comparable reports for each year.

5.4 COMPARISON BY SECTOR

The Evaluation Department has compared the results by sector for two of the most commonly adjusted indicators: environmental performance and transition impact. The results are shown in Chart 5.11 and 5.12. The analysis shows a great deal of variation between sector teams. Chart 5.11 shows that the teams whose transition impact ratings were most frequently downgraded by EvD over the last five years were Natural Resources (71 per cent) and Municipal and Environmental Infrastructure (54 per cent). Both these teams featured among the top three in the analysis in the AEOR for 2009.

47Chart 5.11: Differences across sector teams in transition impact ratings

assigned by operation staff and independent evaluators, 2005-2009

0%

20%

40%

60%

80%

Financ

ial In

stitut

ions

Agribu

siness

Man

ufactu

ring a

nd Serv

ices

Munici

pal &

Env In

f

Natural

Reso

urces

Power

and E

nergy

Propert

y and

Touris

m

Teleco

ms Info

rmati

cs & M

edia

Transpo

rt

Downgradedby 3 points


Downgradedby 1 point

Chart 5.12 shows the corresponding results for environmental performance. The sector teams showing the greatest discrepancy are Municipal and Environmental Infrastructure (73 per cent) and Property and Tourism (62 per cent). The result for Municipal and Environmental Infrastructure is particularly worrying because this sector often has an environmental focus (particularly in water and wastewater treatment projects). It is notable that the other sectors associated with large potential environmental effects, Natural Resources, Power and Energy and Manufacturing and Services (which includes heavy industry) show less discrepancy between XMR ratings and evaluation ratings.

Chart 5.12: Differences across sector teams in environmental performance ratings assigned by operation staff and independent evaluators, 2005-2009

0%

20%

40%

60%

80%

Financ

ial In

stitut

ions

Agribu

siness

Manufa

cturin

g and

Service

s

Munici

pal &

Env In

f

Natural

Resourc

es

Power

and E

nergy

Propert

y and

Touris

m

Teleco

ms Info

rmati

cs & M

edia

Transpo

rt




Downgradedby 1 point

48 5.5 MAJOR CONCLUSIONS

Overall, XMR ratings over the last five years were validated by independent evaluation in 60 per cent of cases. Six per cent of XMR ratings were upgraded by evaluators and 34 per cent downgraded. These figures show a decline in results reported in previous AEORs, which considered ratings over the entire period from 1996. XMR ratings were much more likely to be downgraded by evaluators if they were subject to an OPER or Special Study evaluation than to a less in-depth XMR Assessment. This pattern is not necessarily cause for concern as the XMRs before EvD sign-off are discussed extensively with the bankers, resulting in more realistic ratings needing less downgrading. On the other hand, projects on which an OPER report will be prepared by EvD are selected based on lessons learned potential and high profile, and differences of opinion on the ratings between management and EvD are common. It is, however, important to look at those performance indicators and sectors where the differences are most significant. Environmental performance was the indicator most likely to be rated lower (46 per cent) by evaluators. Transition impact, company financial performance and Bank handling were also often marked down by evaluators. While a disagreement over Bank handling is to some extent to be expected, the rating of company financial performance leaves less room for subjective judgement, and it is hoped that a recent amendment to the Evaluation Policy, whereby evaluators can take into account local conditions, will reduce the differences in this area. The gap between XMR and evaluation ratings appears to be increasing, particularly in relation to overall performance and environmental performance. A closer analysis of two of the indicators showing the greatest proportion of downgrades shows considerable differences between sector teams. The Evaluation Department currently provides training for new bankers drafting XMRs for the first time, and uses such opportunities to warn against over-optimistic ratings. However, it has been noted in the past that the final decision on ratings in an XMR may be taken by more senior staff, who do not participate in these training sessions. Ongoing communication with senior Banking staff continues, mostly through the process of discussing draft OPER reports, and this issue will continue to be raised in that context. EvD has discussed the environmental performance ratings with the Environment and Sustainability Department, who confirmed that they would give additional focus in future to the review and rating of the environmental indicators in XMRs. In cooperation with the Managing Director for Monitoring, EvD is organising discussions with the sector teams showing the most substantial differences in relation to transition impact in particular.

49 6. ROLE OF THE BOARD’S AUDIT COMMITTEE IN OVERSEEING THE

EVALUATION FUNCTION

6.1 INTRODUCTION

This part of the report highlights how the Audit Committee, the Board Committee that oversees the evaluation function in the EBRD, has reacted to important evaluation findings and lessons learned. Section 6.2 lists the evaluation reports that have been discussed in the Audit Committee during 2009, which shows how the Committee has carried out its oversight function. In Section 6.3, EvD reviews some key issues that the Committee has discussed and which are reflected in the minutes of the Audit Committee meetings during 2009.40

6.2 REVIEW OF EVALUATION REPORTS BY THE AUDIT COMMITTEE

During 2009 the Audit Committee met 13 times to discuss in total 26 evaluation reports. Among them were 16 OPER reports on investment operations, three Special Studies, the Annual Evaluation Overview Report for 2009, three reports on EvD’s work programme and three other special reports. Details are presented in Table 6.1.

Table 6.1: Reports in respect of evaluation discussed in the Audit Committee during 2009

12 January 2009: Evaluation Special Study on:

- Danube River Basin, Regional 26 January 2009: Report on EvD’s work programme:

- Work Programme Final Report for 2009 Operation Performance Evaluation Review on:

- Togliatti Urban Transport Development Project, Russian Federation

23 February 2009 Operation Performance Evaluation Reviews on:

- Agrokor Equity Investment, Croatia 6 April 2009: Report on EvD’s work programme:

- Work Programme Completion Report for 2008 Operation Performance Evaluation Reviews (technical cooperation):

- Pre-privatisation loan for Kombinat Aluminiyuma Podgorica Environment, Health and Safety, Montenegro

27 April 2009: Evaluation Special Study on:

- Equity Exits, Regional Special reports:

- Lessons from the Financial Sector in an historic perspective

- Follow-up of evaluation recommendations by Management 2008

19 May 2009: Operation Performance Evaluation Reviews on:

- Road Sector Reform 1&2, Russian Federation - Almaty-Bishkek Regional Road Rehabilitation

Project, Kazakhstan and Kyrgyz Republic - RSB Term Securitisation, Russian Federation

1 June 2009: Evaluation Special Study on:

- Direct Investment Facility (DIF), Regional Operation Performance Evaluation Reviews on:

- Collaboration with Société Générale, Regional - Collaboration with UniCredit Group, Regional - Frontera Resources, Regional

24 June 2009: Operation Performance Evaluation Review on:

- Mittal Steel Skopje, FYR Macedonia - Federal Grid Company Modernisation, Russian Federation

7 July 2009: Special report:

- Annual Evaluation Overview Report for 2009 27 July 2009: Operation Performance Evaluation Review on:

- Relationship with Raiffeisen Bank, Regional 25 September 2009: Report on EvD’s work programme:

- EvD Work Programme Preliminary Report for 2010 Operation Performance Evaluation Review on:

- Mid-term review of Rosvodokanal, Russian Federation 5 October 2009: Operation Performance Evaluation Review on:

- EBRD/GEF Environmental Credit Line, Slovenia - Atyrau Airport Project, Kazakhstan

28 October 2009: Operation Performance Evaluation Review on:

- Tulpar II, Kazakhstan Special Report:

- Country Level Evaluation Approach Paper

40 Issues are addressed from Audit Committee deliberations in 2009 that will not be discussed in the report on “Follow-up of Evaluation Recommendations”, a joint report of EvD and management.

50

6.3 OBSERVATIONS ON EVALUATION FINDINGS AND LESSONS LEARNED AS PRESENTED IN THE MINUTES OF MEETINGS OF THE AUDIT COMMITTEE

The Committee concentrated on reviewing the evaluation recommendations in the evaluation reports on which EvD and Management jointly report on an annual basis through the “Follow-up of Evaluation Recommendations” report.41 However, other issues not dealt with in this report were also discussed by the Audit Committee. A selection of those issues, including some from the AEOR of 2009, is highlighted below:

• Country-level evaluation. The Work Programme Completion Report for 2009 and the Work Programme Final Report for 2010 referred to a country-level evaluation (CLE) Special Study whereby the evaluation exercise would concentrate on certain sectors and themes of existing country strategies. For the 2009 CLE exercise (to be finalised in 2010), the following themes/sectors were selected: in Russia, corporate governance in large enterprises and the Power and Energy sector; and in Romania, the financial intermediary sector. The 2009 CLE exercise is considered a pilot case in respect of the CLE methodology applied. Country-level evaluation is now incorporated in the Bank’s Evaluation Policy although further discussion on the methodology to be applied is expected in the course of 2010.

• The use of the log-frame approach. There was support in the Audit Committee for the

suggestion from EvD in one of its OPER reports to consider the use of the log-frame approach in assessing what the Bank wanted to achieve in projects. While it was not something that would change the way the Bank did business, it was considered a straightforward method of providing greater clarity up front on what was a project’s purpose beyond its immediate output and what were the necessary conditions for achieving that purpose. It was a simple model and could be used in the context of the EBRD’s business model generally. The Committee agreed that the use of the instrument should be taken into serious consideration by management and be discussed in the context of the discussion within the Bank on the integrated approach.

• Joint evaluation with other multilateral development banks (MDBs). As in previous

years the importance for the evaluation functions of the multilateral development banks to focus on joint evaluation was highlighted. The bottlenecks for the production of a larger number of joint evaluations were recognised and the efforts by the evaluators to overcome such obstacles through active coordination among MDB evaluators were appreciated. EvD continues doing landmark joint evaluations with the IFC, the IsDB, the ADB, and the EIB and continues promoting joint evaluation among the ECG members and observers. The increased importance of joint evaluation is again anchored in EvD’s Work Programme Report for 2010.

• Realistic transition targets in municipal and environmental infrastructure projects.

In one of the municipal and environmental infrastructure (MEI) projects presented to the Audit Committee it was considered adequate that EvD would normally base their evaluations on the original timeline and transition objectives of the project. The Committee felt that EvD’s emphasis on the importance of realistic timelines and transition targets and – to the extent possible – an adequate synchronisation of the Technical Assistance supported reforms and loan distributions was also essential in MEI projects.

41 Comments on the evaluation recommendations by the Audit Committee will be presented in document

Follow up of Evaluation Recommendations by Management 2010, which will be discussed in the Audit Committee and in the Board of Directors.

51

7. CHIEF EVALUATOR’S ASSESSMENT 7.1 PERFORMANCE OF THE EBRD ACTIVITIES BASED ON EVALUATION FINDINGS42 By evaluating its operations, the EBRD is able to assess its performance and account for its decisions. The Bank looks at the outcomes of policies and projects, determines how successful they were and tries to use these lessons to improve operations in the future. Of all the EBRD projects evaluated in the period 1996 to 2009, 79 per cent received an Excellent–Satisfactory rating on transition impact.

Impact on the transition process The share of projects with an Excellent–Satisfactory transition impact rating in 2009 was 75 per cent. In that year 25 per cent of evaluated projects were rated Marginal–Negative for transition impact. This is a slightly poorer result than the outcomes seen in the last few years, but still compares well with the period 1997-2002, when higher numbers of projects were rated Marginal–Negative. Projects evaluated during those years had been approved and implemented in the late 1990s when the economic climate in the region was more difficult than in recent years. This may have damaged the sustainability of some private sector projects and prevented them from realising their full potential. Current economic difficulties have begun to have a negative effect on evaluation results in 2009. The cumulative results for the transition impact of a total of 679 projects evaluated since 1996 show that 55 per cent achieved a rating of Good or Excellent and a further 24 per cent were assessed as Satisfactory. Weighting the results by volume of investment yields better outcomes, with 86 per cent Satisfactory or better in 1996-2009. Overall performance of the EBRD’s activities The overall performance rating gives a high weighting to transition impact but also includes other indicators, such as the fulfilment of project objectives, financial performance, environmental performance and additionality (the Bank’s ability to complement rather than replace private sources of finance). Since 1996, 57 per cent of evaluated projects achieved a rating of Successful or Highly Successful. Weighting the results by volume of investment gives a figure of 69 per cent Successful or Highly Successful over the same period. The proportion of projects rated Successful or Highly Successful has been falling since 2004, when they reached 73 per cent. In 2009 the corresponding figure was 51 per cent, the lowest level since 2002. The decline in ratings each year since 2004 was investigated in the AEOR for 2009, in which it was suggested that during the expansionary period of 2003-06, when there were large inflows of foreign direct investment (FDI) into the EBRD's countries of operations, the EBRD may have felt the need to take on more challenging projects in order to remain additional. The lower outcomes in some cases reflect the greater risk. In 2009, evaluated projects began to show the effects of the current economic turmoil, which was seen particularly in financial performance ratings. In respect of environmental performance the proportion of projects rated Good or Excellent has

42 The text in this section is taken from Chapter 10 “Evaluating EBRD Activities” in the Bank’s Annual Report of 2008.

52

been falling steadily since 2004. As the Bank strives towards excellence in respect of the environment, this trend should be carefully watched. Based on the above findings whereby transition impact shows continued positive results and the lower overall performance ratings demonstrate that the Bank operates in difficult environments, EvD concludes that the EBRD has been successful in operating according to its mandate. However, the downward trend since 2004 in the higher rating categories on environmental performance gives an indication that this is an area that should be watched. It is important to note that if performance outcomes are weighted by volume, higher scores are obtained, as is demonstrated in parts of the document. 7.2 REVIEW OF THE INDEPENDENCE OF THE EVALUATION DEPARTMENT As in previous years, it is essential to highlight the Evaluation Department (EvD) as an important accountability tool for the Board of Directors. In that respect, it is important to view the independence of the function in the context of the governance structure of the group of multilateral development banks (MDBs) and to assess how the independence of EvD has progressed in the EBRD. Details on the international standards on independence for evaluation functions are presented in the good practice standards (GPSs) on Independence of International Financial Institutions’ Central Evaluation Departments. These GPSs have been prepared by the Evaluation Cooperation Group (ECG).43

In the EBRD, the Evaluation Department became operational in the early 1990s, whereby the Head of Evaluation (now Chief Evaluator) reported to a Vice President not involved in operational activities. Following good practice standards, the Evaluation Department, at an initiative by the Board of Directors, became fully independent from management in June 2005. The independence of EvD is anchored in the Evaluation Policy of the EBRD, which was updated in 2010. It is stated Section 4 of the Evaluation Policy: “The Chief Evaluator is directly and only responsible to and only takes his/her instructions from the Board of Directors as a whole (and not from any Board committee, except as may be provided under the Terms of Reference of any such committee, or from any individual Board member). He/she is not part of management.” Now after five years of directly reporting to the Board of Directors and full independence from management, and taking into account the recently-approved Update of the Evaluation Policy of the EBRD, the Chief Evaluator concludes that independence of the Evaluation Department is functioning well. It is important to note that while the collaboration with management has improved over the past years, EvD manages to maintain the necessary rigor in its evaluation reports, which results in a constructive dialogue with management on key evaluation findings. The Chief Evaluator is of the view that the independence of the evaluation function continues to be very well secured in the recently Board-approved Update of the

43 The ECG constitutes a collaborative body in which the heads of evaluation departments in the MDBs collaborate and work towards harmonisation of evaluation procedures and practices. The ECG was established by the Heads of the evaluation departments in the MDBs in 1996.

53

Evaluation Policy of the EBRD (2010) and that in the oversight role of the Board through its Audit Committee, extensive attention is given to the evaluation findings and recommendations. On the side of management, the lessons learned are taken to heart during the process of project preparation and monitoring and the self-evaluation process remains of good quality. The Chief Evaluator concludes that the independence of EvD is excellently preserved in the Bank and that all stakeholders continue working well together to learn from past experience. Especially in a time of financial crisis, this collaboration has demonstrated to be essential. As presented in Chapter 6, the Audit Committee is extensively reviewing evaluation reports and contributes to the system of following up evaluation recommendations in an optimum way. During 2009 EvD contributed to the dialogue on CRR4 through preparing an evaluation of the implementation of CRR3. The President is also helping to preserve actively the independence of the evaluation function through securing an optimum functioning of the management commenting process on evaluation reports. The comprehensive and constructive management comments on evaluation reports and the interaction with EvD respectively can be seen as proof of that. The Evaluation Policy established in 2005 has in the mean time been updated and appears on the Bank’s website as “Update of the Evaluation Policy of the EBRD”. This policy update takes into account the following issues proposed in the AEOR for 2009:

1. In Section 2.4.4 and beyond, efficiency improvements were introduced through the introduction of a “short-form” of the Expanded Monitoring Reports in cases where an XMR Review is made by EvD and no more extensive evaluation product such as an OPER report or XMR Assessment is prepared; the “short-form” XMR can also be an efficient self-evaluation product in cases of frameworks. For all the projects that will be evaluated through an OPER report or XMR Assessment, “long-form” XMRs need to be produced by the Operation Team.

2. The system of Follow-up of Evaluation Recommendations approved by the Board of Directors in 2006 and fully implemented in 2007 was incorporated as Section 2.9.

3. In 2008 an improved selection process for the evaluation of projects ready for evaluation based on random sampling was developed and approved by the Audit Committee and incorporated in the work programmes of EvD since then. Section 2.4.5 on evaluation coverage of investment operations provides details on the selection process applied.

4. Section 3.3.5 on “Access to Information” by staff in the Evaluation Department, as concluded between management and EvD in September 2006, was incorporated.

5. The Update of the Evaluation Policy now also incorporates a selection procedure for the Chief Evaluator, which is highlighted in Chapter 4 on “Independence of the Evaluation Function”.

Other changes incorporated in the “Update of the Evaluation Policy of the EBRD” are the following:

1. Country-level evaluation (CLE) has been introduced in Sections 2.1 and 2.5.3, although the Board still needs to approve the methodology in respect of CLE based on a pilot Country-level Evaluation Special Study that is ongoing at the moment.

54

2. In Section 2.5.2, a common practice was introduced whereby before a new sector strategy is prepared by management, an evaluation must have taken place first. This enhances the learning process.

3. In Section 2.7, new text has been introduced whereby the Work Programme Preliminary Report, containing EvD’s budget, is discussed in back-to-back meetings of the Audit Committee and the Budget and Administrative Affairs Committee, after which the Board of Directors will decide on EvD’s budget in the same meeting where the full budget of the Bank is approved, in a separate vote on EvD’s budget.

4. The text is simplified in Section 4(b) in respect of “Term and removal of Chief Evaluator”. Two terms of four years (maximum eight years) have been maintained but that it can be extended “beyond the retirement age” has been removed.

7.3 ASSESSMENT OF ENVIRONMENTAL AND SOCIAL ISSUES 7.3.1 Environmental and social impact (ESI) rating For the past two years EvD has been pilot testing the ESI rating, the results of which are summarised in Section 1.4 and Appendix 8 of this AEOR. The ESI indicator is structured to be parallel to the Bank’s Transition Impact Monitoring System (TIMS). EvD now recommends that the Bank move to full utilisation of the ESI indicator. ESD currently provides an environmental and social categorisation at the time of Board approval. Implementation of ESI requires that ESD also provide an ex-ante rating of the potential for change, and develop an ESI monitoring system in parallel to the Bank’s TIMS system. This is closely aligned with the recommendation under the first bullet in Section 7.3.2 below. EvD would then use these data as the basis for ex-post evaluation. 7.3.2 Environmental and social categorisation During 2009-2010, EvD and ESD conducted a dialogue on the Bank’s environmental and social categorisation process, summarised in Appendix 11. Based on this dialogue EvD then developed the following recommendations as steps that could be undertaken and management/ESD responded. These recommendations and ESD’s responses in italics are summarised here: • ESD should discuss with management whether providing project environmental

and social risk ratings would improve decision making within the Bank.44

The Environmental Review Summary remains the document in which ESD summarises environmental and social risks, but a rating is not provided. Rather, post-approval, ESD then assigns a risk rating for project monitoring45 purposes only.

44 This is consistent with paragraph 19(i) of the ESP. Currently ESD’s categorisation process gives emphasis to paragraph 19(ii). 45 Project monitoring risks are important and useful, but are inherently different from structural risks. A primary function of the Bank is to carry out project due diligence, which isfundamentally about understanding and mitigating risks prior to Board approval, and environmental and social risks should be part of this process.

55

• The Operations Manual (internal website) and the categorisation guidelines

(external website) need to be updated to reflect the new 2008 ESP. The current texts refer to the 2003 EP and are therefore out of date.

There is agreement on this issue. The first focus of attention after the ESP was approved was to train the Banking Department and relevant support groups and develop necessary guidance notes. Banking training has been completed and a number of the Resident Offices (ROs) have also been trained (Moscow, Central Asia, Caucuses). ESD notes this point about the Online Operations Manual and will work with the Banking and Communications Departments to update relevant sections of the ESP in 2010.

• Guidance on categorisation of social impacts needs to be developed, as part of the implementation procedures. While guidance is provided for environmental impacts, no such guidance has been provided for social impacts. Such guidance notes, once approved, should be posted on the external website of the Bank.

The sensitivity and subjectivity of social issues can vary dramatically with slightly different characteristics, and so it is not easy to draft guidance on categorisation that can be applied universally. Therefore, ESD reviews each project on a case-by-case basis and goes through a robust two-stage internal review and oversight process to justify project requirements and to confirm or challenge the categorisation. A social specialist is assigned to each project and has input into categorisation. ESD has allocated additional resources on social issues, andone additional social expert and one OHS expert have been recruited in 2009.

• Categorisation of equity and working capital projects needs to be based on the environmental/social liabilities associated with all the operations of the company.

ESD notes that the 2008 ESP already requires past and current operational issues to be taken into consideration in categorisation, along with the impacts of the Bank’s investment, and includes the concept of “area of influence”, which goes beyond the proceeds of the Bank’s investment. The project appraisal also looks at opportunities, not just liabilities.

• Categorisation of refinancing projects should be based on the underlining environmental and social risks associated with the debt that is being refinanced and the adequacy with which commitments are being undertaken.

As in the previous item, this is already taking place, as ESD applies the ESP systematically and most of these type of projects will be categorised B under 2008 ESP. Any refinancing, typically balance sheet restructuring, is by nature general corporate finance and the Client’s business activity is considered to be the Bank’s project as per ESP. The Bank’s environmental and social due diligence will, therefore, not only assess the risk with the refinanced debt, but with all of the client’s operations. Any specific underlying risk with the debt to be refinanced may not be directly relevant to the Bank as the Bank is likely to structure its own security package regardless of what security was provided for the debt to be refinanced.

• Because of the nature of social impacts and the need for public consultation, social

impacts should be given the priority in setting the categorisation – that is, if a project is a Category B for environment but a Category A for social impacts, the project should be categorised as Category A.

56

The 2008 ESP contains specific performance requirements and language on social impacts. ESD has hired additional staff to address social issues; however, this is an area that continues to present challenges and the Bank’s approach is “learning-by-doing”.

7.4 ASSESSMENT OF TECHNICAL COOPERATION RELATED ISSUES Recommendations in respect of technical cooperation based on evaluation experiences in 2009 are the following: • Further to the follow-up of evaluation recommendations on technical

cooperation (TC) presented in Chapter 4, in respect of the successful launch of the Bank’s first dedicated TC training, EvD recommends that more senior bankers should be motivated to participate.

• It is also considered important to extend existing guidance to bankers on project design and required time phases to realise the TC. As part of this training, Bank units such as CSU should continue advising Operation Teams on fine-tuned project planning that essentially pursues a bottom-up approach.

• Project planning should be strictly based on the time to be spent by foreign and local consultants in-country for the execution of tasks, rather than determining the TC input through an approved budget.

• Evaluation of TC operations revealed that it is essential for the Bank to foster the cooperation between the Communications Department and OCU in order to plan for donor visibility for each project at the outset. Ideally, objectives and benchmarks would be set as well, so that reporting to donors on visibility is based on agreed parameters.

7.5 PROCESS REVIEW OF THE SYSTEM OF FOLLOW-UP OF EVALUATION

RECOMMENDATIONS 2010

7.5.1 Background Since the Evaluation Department became fully independent from management in June 2005, management has had the opportunity to provide formal management’s comments (MCs) to evaluation reports. However, as no system existed to inform the Board of Directors on the follow-up of evaluation recommendations by management, EvD proposed the establishment of a new system as presented in the AEOR for 2006. The Board of Directors in July 2006 confirmed the proposed system. A new report “Follow-up of Evaluation Recommendations 2010” is now ready for final management’s omments. Although the joint report has not been presented to the Audit Committee and the Board yet, EvD can already presents its views on the working of the system.

7.5.2 Process analysis of the system of “Follow-up of Evaluation Recommendations 2010”

Management has responded in a balanced and constructive way to evaluation recommendations, referring back extensively to the respective management’s comments and sometimes to Audit Committee deliberations on the evaluation as is reflected in the above-mentioned report. In 2009, in total 12 evaluation reports on specific themes, sectors and Bank projects incorporated recommendations that had been reviewed by the Audit Committee and in respect of which management reported on the follow-up taken. The process also benefits from the fact that the Audit Committee pays good attention to the evaluation recommendations in its deliberations.

57

It is expected that the joint report of EvD and management on the “Follow-up of Evaluation Recommendations by Management 2010” will be presented to the Audit Committee for review in June 2010 and to the Board of Directors for approval in July 2010. The Chief Evaluator is of the view, based on the preparations of the document so far, that the process of “Follow-up of Evaluation Recommendations 2010”, whereby management reports on the follow-up of evaluation recommendations presented in evaluation reports that were distributed to the EBRD Board in 2009 and whereby a review is made by EvD on management’s accomplishments respectively, worked well. Management has commented on the progress of follow-up to all the recommendations constructively and it became clear that many lessons have been learned.

ANNUAL EVALUATION OVERVIEW REPORT 2009

LIST OF APPENDICES

Appendix 1 Basic data sheet: Operation Performance Ratings on the 35 OPERs and

Special Studies, and 32 XMR Assessments prepared in 2009 Appendix 2 Basic data sheet: Operation Performance Ratings on the 274 OPERs and 345

XMR Assessments prepared in 1996-2008 Appendix 3 1993-2009 Technical Cooperation Operation Performance Evaluation Review (OPER) Reports and Special Studies Appendix 4a Selected Lessons Learned from Investment Operations Evaluated in 2009 Appendix 4b Selected Lessons Learned from Technical Cooperation Operations Evaluated

in 2009 Appendix 5 Evaluation Database Appendix 6 Checklist of Transition Indicators Appendix 7 Transition Impact Analysis Appendix 8 Outcome of Performance Ratings of the Bank's Investment Operations Appendix 9 Analysis of factors affecting performance in evaluation reports: supporting

materials to Chapter 3. Appendix 10 Environmental and Social Categorisation Appendix 11 Evaluation Coverage of TC Funds Allocated by Donors. Appendix 12 Evaluation Benchmarks

Evaluation Database for 2009: Operation performance ratings on the 35 operations covered by OPERs and Special Studies

Operation Year of Board

Approval

Country Name Industry Sector Portfolio Class

Operation Type1

Transition Impact2

Environmental Performance of the

Project and Sponsor3

Extent of Environmental

Change4

Overall Rating5

Project 1 2004 RUSSIAN FEDERATION Agribusiness PRIVATE E Satisfactory Satisfactory Some Partly SuccessfulProject 2 2006 RUSSIAN FEDERATION Transport PRIVATE L/E Marginal Satisfactory None/Negative UnsuccessfulProject 3 2005 RUSSIAN FEDERATION Bank Lending PRIVATE L Good NR NR SuccessfulProject 4 2005 RUSSIAN FEDERATION Bank Lending PRIVATE L Good NR NR Partly SuccessfulProject 5 2005 RUSSIAN FEDERATION Bank Equity PRIVATE E Good NR NR SuccessfulProject 6 2005 RUSSIAN FEDERATION Bank Equity PRIVATE E Good NR NR SuccessfulProject 7 2005 RUSSIAN FEDERATION Bank Equity PRIVATE E Satisfactory NR NR Partly SuccessfulProject 8 2005 RUSSIAN FEDERATION Bank Equity PRIVATE E Good NR NR Partly SuccessfulProject 9 2005 RUSSIAN FEDERATION Bank Lending PRIVATE L Good NR NR SuccessfulProject 10 2005 RUSSIAN FEDERATION Bank Lending PRIVATE L Good NR NR SuccessfulProject 11 2000 BELARUS Bank Lending PRIVATE L Marginal Satisfactory Some Partly SuccessfulProject 12 2000 SERBIA Bank Lending PRIVATE L Marginal Satisfactory Some Partly SuccessfulProject 13 2000 <REGIONAL> Bank Equity PRIVATE E Marginal Satisfactory Some Partly SuccessfulProject 14 2000 <REGIONAL> Equity Funds PRIVATE E Marginal Satisfactory Some Partly SuccessfulProject 15 2006 AZERBAIJAN Manufacturing and Services PRIVATE E Good Satisfactory Substantial SuccessfulProject 16 2005 GEORGIA Bank Lending PRIVATE L Marginal Marginal Some Partly SuccessfulProject 17 2005 GEORGIA Non Bank Financial Institutions PRIVATE E Marginal Marginal Some Partly SuccessfulProject 18 2005 GEORGIA Bank Lending PRIVATE L Marginal Marginal Some Partly SuccessfulProject 19 2005 GEORGIA Bank Lending PRIVATE L Marginal Marginal Some Partly SuccessfulProject 20 2005 GEORGIA Non Bank Financial Institutions PRIVATE L Marginal Marginal Some Partly SuccessfulProject 21 2006 RUSSIAN FEDERATION Non Bank Financial Institutions PRIVATE L Excellent Good Some Highly SuccessfulProject 22 2006 RUSSIAN FEDERATION Bank Lending PRIVATE L Excellent Good Some Highly SuccessfulProject 23 2007 GEORGIA Property and Tourism PRIVATE L Marginal Marginal None/Negative UnsuccessfulProject 24 2007 GEORGIA Property and Tourism PRIVATE E Marginal Marginal None/Negative UnsuccessfulProject 25 2006 RUSSIAN FEDERATION Power and Energy PRIVATE E Good Satisfactory Substantial SuccessfulProject 26 2004 AZERBAIJAN Transport STATE L Satisfactory Good Substantial SuccessfulProject 27 2004 UZBEKISTAN Manufacturing and Services PRIVATE L Marginal Satisfactory Some UnsuccessfulProject 28 2005 KAZAKHSTAN Bank Lending PRIVATE L Satisfactory Good Some Partly SuccessfulProject 29 2005 KAZAKHSTAN Bank Equity PRIVATE E Satisfactory Good Some Partly SuccessfulProject 30 2005 KAZAKHSTAN Bank Equity PRIVATE E Satisfactory Good Some Partly SuccessfulProject 31 2007 KAZAKHSTAN Small Business Finance PRIVATE L Unsatisfactory Satisfactory Some UnsuccessfulProject 32 2006 RUSSIAN FEDERATION Agribusiness PRIVATE L Good Good Substantial SuccessfulProject 33 2005 UZBEKISTAN Manufacturing and Services PRIVATE L Satisfactory Satisfactory NA Partly SuccessfulProject 34 2007 RUSSIAN FEDERATION Manufacturing and Services PRIVATE L Good Satisfactory Some SuccessfulProject 35 2005 UKRAINE Bank Lending PRIVATE L Unsatisfactory Satisfactory Some Unsuccessful

1 E=Equity; L=Loan2 The range is Excellent/Good/Satisfactory/Marginal/Unsatisfactory/Negative3 The range is Excellent/Good/Satisfactory/Marginal/Unsatisfactory/Highly Unsatisfactory4 The range is Outstanding/Substantial/Some/None/Negative5 The range is Highly Successful/Successful/Partly Successful/Unsuccessful

Appendix 1Page 1 of 2

Evaluation Database for 2009:Operation performance ratings on the 32 operations covered by XMR assessments


Approval


Operation Type1

Transition Impact2

Environmental Performance of the Project and

Sponsor3


Change4

Overall Rating5

Project 1 2003 KAZAKHSTAN Transport STATE L Satisfactory Marginal Some Partly SuccessfulProject 2 2001 <REGIONAL> Equity Funds PRIVATE E Good Satisfactory Some SuccessfulProject 3 2006 POLAND Manufacturing and Services PRIVATE L Good Good Some SuccessfulProject 4 2007 <REGIONAL> Telecoms Informatics & Media PRIVATE L Good Satisfactory None/Negative SuccessfulProject 5 2004 POLAND Municipal & Env Inf PRIVATE E Good Good Substantial SuccessfulProject 6 2006 SERBIA Bank Equity PRIVATE E Good Satisfactory Some SuccessfulProject 7 2007 KAZAKHSTAN Bank Equity PRIVATE E Good Satisfactory Some Partly SuccessfulProject 8 2005 RUSSIAN FEDERATION Manufacturing and Services PRIVATE L Good Good Some Partly SuccessfulProject 9 2005 ROMANIA Agribusiness PRIVATE E Excellent Good Substantial Partly SuccessfulProject 10 2005 ROMANIA Agribusiness PRIVATE E Excellent Good Substantial Partly SuccessfulProject 11 2005 RUSSIAN FEDERATION Bank Lending PRIVATE L Good Good None/Negative SuccessfulProject 12 2006 SERBIA Agribusiness PRIVATE L Good Good Substantial SuccessfulProject 13 2001 CROATIA Transport STATE L Marginal Good Some Partly SuccessfulProject 14 2007 AZERBAIJAN Bank Lending PRIVATE L Good Good Some SuccessfulProject 15 2007 RUSSIAN FEDERATION Non Bank Financial Institutions PRIVATE L Satisfactory NR NR SuccessfulProject 16 2005 RUSSIAN FEDERATION Non Bank Financial Institutions PRIVATE L Good Satisfactory None/Negative SuccessfulProject 17 2005 RUSSIAN FEDERATION Non Bank Financial Institutions PRIVATE L Good Satisfactory None/Negative SuccessfulProject 18 2006 RUSSIAN FEDERATION Bank Equity PRIVATE E Good Good Some SuccessfulProject 19 2006 LITHUANIA Bank Equity PRIVATE E Good Good Some SuccessfulProject 20 2000 POLAND Municipal & Env Inf STATE L Satisfactory Marginal Substantial SuccessfulProject 21 2006 CROATIA Municipal & Env Inf STATE L Good Good Substantial SuccessfulProject 22 2001 SERBIA Transport STATE L Good Satisfactory Some SuccessfulProject 23 2006 UKRAINE Manufacturing and Services PRIVATE E Good Good Some Partly SuccessfulProject 24 1997 CROATIA Equity Funds PRIVATE E Good Good None/Negative Partly SuccessfulProject 25 2006 RUSSIAN FEDERATION Bank Lending PRIVATE L Good Satisfactory Some SuccessfulProject 26 2006 LITHUANIA Municipal & Env Inf STATE L Good Good Substantial SuccessfulProject 27 2005 ROMANIA Power and Energy PRIVATE E Excellent Good Substantial SuccessfulProject 28 2005 SERBIA Manufacturing and Services PRIVATE L Good Marginal Some SuccessfulProject 29 2006 ROMANIA Municipal & Env Inf STATE L Satisfactory Satisfactory Some SuccessfulProject 30 2006 ROMANIA Municipal & Env Inf STATE L Satisfactory Satisfactory Some SuccessfulProject 31 2003 RUSSIAN FEDERATION Manufacturing and Services PRIVATE L Good Good Some Partly SuccessfulProject 32 2003 RUSSIAN FEDERATION Manufacturing and Services PRIVATE L Good Good Some Partly Successful

3 The range is Excellent/Good/Satisfactory/Marginal/Unsatisfactory/Highly Unsatisfactory4 The range is Outstanding/Substanital/Some/None/Negative5 The range is Highly Successful/Successful/Partly Successful/Unsuccessful

Appendix 1

Page 2 of 2

Basic data sheet: Operation performance ratings on the 274 OPERs prepared in 1996-2008


Approval

Year of evaluation


Operation Type1

Transition Impact2


Sponsor3


Change4

Overall Rating5

Project 1 1992 1996 ROMANIA Bank Lending PRIVATE L Good Good Substantial Highly SuccessfulProject 2 1993 1996 HUNGARY Manufacturing and Services PRIVATE E Excellent Excellent Substantial Highly SuccessfulProject 3 1992 1996 UKRAINE Equity Funds PRIVATE E Good Satisfactory Some SuccessfulProject 4 1994 1996 POLAND Property and Tourism PRIVATE L Good Good Some Highly SuccessfulProject 5 1993 1996 BULGARIA Bank Equity PRIVATE E Marginal Marginal Some Partly SuccessfulProject 6 1993 1996 HUNGARY Agribusiness STATE L Satisfactory Satisfactory Some Partly SuccessfulProject 7 1994 1996 ESTONIA Bank Lending PRIVATE L/E Good Good Some SuccessfulProject 8 1992 1996 HUNGARY Transport STATE L Good Satisfactory Substantial SuccessfulProject 9 1992 1996 POLAND Manufacturing and Services PRIVATE L/E Satisfactory Satisfactory Substantial Partly SuccessfulProject 10 1992 1996 <REGIONAL> Equity Funds PRIVATE E Excellent Good Some Highly SuccessfulProject 11 1994 1996 CZECH REPUBLIC Manufacturing and Services PRIVATE L Marginal Unsatisfactory None/Negative UnsuccessfulProject 12 1993 1996 LATVIA Transport STATE L Good Satisfactory None/Negative SuccessfulProject 13 1994 1996 POLAND Manufacturing and Services PRIVATE L Good Good Substantial SuccessfulProject 14 1994 1996 RUSSIAN FEDERATION Transport PRIVATE L Good Good Some Highly SuccessfulProject 15 1993 1997 UZBEKISTAN Bank Lending PRIVATE L Satisfactory Satisfactory Some SuccessfulProject 16 1993 1997 HUNGARY Agribusiness PRIVATE L Marginal Satisfactory None/Negative UnsuccessfulProject 17 1993 1997 RUSSIAN FEDERATION Natural Resources STATE L Marginal Marginal Some Partly SuccessfulProject 18 1993 1997 UZBEKISTAN Natural Resources PRIVATE L Marginal Marginal Some UnsuccessfulProject 19 1994 1997 POLAND Agribusiness PRIVATE L Excellent Good Some SuccessfulProject 20 1994 1997 SLOVENIA Manufacturing and Services PRIVATE L/E Good Good Substantial SuccessfulProject 21 1992 1997 CZECH REPUBLIC Manufacturing and Services PRIVATE L Unsatisfactory Unsatisfactory Some UnsuccessfulProject 22 1993 1997 SLOVENIA Bank Lending PRIVATE L/E Excellent Good Some Highly SuccessfulProject 23 1995 1997 CROATIA Manufacturing and Services PRIVATE L/E Excellent Excellent Substantial Highly SuccessfulProject 24 1995 1997 GEORGIA Transport STATE L Good Satisfactory Some Partly SuccessfulProject 25 1991 1997 HUNGARY Telecoms Informatics & Media PRIVATE E Excellent Good Some Highly SuccessfulProject 26 1992 1997 LATVIA Power and Energy STATE L Good Good Substantial SuccessfulProject 27 1994 1997 HUNGARY Property and Tourism PRIVATE L/E Marginal Satisfactory None/Negative Partly SuccessfulProject 28 1995 1997 UKRAINE Natural Resources PRIVATE L Marginal Marginal Some Partly SuccessfulProject 29 1994 1997 BULGARIA Bank Equity PRIVATE E Satisfactory Unsatisfactory None/Negative Partly SuccessfulProject 30 1995 1998 RUSSIAN FEDERATION Manufacturing and Services PRIVATE L Unsatisfactory Satisfactory Some UnsuccessfulProject 31 1992 1998 RUSSIAN FEDERATION Property and Tourism PRIVATE L/E Satisfactory Satisfactory Some Partly SuccessfulProject 32 1994 1998 RUSSIAN FEDERATION Bank Equity PRIVATE E Marginal Marginal None/Negative UnsuccessfulProject 33 1994 1998 SLOVAK REPUBLIC Natural Resources PRIVATE L/E Negative Satisfactory Substantial UnsuccessfulProject 34 1993 1998 ROMANIA Transport STATE L Good Satisfactory Some SuccessfulProject 35 1992 1998 BULGARIA Power and Energy STATE L Good Unsatisfactory None/Negative Partly SuccessfulProject 36 1996 1998 RUSSIAN FEDERATION Natural Resources PRIVATE L Good Excellent Substantial Partly SuccessfulProject 37 1993 1998 POLAND Manufacturing and Services PRIVATE L Marginal Excellent Substantial Partly SuccessfulProject 38 1995 1998 <REGIONAL> Natural Resources PRIVATE L/E Satisfactory Satisfactory Substantial Successful




Approval



Operation Type1

Transition Impact2


Sponsor3


Change4

Overall Rating5

Project 39 1995 1998 HUNGARY Bank Equity PRIVATE E Good Good Some SuccessfulProject 40 1997 1998 ESTONIA Bank Equity PRIVATE E Negative Unsatisfactory None/Negative UnsuccessfulProject 41 1993 1998 SLOVENIA Transport STATE L Satisfactory Good Some SuccessfulProject 42 1994 1998 ROMANIA Municipal & Env Inf STATE L Good Satisfactory Substantial SuccessfulProject 43 1995 1998 RUSSIAN FEDERATION Equity Funds PRIVATE E Marginal Unsatisfactory None/Negative UnsuccessfulProject 44 1995 1998 HUNGARY Municipal & Env Inf PRIVATE L Good Satisfactory Substantial SuccessfulProject 45 1994 1998 ROMANIA Property and Tourism PRIVATE L Satisfactory Good Some SuccessfulProject 46 1995 1998 UKRAINE Agribusiness PRIVATE L Good Excellent Substantial SuccessfulProject 47 1993 1998 ARMENIA Power and Energy STATE L Negative Marginal None/Negative UnsuccessfulProject 48 1996 1998 RUSSIAN FEDERATION Transport PRIVATE L Unsatisfactory Marginal None/Negative UnsuccessfulProject 49 1996 1999 POLAND Agribusiness PRIVATE L/E Marginal Excellent Some UnsuccessfulProject 50 1996 1999 RUSSIAN FEDERATION Transport PRIVATE L/E Unsatisfactory Marginal Some Partly SuccessfulProject 51 1997 1999 BOSNIA AND HERZEGOVINA Agribusiness PRIVATE L Good Satisfactory Some SuccessfulProject 52 1994 1999 BELARUS Transport STATE L Satisfactory Good Some SuccessfulProject 53 1995 1999 ESTONIA Municipal & Env Inf STATE L Good Satisfactory Substantial SuccessfulProject 54 1997 1999 CZECH REPUBLIC Manufacturing and Services PRIVATE E Unsatisfactory Excellent Some UnsuccessfulProject 55 1994 1999 MOLDOVA Agribusiness STATE L Satisfactory Satisfactory None/Negative UnsuccessfulProject 56 1995 1999 LATVIA Bank Lending PRIVATE L Marginal Satisfactory Some UnsuccessfulProject 57 1994 1999 ESTONIA Bank Lending PRIVATE L/E Satisfactory Good Some SuccessfulProject 58 1996 1999 RUSSIAN FEDERATION Natural Resources PRIVATE L Satisfactory Good Substantial SuccessfulProject 59 1997 1999 ESTONIA Manufacturing and Services PRIVATE L Good Excellent Some SuccessfulProject 60 1997 1999 BOSNIA AND HERZEGOVINA Small Business Finance PRIVATE E Satisfactory Good None/Negative SuccessfulProject 61 1997 1999 POLAND Agribusiness PRIVATE L Good Excellent Substantial Partly SuccessfulProject 62 1995 1999 LITHUANIA Bank Lending PRIVATE L Excellent Good Some Highly SuccessfulProject 63 1994 1999 RUSSIAN FEDERATION Telecoms Informatics & Media PRIVATE L/E Good Satisfactory None/Negative Highly SuccessfulProject 64 1995 1999 KYRGYZ REPUBLIC Agribusiness PRIVATE L Negative Marginal Some UnsuccessfulProject 65 1994 1999 AZERBAIJAN Power and Energy STATE L Satisfactory Satisfactory Substantial Partly SuccessfulProject 66 1996 1999 SLOVAK REPUBLIC Bank Lending PRIVATE L Marginal Satisfactory None/Negative SuccessfulProject 67 1996 1999 RUSSIAN FEDERATION Bank Lending PRIVATE L Negative Marginal None/Negative UnsuccessfulProject 68 1993 1999 HUNGARY Transport PRIVATE L/E Negative Excellent Some UnsuccessfulProject 69 1996 1999 MOLDOVA Small Business Finance PRIVATE L Satisfactory Satisfactory Some Partly SuccessfulProject 70 1997 2000 RUSSIAN FEDERATION Agribusiness PRIVATE L Excellent Excellent Some SuccessfulProject 71 1997 2000 KAZAKHSTAN Manufacturing and Services PRIVATE L Excellent Good Substantial Highly SuccessfulProject 72 1997 2000 RUSSIAN FEDERATION Manufacturing and Services PRIVATE E Good Good Some SuccessfulProject 73 1996 2000 BULGARIA Agribusiness PRIVATE L/E Unsatisfactory Marginal None/Negative UnsuccessfulProject 74 1997 2000 ROMANIA Telecoms Informatics & Media PRIVATE L Good Good None/Negative SuccessfulProject 75 1995 2000 MOLDOVA Transport STATE L Unsatisfactory Good Some UnsuccessfulProject 76 1996 2000 RUSSIAN FEDERATION Natural Resources PRIVATE L/E Good Good Substantial Successful




Approval



Operation Type1

Transition Impact2


Sponsor3


Change4

Overall Rating5

Project 77 1998 2000 BULGARIA Manufacturing and Services PRIVATE L/E Satisfactory Marginal Some Partly SuccessfulProject 78 1992 2000 POLAND Manufacturing and Services PRIVATE L Good Excellent Outstanding Partly SuccessfulProject 79 1994 2000 CZECH REPUBLIC Transport STATE L Negative Satisfactory None/Negative UnsuccessfulProject 80 1996 2000 RUSSIAN FEDERATION Bank Lending PRIVATE L Marginal Good Some Partly SuccessfulProject 81 1996 2000 <REGIONAL> Manufacturing and Services PRIVATE E Good Good Substantial SuccessfulProject 82 1998 2000 CROATIA Bank Lending PRIVATE L Excellent Good Some Highly SuccessfulProject 83 1999 2000 LITHUANIA Agribusiness PRIVATE E Marginal Good Some Partly SuccessfulProject 84 1997 2000 BULGARIA Bank Equity PRIVATE L/E Good Good Some SuccessfulProject 85 1993 2000 SLOVENIA Power and Energy STATE L Good Excellent Some Highly SuccessfulProject 86 1998 2000 CZECH REPUBLIC Bank Equity PRIVATE E Excellent Satisfactory Some Highly SuccessfulProject 87 1997 2000 RUSSIAN FEDERATION Municipal & Env Inf STATE L Good Satisfactory None/Negative Partly SuccessfulProject 88 1997 2001 POLAND Manufacturing and Services PRIVATE E Negative Marginal Substantial UnsuccessfulProject 89 1995 2001 AZERBAIJAN Municipal & Env Inf STATE L Marginal Satisfactory Some Partly SuccessfulProject 90 1998 2001 KAZAKHSTAN Bank Lending PRIVATE L Good Good Some SuccessfulProject 91 1998 2001 AZERBAIJAN Natural Resources PRIVATE L Good Good Some SuccessfulProject 92 1993 2001 SLOVAK REPUBLIC Manufacturing and Services PRIVATE L/E Satisfactory Good Outstanding SuccessfulProject 93 1995 2001 TURKMENISTAN Manufacturing and Services PRIVATE L/E Marginal Good Some Partly SuccessfulProject 94 1996 2001 FYR MACEDONIA Bank Equity PRIVATE L/E Good Good Some SuccessfulProject 95 1997 2001 GEORGIA Agribusiness PRIVATE L/E Good Good Substantial Partly SuccessfulProject 96 1998 2001 CROATIA Agribusiness PRIVATE E Unsatisfactory Good Some UnsuccessfulProject 97 1997 2001 ESTONIA Transport STATE L Good Good Some SuccessfulProject 98 1997 2001 UZBEKISTAN Natural Resources STATE L Satisfactory Good Substantial SuccessfulProject 99 1994 2001 <REGIONAL> Non Bank Financial Institutions PRIVATE E Good Good None/Negative SuccessfulProject 100 1997 2001 POLAND Manufacturing and Services PRIVATE E Unsatisfactory Satisfactory None/Negative UnsuccessfulProject 101 1998 2001 SLOVENIA Manufacturing and Services PRIVATE L/E Marginal Marginal Some Partly SuccessfulProject 102 1997 2001 POLAND Agribusiness PRIVATE L/E Unsatisfactory Good Some UnsuccessfulProject 103 1997 2001 RUSSIAN FEDERATION Manufacturing and Services PRIVATE L Marginal Marginal Substantial UnsuccessfulProject 104 1996 2001 LATVIA Property and Tourism PRIVATE L Marginal Excellent Some Partly SuccessfulProject 105 1995 2001 RUSSIAN FEDERATION Transport PRIVATE L Good Excellent Substantial SuccessfulProject 106 1998 2001 POLAND Bank Equity PRIVATE E Good Good Some SuccessfulProject 107 1998 2001 SLOVAK REPUBLIC Agribusiness PRIVATE L Good Good Substantial SuccessfulProject 108 1994 2001 RUSSIAN FEDERATION Bank Lending PRIVATE L Unsatisfactory NR NR UnsuccessfulProject 109 1997 2002 CZECH REPUBLIC Non Bank Financial Institutions PRIVATE E Good Excellent None/Negative SuccessfulProject 110 1999 2002 <REGIONAL> Telecoms Informatics & Media PRIVATE E Good Good None/Negative SuccessfulProject 111 1999 2002 CROATIA Municipal & Env Inf STATE L Good Good Substantial SuccessfulProject 112 1995 2002 UKRAINE Transport PRIVATE L Good Excellent Substantial SuccessfulProject 113 2000 2002 LITHUANIA Telecoms Informatics & Media PRIVATE E Good Good None/Negative Partly SuccessfulProject 114 1996 2002 LATVIA Municipal & Env Inf STATE L Good Excellent Substantial Successful




Approval



Operation Type1

Transition Impact2


Sponsor3


Change4

Overall Rating5

Project 115 1995 2002 FYR MACEDONIA Bank Equity PRIVATE L/E Satisfactory Good Some Partly SuccessfulProject 116 1996 2002 KAZAKHSTAN Transport STATE L Marginal Good Some Partly SuccessfulProject 117 1998 2002 RUSSIAN FEDERATION Telecoms Informatics & Media PRIVATE L Satisfactory Good None/Negative Partly SuccessfulProject 118 2001 2002 POLAND Bank Lending PRIVATE L Good Good None/Negative SuccessfulProject 119 1994 2002 BULGARIA Equity Funds PRIVATE E Marginal Good Some UnsuccessfulProject 120 2000 2002 RUSSIAN FEDERATION Natural Resources PRIVATE L Marginal Excellent Substantial Partly SuccessfulProject 121 2000 2002 RUSSIAN FEDERATION Manufacturing and Services PRIVATE L Good Marginal None/Negative Partly SuccessfulProject 122 1999 2002 CROATIA Bank Lending PRIVATE L Satisfactory Good Some Partly SuccessfulProject 123 1995 2002 RUSSIAN FEDERATION Manufacturing and Services PRIVATE L Negative Marginal None/Negative UnsuccessfulProject 124 1999 2002 <REGIONAL> Bank Equity PRIVATE L/E Good Good Some SuccessfulProject 125 1993 2002 RUSSIAN FEDERATION Small Business Finance PRIVATE L/E Satisfactory NR NR Partly SuccessulProject 126 2000 2002 RUSSIAN FEDERATION Manufacturing and Services PRIVATE L Excellent Good Substantial Highly SuccessfulProject 127 1995 2002 RUSSIAN FEDERATION Agribusiness PRIVATE E Good Marginal Some SuccessfulProject 128 1996 2002 GEORGIA Bank Lending PRIVATE L Good Marginal Some Partly SuccessfulProject 129 1995 2002 UKRAINE Manufacturing and Services PRIVATE L Unsatisfactory Satisfactory None/Negative UnsuccessfulProject 130 1999 2002 UKRAINE Agribusiness PRIVATE L Good Excellent Substantial SuccessfulProject 131 1999 2002 HUNGARY Bank Equity PRIVATE E Marginal Marginal None/Negative Partly SuccessfulProject 132 1997 2003 RUSSIAN FEDERATION Natural Resources PRIVATE L Good Satisfactory Substantial SuccessfulProject 133 2001 2003 UKRAINE Manufacturing and Services PRIVATE L Good Good Substantial SuccessfulProject 134 1998 2003 HUNGARY Municipal & Env Inf PRIVATE L Marginal Satisfactory None/Negative Partly SuccessfulProject 135 2000 2003 BULGARIA Municipal & Env Inf PRIVATE L Good Good Some SuccessfulProject 136 1997 2003 KYRGYZ REPUBLIC Property and Tourism PRIVATE L Good Good Some SuccessfulProject 137 2001 2003 ROMANIA Manufacturing and Services PRIVATE L Good Satisfactory Some SuccessfulProject 138 1997 2003 HUNGARY Manufacturing and Services PRIVATE E Satisfactory Marginal None/Negative Partly SuccessfulProject 139 1995 2003 CZECH REPUBLIC Manufacturing and Services PRIVATE L Satisfactory Marginal Some UnsuccessfulProject 140 1996 2003 UZBEKISTAN Bank Lending PRIVATE L Unsatisfactory Satisfactory None/Negative UnsuccessfulProject 141 2000 2003 FYR MACEDONIA Natural Resources PRIVATE L Marginal Excellent Substantial Partly SuccessfulProject 142 1997 2003 ROMANIA Telecoms Informatics & Media PRIVATE L Good Good Some SuccessfulProject 143 1997 2003 UZBEKISTAN Power and Energy STATE L Marginal Good Some Partly SuccessfulProject 144 2002 2003 RUSSIAN FEDERATION Agribusiness PRIVATE L Satisfactory Satisfactory None/Negative SuccessfulProject 145 2000 2003 RUSSIAN FEDERATION Transport PRIVATE L Satisfactory Good Some SuccessfulProject 146 1997 2003 ESTONIA Bank Lending PRIVATE L Satisfactory Satisfactory Some SuccessfulProject 147 2000 2003 SLOVAK REPUBLIC Bank Equity PRIVATE E Good Satisfactory Some SuccessfulProject 148 1999 2003 KAZAKHSTAN Telecoms Informatics & Media PRIVATE L Satisfactory Satisfactory None/Negative SuccessfulProject 149 1999 2003 GEORGIA Power and Energy PRIVATE L Satisfactory Good Substantial Partly SuccessfulProject 150 1999 2003 ALBANIA Small Business Finance PRIVATE E Good Satisfactory Some Partly SuccessfulProject 151 2000 2003 GEORGIA Small Business Finance PRIVATE L/E Good Satisfactory Some Partly SuccessfulProject 152 2001 2003 SERBIA Small Business Finance PRIVATE E Good Satisfactory Some Partly Successful




Approval



Operation Type1

Transition Impact2


Sponsor3


Change4

Overall Rating5

Project 153 2000 2003 MOLDOVA Small Business Finance PRIVATE L/E Satisfactory Satisfactory Some Partly SuccessfulProject 154 1999 2003 UKRAINE Small Business Finance PRIVATE L/E Good Satisfactory Some Partly SuccessfulProject 155 1997 2004 UKRAINE Equity Funds PRIVATE E Good Satisfactory Some SuccessfulProject 156 1995 2004 MOLDOVA Transport PRIVATE L/E Negative Unsatisfactory None/Negative UnsuccessfulProject 157 2002 2004 RUSSIAN FEDERATION Power and Energy PRIVATE L Good Good Substantial SuccessfulProject 158 2002 2004 RUSSIAN FEDERATION Agribusiness PRIVATE L Good Excellent Some SuccessfulProject 159 1998 2004 SERBIA Equity Funds PRIVATE E Marginal Satisfactory None/Negative UnsuccessfulProject 160 2000 2004 POLAND Non Bank Financial Institutions PRIVATE L Excellent Good Some Highly SuccessfulProject 161 1999 2004 <REGIONAL> Property and Tourism PRIVATE E Satisfactory Marginal None/Negative Partly SuccessfulProject 162 1999 2004 UKRAINE Natural Resources PRIVATE L Excellent Good Substantial Highly SuccessfulProject 163 1998 2004 KAZAKHSTAN Power and Energy PRIVATE L Marginal Satisfactory Outstanding UnsuccessfulProject 164 2001 2004 <REGIONAL> Transport PRIVATE L Marginal Good Some Partly SuccessfulProject 165 2001 2004 RUSSIAN FEDERATION Manufacturing and Services PRIVATE E Excellent Excellent Outstanding Highly SuccessfulProject 166 1999 2004 CROATIA Telecoms Informatics & Media PRIVATE L Good Good Some SuccessfulProject 167 1996 2004 <REGIONAL> Municipal & Env Inf PRIVATE L/E Good Good Substantial SuccessfulProject 168 2001 2004 RUSSIAN FEDERATION Power and Energy PRIVATE L Good Excellent Some SuccessfulProject 169 1999 2004 FYR MACEDONIA Manufacturing and Services PRIVATE L Good Satisfactory Substantial Partly SuccessfulProject 170 1999 2004 BULGARIA Non Bank Financial Institutions PRIVATE E Good Good None/Negative SuccessfulProject 171 1996 2004 RUSSIAN FEDERATION Transport STATE L Good Good Some SuccessfulProject 172 2000 2004 UKRAINE Transport PRIVATE L Satisfactory Good Some SuccessfulProject 173 2002 2004 RUSSIAN FEDERATION Non Bank Financial Institutions PRIVATE L Good Good None/Negative SuccessfulProject 174 1999 2004 KAZAKHSTAN Property and Tourism PRIVATE L Satisfactory Satisfactory Some SuccessfulProject 175 1997 2004 RUSSIAN FEDERATION Municipal & Env Inf STATE L Satisfactory Satisfactory None/Negative SuccessfulProject 176 2000 2004 ESTONIA Municipal & Env Inf PRIVATE L/E Good Good Substantial SuccessfulProject 177 2000 2004 AZERBAIJAN Natural Resources PRIVATE L/E Good Good Substantial SuccessfulProject 178 2003 2005 HUNGARY Natural Resources PRIVATE L Good Excellent Outstanding SuccessfulProject 179 1999 2005 TURKMENISTAN Natural Resources PRIVATE L Good Good Substantial SuccessfulProject 180 2000 2005 RUSSIAN FEDERATION Manufacturing and Services PRIVATE L Satisfactory Satisfactory Some SuccessfulProject 181 2003 2005 ROMANIA Telecoms Informatics & Media PRIVATE L Good Good Some Highly SuccessfulProject 182 1995 2005 <REGIONAL> Agribusiness PRIVATE E Good Good Substantial SuccessfulProject 183 1997 2005 KAZAKHSTAN Small Business Finance PRIVATE L Good Satisfactory Some SuccessfulProject 184 2002 2005 CROATIA Bank Lending PRIVATE L Good Satisfactory None/Negative SuccessfulProject 185 2000 2005 <REGIONAL> Non Bank Financial Institutions PRIVATE L Unsatisfactory Marginal None/Negative UnsuccessfulProject 186 2000 2005 POLAND Municipal & Env Inf STATE L Good Good Some SuccessfulProject 187 2000 2005 CZECH REPUBLIC Municipal & Env Inf PRIVATE L Excellent Excellent Outstanding Highly SuccessfulProject 188 2003 2005 LITHUANIA Municipal & Env Inf STATE L Satisfactory Good Substantial SuccessfulProject 189 2001 2005 CROATIA Bank Lending PRIVATE L Marginal Good Some UnsuccessfulProject 190 2000 2005 CZECH REPUBLIC Telecoms Informatics & Media PRIVATE E Good Satisfactory None/Negative Successful




Approval



Operation Type1

Transition Impact2


Sponsor3


Change4

Overall Rating5

Project 191 1999 2005 AZERBAIJAN Transport STATE L Good Satisfactory Substantial SuccessfulProject 192 1997 2005 MOLDOVA Bank Equity PRIVATE L/E Satisfactory Satisfactory Substantial Partly SuccessfulProject 193 1998 2005 GEORGIA Transport STATE L Satisfactory Marginal Some SuccessfulProject 194 2003 2005 UKRAINE Manufacturing and Services PRIVATE L Excellent Good Substantial Highly SuccessfulProject 195 2003 2005 AZERBAIJAN Property and Tourism PRIVATE L Marginal Satisfactory None/Negative Partly SuccessfulProject 196 1999 2005 GEORGIA Bank Equity PRIVATE L/E Good Satisfactory Some SuccessfulProject 197 1994 2005 RUSSIAN FEDERATION Equity Funds PRIVATE E Good Good Some SuccessfulProject 198 2003 2005 KAZAKHSTAN Bank Equity PRIVATE E Unsatisfactory Marginal None/Negative UnsuccessfulProject 199 2001 2005 UKRAINE Agribusiness PRIVATE L Satisfactory Satisfactory None/Negative SuccessfulProject 200 2003 2005 SERBIA Agribusiness PRIVATE L Good Marginal Some SuccessfulProject 201 2002 2006 <REGIONAL> Agribusiness PRIVATE L Satisfactory Good Some Partly SuccessfulProject 202 2003 2006 HUNGARY Municipal & Env Inf PRIVATE L Satisfactory Excellent Outstanding SuccessfulProject 203 1999 2006 KAZAKHSTAN Transport STATE L Satisfactory Satisfactory Some SuccessfulProject 204 2001 2006 POLAND Agribusiness PRIVATE L Satisfactory Satisfactory Substantial Partly SuccessfulProject 205 2004 2006 RUSSIAN FEDERATION Bank Equity PRIVATE E Excellent Good Some Highly SuccessfulProject 206 1996 2006 <REGIONAL> Equity Funds PRIVATE E Good Satisfactory Some SuccessfulProject 207 1996 2006 KAZAKHSTAN Equity Funds PRIVATE E Marginal Satisfactory Some Partly SuccessfulProject 208 1995 2006 SLOVAK REPUBLIC Equity Funds PRIVATE E Marginal Satisfactory Some Partly SuccessfulProject 209 1995 2006 ROMANIA Equity Funds PRIVATE E Good Satisfactory Some SuccessfulProject 210 1997 2006 BULGARIA Equity Funds PRIVATE E Unsatisfactory Satisfactory Some UnsuccessfulProject 211 2003 2006 SLOVAK REPUBLIC Non Bank Financial Institutions PRIVATE E Excellent Excellent Substantial Highly SuccessfulProject 212 1996 2006 <REGIONAL> Equity Funds PRIVATE E Good Satisfactory Some SuccessfulProject 213 1998 2006 UZBEKISTAN Municipal & Env Inf STATE L Marginal Satisfactory Outstanding Partly SuccessfulProject 214 2002 2006 RUSSIAN FEDERATION Municipal & Env Inf PRIVATE L Satisfactory Satisfactory Outstanding SuccessfulProject 215 2002 2006 RUSSIAN FEDERATION Manufacturing and Services PRIVATE L Excellent Good Some Highly SuccessfulProject 216 2000 2006 CROATIA Manufacturing and Services PRIVATE L Good Satisfactory Substantial SuccessfulProject 217 1999 2006 KAZAKHSTAN Power and Energy STATE L Good Good Some SuccessfulProject 218 2001 2006 RUSSIAN FEDERATION Manufacturing and Services PRIVATE L Good Good Substantial SuccessfulProject 219 2000 2006 POLAND Property and Tourism PRIVATE L Satisfactory Good Substantial Partly SuccessfulProject 220 2000 2006 <REGIONAL> Telecoms Informatics & Media PRIVATE E Unsatisfactory NA NA UnsuccessfulProject 221 2004 2006 <REGIONAL> Natural Resources PRIVATE E Good Good Substantial SuccessfulProject 222 2002 2006 RUSSIAN FEDERATION Natural Resources PRIVATE L Good Good Some SuccessfulProject 223 2001 2006 SLOVAK REPUBLIC Manufacturing and Services PRIVATE L Unsatisfactory Marginal None/Negative UnsuccessfulProject 224 1998 2006 KAZAKHSTAN Bank Lending PRIVATE L Marginal Marginal Substantial Partly SuccessfulProject 225 1999 2006 POLAND Municipal & Env Inf STATE L Good Good Substantial SuccessfulProject 226 2002 2006 UZBEKISTAN Small Business Finance PRIVATE L Good Satisfactory Some SuccessfulProject 227 2003 2006 RUSSIAN FEDERATION Natural Resources STATE L Satisfactory Good Some Partly SuccessfulProject 228 1996 2007 UZBEKISTAN Manufacturing and Services PRIVATE L/E Negative NA NA Unsuccessful




Approval



Operation Type1

Transition Impact2


Sponsor3


Change4

Overall Rating5

Project 229 2005 2007 RUSSIAN FEDERATION Telecoms Informatics & Media PRIVATE L/E Good NA NA SuccessfulProject 230 2005 2007 KAZAKHSTAN Telecoms Informatics & Media PRIVATE L Good NA NA SuccessfulProject 231 2002 2007 POLAND Property and Tourism PRIVATE L Good Marginal None/Negative SuccessfulProject 232 2004 2007 SERBIA Manufacturing and Services PRIVATE L Good Satisfactory Substantial SuccessfulProject 233 2003 2007 <REGIONAL> Natural Resources PRIVATE L Satisfactory Excellent Substantial Partly SuccessfulProject 234 1999 2007 UKRAINE Transport STATE L Satisfactory Satisfactory Some SuccessfulProject 235 2004 2007 RUSSIAN FEDERATION Bank Equity PRIVATE L/E Good Satisfactory Some SuccessfulProject 236 2003 2007 AZERBAIJAN Natural Resources PRIVATE L Satisfactory Good Substantial Partly SuccessfulProject 237 2002 2007 RUSSIAN FEDERATION Manufacturing and Services PRIVATE L Satisfactory Satisfactory None/Negative Partly SuccessfulProject 238 2003 2007 RUSSIAN FEDERATION Natural Resources PRIVATE L Marginal Unsatisfactory None/Negative UnsuccessfulProject 239 2005 2007 RUSSIAN FEDERATION Manufacturing and Services PRIVATE L Excellent Satisfactory Some SuccessfulProject 240 1996 2007 POLAND Equity Funds PRIVATE E Excellent Good Substantial SuccessfulProject 241 2003 2007 AZERBAIJAN Bank Equity PRIVATE L/E Excellent Marginal None/Negative SuccessfulProject 242 2004 2007 FYR MACEDONIA Bank Lending PRIVATE L Satisfactory Satisfactory Some SuccessfulProject 243 2003 2007 SLOVAK REPUBLIC Power and Energy PRIVATE L Good Good Some SuccessfulProject 244 2003 2007 <REGIONAL> Agribusiness PRIVATE L Good Satisfactory Some SuccessfulProject 245 2004 2007 RUSSIAN FEDERATION Agribusiness PRIVATE L Good Satisfactory Outstanding SuccessfulProject 246 2004 2007 ALBANIA Manufacturing and Services PRIVATE L Good Satisfactory Substantial SuccessfulProject 247 1999 2007 LITHUANIA Manufacturing and Services PRIVATE L/E Unsatisfactory Satisfactory Some UnsuccessfulProject 248 2002 2007 CROATIA Transport STATE L Satisfactory NA NA Partly SuccessfulProject 249 2003 2007 ROMANIA Bank Equity PRIVATE E Excellent Excellent Substantial Highly SuccessfulProject 250 2001 2007 SERBIA Municipal & Env Inf STATE L Satisfactory Marginal Outstanding Partly SuccessfulProject 251 2003 2007 TAJIKISTAN Small Business Finance PRIVATE L Excellent Good Some SuccessfulProject 252 2005 2008 BELARUS Agribusiness PRIVATE L/E Good Satisfactory Substantial SuccessfulProject 253 2002 2008 ROMANIA Municipal & Env Inf PRIVATE L Excellent Excellent Substantial Highly SuccessfulProject 254 2000 2008 FYR MACEDONIA Municipal & Env Inf STATE L Satisfactory Marginal Some Partly SuccessfulProject 255 1996 2008 UKRAINE Power and Energy STATE L Marginal Unsatisfactory None/Negative UnsuccessfulProject 256 2002 2008 SERBIA Transport STATE L Good Good Substantial SuccessfulProject 257 2006 2008 RUSSIAN FEDERATION Manufacturing and Services PRIVATE E Satisfactory Satisfactory Some Partly SuccessfulProject 258 2001 2008 <REGIONAL> Property and Tourism PRIVATE L/E Good Excellent Some SuccessfulProject 259 2004 2008 RUSSIAN FEDERATION Municipal & Env Inf STATE L Unsatisfactory Unsatisfactory Some UnsuccessfulProject 260 2005 2008 RUSSIAN FEDERATION Power and Energy STATE L Marginal Satisfactory Some Partly SuccessfulProject 261 2000 2008 KAZAKHSTAN Transport STATE L Satisfactory Satisfactory None/Negative Partly SuccessfulProject 262 2004 2008 UKRAINE Agribusiness PRIVATE L Good Good Substantial SuccessfulProject 263 2003 2008 <REGIONAL> Agribusiness PRIVATE L Excellent Excellent Substantial SuccessfulProject 264 2006 2008 RUSSIAN FEDERATION Non Bank Financial Institutions PRIVATE L Satisfactory NA NA Partly SuccessfulProject 265 2006 2008 CROATIA Agribusiness PRIVATE E Satisfactory Marginal Some Partly SuccessfulProject 266 2005 2008 KAZAKHSTAN Property and Tourism PRIVATE L Satisfactory Satisfactory Some Partly Successful




Approval



Operation Type1

Transition Impact2


Sponsor3


Change4

Overall Rating5

Project 267 2002 2008 RUSSIAN FEDERATION Transport STATE L Satisfactory Marginal Some SuccessfulProject 268 2000 2008 <REGIONAL> Natural Resources PRIVATE L Unsatisfactory Unsatisfactory None/Negative UnsuccessfulProject 269 2000 2008 <REGIONAL> Bank Lending PRIVATE L Good Good Some SuccessfulProject 270 2003 2008 <REGIONAL> Bank Lending PRIVATE L Good Good Some SuccessfulProject 271 2005 2008 FYR MACEDONIA Manufacturing and Services PRIVATE L Good Marginal Some Partly SuccessfulProject 272 2003 2008 SLOVENIA Bank Lending PRIVATE L Good Excellent Substantial SuccessfulProject 273 1998 2008 <REGIONAL> Manufacturing and Services PRIVATE L/E Good Satisfactory Some SuccessfulProject 274 1999 2008 ALBANIA Power and Energy STATE L Satisfactory Good Substantial Partly Successful

2 The range is Excellent/Good/Satisfactory/Marginal/Unsatisfactory/Negative3 The range is Excellent/Good/Satisfactory/Marginal/Unsatisfactory/Highly Unsatisfactory4 The range is Outstanding/Substanital/Some/None/Negative5 The range is Highly Successful/Successful/Partly Successful/Unsuccessful


Basic data sheet: Operation performance ratings on the 344 XMR assessments prepared in 1996-2008


Approval



Operation Type1

Transition Impact2


Sponsor3


Change4

Overall Rating5

Project 1 1993 1996 SLOVAK REPUBLIC Natural Resources PRIVATE L Good Good Substantial SuccessfulProject 2 1992 1996 POLAND Bank Lending PRIVATE L Marginal Satisfactory None/Negative UnsuccessfulProject 3 1993 1996 HUNGARY Bank Lending PRIVATE L Good Satisfactory Some SuccessfulProject 4 1993 1996 ROMANIA Non Bank Financial Institutions PRIVATE E Marginal Satisfactory Some UnsuccessfulProject 5 1991 1996 CZECH REPUBLIC Manufacturing and Services PRIVATE L Marginal Satisfactory Some UnsuccessfulProject 6 1993 1996 ALBANIA Agribusiness PRIVATE L Good Satisfactory Some Partly SuccessfulProject 7 1994 1996 RUSSIAN FEDERATION Transport PRIVATE L Good Satisfactory Some SuccessfulProject 8 1996 1996 ESTONIA Bank Lending PRIVATE L Good Good Substantial SuccessfulProject 9 1994 1996 HUNGARY Bank Equity PRIVATE E Good Satisfactory None/Negative SuccessfulProject 10 1994 1996 POLAND Manufacturing and Services PRIVATE L Good Good Some SuccessfulProject 11 1994 1996 LATVIA Bank Equity PRIVATE E Satisfactory Satisfactory None/Negative Partly SuccessfulProject 12 1992 1996 ROMANIA Natural Resources STATE L Good Good Substantial SuccessfulProject 13 1991 1996 CZECH REPUBLIC Manufacturing and Services PRIVATE L Unsatisfactory Unsatisfactory None/Negative UnsuccessfulProject 14 1992 1996 RUSSIAN FEDERATION Natural Resources PRIVATE L Satisfactory Satisfactory Some Partly SuccessfulProject 15 1993 1996 HUNGARY Property and Tourism PRIVATE E Good Satisfactory None/Negative Partly SuccessfulProject 16 1992 1996 POLAND Manufacturing and Services PRIVATE L Satisfactory Satisfactory Some SuccessfulProject 17 1993 1996 BULGARIA Agribusiness PRIVATE E Good Satisfactory Some SuccessfulProject 18 1992 1996 HUNGARY Property and Tourism PRIVATE L Satisfactory Satisfactory None/Negative Partly SuccessfulProject 19 1993 1996 POLAND Manufacturing and Services PRIVATE L Good Satisfactory Some SuccessfulProject 20 1993 1996 SLOVAK REPUBLIC Bank Equity PRIVATE E Good Satisfactory Some SuccessfulProject 21 1994 1997 HUNGARY Natural Resources PRIVATE L Good Satisfactory Some SuccessfulProject 22 1994 1997 BULGARIA Agribusiness PRIVATE L Satisfactory Good Substantial Partly SuccessfulProject 23 1994 1997 LITHUANIA Bank Equity PRIVATE L/E Satisfactory Good Substantial Partly SuccessfulProject 24 1994 1997 CZECH REPUBLIC Manufacturing and Services PRIVATE L Good Satisfactory Substantial SuccessfulProject 25 1993 1997 UKRAINE Transport STATE L Good Satisfactory None/Negative SuccessfulProject 26 1994 1997 ROMANIA Agribusiness PRIVATE L Marginal Good Substantial Partly SuccessfulProject 27 1996 1997 ROMANIA Manufacturing and Services PRIVATE E Marginal Good Substantial Partly SuccessfulProject 28 1994 1997 ROMANIA Bank Lending PRIVATE L Satisfactory Satisfactory Some Partly SuccessfulProject 29 1993 1997 ROMANIA Bank Equity PRIVATE L Good Good Substantial SuccessfulProject 30 1994 1997 POLAND Bank Equity PRIVATE E Good Satisfactory Some SuccessfulProject 31 1992 1997 ESTONIA Power and Energy STATE L Good Good Substantial SuccessfulProject 32 1992 1997 <REGIONAL> Telecoms Informatics & Media STATE L Satisfactory Satisfactory None/Negative Partly SuccessfulProject 33 1993 1997 RUSSIAN FEDERATION Equity Funds PRIVATE E Good Satisfactory Some SuccessfulProject 34 1995 1997 HUNGARY Manufacturing and Services PRIVATE E Excellent Good Substantial Highly SuccessfulProject 35 1994 1997 CZECH REPUBLIC Bank Lending PRIVATE L Marginal Satisfactory Some UnsuccessfulProject 36 1992 1997 LITHUANIA Power and Energy STATE L Good Good Substantial SuccessfulProject 37 1996 1997 POLAND Telecoms Informatics & Media PRIVATE L/E Good Satisfactory None/Negative Partly SuccessfulProject 38 1991 1997 POLAND Telecoms Informatics & Media PRIVATE L Good Satisfactory None/Negative Successful




Approval



Operation Type1

Transition Impact2


Sponsor3


Change4

Overall Rating5

Project 39 1995 1997 CZECH REPUBLIC Manufacturing and Services PRIVATE L Good Good Some SuccessfulProject 40 1993 1997 SLOVAK REPUBLIC Telecoms Informatics & Media STATE L Satisfactory Satisfactory None/Negative Partly SuccessfulProject 41 1994 1997 SLOVAK REPUBLIC Bank Lending PRIVATE L Marginal Satisfactory Some Partly SuccessfulProject 42 1995 1998 RUSSIAN FEDERATION Non Bank Financial Institutions PRIVATE E Good Satisfactory None/Negative Partly SuccessfulProject 43 1994 1998 ESTONIA Municipal & Env Inf STATE L Satisfactory Good Substantial Partly SuccessfulProject 44 1993 1998 POLAND Manufacturing and Services PRIVATE L Good Excellent Substantial SuccessfulProject 45 1993 1998 RUSSIAN FEDERATION Manufacturing and Services PRIVATE E Unsatisfactory Good Substantial UnsuccessfulProject 46 1993 1998 BELARUS Agribusiness STATE L Marginal Excellent Substantial UnsuccessfulProject 47 1992 1998 BULGARIA Telecoms Informatics & Media STATE L Excellent Good Some SuccessfulProject 48 1994 1998 RUSSIAN FEDERATION Transport PRIVATE L Marginal Satisfactory Some Partly SuccessfulProject 49 1992 1998 ALBANIA Telecoms Informatics & Media STATE L Good Good Some SuccessfulProject 50 1993 1998 SLOVENIA Manufacturing and Services PRIVATE L Satisfactory Excellent None/Negative SuccessfulProject 51 1995 1998 HUNGARY Manufacturing and Services PRIVATE E Excellent Excellent Substantial Highly SuccessfulProject 52 1996 1998 ESTONIA Bank Lending PRIVATE L Good Good Some Highly SuccessfulProject 53 1995 1998 POLAND Non Bank Financial Institutions PRIVATE E Excellent Satisfactory None/Negative SuccessfulProject 54 1995 1998 RUSSIAN FEDERATION Bank Lending PRIVATE L Good Good Substantial SuccessfulProject 55 1994 1998 UKRAINE Telecoms Informatics & Media STATE L Good Good Some SuccessfulProject 56 1995 1998 HUNGARY Transport PRIVATE L Good Excellent Substantial SuccessfulProject 57 1994 1998 POLAND Equity Funds PRIVATE L Good Good Some SuccessfulProject 58 1996 1998 CROATIA Agribusiness PRIVATE L Good Satisfactory Some SuccessfulProject 59 1995 1998 LITHUANIA Manufacturing and Services PRIVATE L Good Good Substantial SuccessfulProject 60 1994 1998 POLAND Power and Energy PRIVATE L Good Excellent Outstanding SuccessfulProject 61 1993 1998 ALBANIA Property and Tourism PRIVATE L/E Excellent Good None/Negative SuccessfulProject 62 1995 1998 UZBEKISTAN Manufacturing and Services PRIVATE E Marginal Good Some UnsuccessfulProject 63 1995 1998 KYRGYZ REPUBLIC Power and Energy STATE L Satisfactory Good None/Negative SuccessfulProject 64 1994 1998 LITHUANIA Transport STATE L Satisfactory Satisfactory Some SuccessfulProject 65 1993 1998 ROMANIA Manufacturing and Services PRIVATE L/E Satisfactory Marginal Substantial Partly SuccessfulProject 66 1995 1998 SLOVENIA Manufacturing and Services PRIVATE L Satisfactory Excellent None/Negative SuccessfulProject 67 1995 1998 RUSSIAN FEDERATION Manufacturing and Services PRIVATE L Marginal Satisfactory Substantial UnsuccessfulProject 68 1996 1998 RUSSIAN FEDERATION Bank Lending PRIVATE L Satisfactory Good Substantial Partly SuccessfulProject 69 1993 1998 ALBANIA Property and Tourism PRIVATE L/E Satisfactory Satisfactory Some Partly SuccessfulProject 70 1995 1998 RUSSIAN FEDERATION Bank Lending PRIVATE L Satisfactory Satisfactory Some SuccessfulProject 71 1995 1998 CROATIA Bank Lending PRIVATE L Excellent Good Some SuccessfulProject 72 1997 1999 TAJIKISTAN Agribusiness PRIVATE L Good Satisfactory Some Partly SuccessfulProject 73 1997 1999 BULGARIA Manufacturing and Services PRIVATE L Excellent Good Substantial SuccessfulProject 74 1996 1999 BULGARIA Manufacturing and Services PRIVATE L Marginal Satisfactory Some Partly SuccessfulProject 75 1995 1999 POLAND Manufacturing and Services PRIVATE E Satisfactory Marginal Some Partly SuccessfulProject 76 1997 1999 POLAND Manufacturing and Services PRIVATE L Satisfactory Marginal Some Successful

Appendix 2

Page 10 of 18



Approval



Operation Type1

Transition Impact2


Sponsor3


Change4

Overall Rating5

Project 77 1997 1999 LATVIA Telecoms Informatics & Media PRIVATE L/E Excellent Good None/Negative SuccessfulProject 78 1994 1999 KYRGYZ REPUBLIC Telecoms Informatics & Media STATE L Good Excellent None/Negative SuccessfulProject 79 1996 1999 RUSSIAN FEDERATION Manufacturing and Services PRIVATE L/E Good Excellent Outstanding Highly SuccessfulProject 80 1994 1999 RUSSIAN FEDERATION Natural Resources PRIVATE L Satisfactory Satisfactory Substantial SuccessfulProject 81 1993 1999 <REGIONAL> Equity Funds PRIVATE E Satisfactory Satisfactory Some SuccessfulProject 82 1993 1999 SLOVENIA Equity Funds PRIVATE E Satisfactory Satisfactory None/Negative SuccessfulProject 83 1993 1999 ROMANIA Telecoms Informatics & Media PRIVATE L Satisfactory Excellent None/Negative SuccessfulProject 84 1994 1999 POLAND Equity Funds PRIVATE E Satisfactory Satisfactory Some Partly SuccessfulProject 85 1997 1999 SLOVENIA Manufacturing and Services PRIVATE E Good Satisfactory None/Negative SuccessfulProject 86 1993 1999 HUNGARY Bank Lending PRIVATE L Good Satisfactory Some Highly SuccessfulProject 87 1996 1999 POLAND Property and Tourism PRIVATE L Satisfactory Good None/Negative SuccessfulProject 88 1992 1999 POLAND Equity Funds PRIVATE E Good Satisfactory None/Negative SuccessfulProject 89 1997 1999 HUNGARY Telecoms Informatics & Media PRIVATE L Excellent Excellent Some SuccessfulProject 90 1995 1999 SLOVAK REPUBLIC Agribusiness PRIVATE L Satisfactory Excellent Some SuccessfulProject 91 1993 1999 ROMANIA Agribusiness STATE L Satisfactory Satisfactory Some UnsuccessfulProject 92 1995 1999 POLAND Manufacturing and Services PRIVATE E Unsatisfactory Marginal Some UnsuccessfulProject 93 1996 1999 UZBEKISTAN Bank Equity PRIVATE E Satisfactory Satisfactory None/Negative Partly SuccessfulProject 94 1996 1999 RUSSIAN FEDERATION Bank Lending PRIVATE L/E Unsatisfactory Unsatisfactory None/Negative UnsuccessfulProject 95 1997 1999 RUSSIAN FEDERATION Bank Equity PRIVATE E Unsatisfactory Unsatisfactory None/Negative UnsuccessfulProject 96 1994 1999 KYRGYZ REPUBLIC Bank Lending PRIVATE L Satisfactory Satisfactory None/Negative Partly SuccessfulProject 97 1996 1999 HUNGARY Bank Lending PRIVATE L Marginal Good Some Partly SuccessfulProject 98 1997 1999 SLOVAK REPUBLIC Bank Lending PRIVATE L Good Excellent Some SuccessfulProject 99 1994 1999 RUSSIAN FEDERATION Manufacturing and Services PRIVATE L Unsatisfactory Unsatisfactory None/Negative UnsuccessfulProject 100 1996 1999 CROATIA Bank Lending PRIVATE L Good Excellent Some SuccessfulProject 101 1992 2000 BELARUS Telecoms Informatics & Media STATE L Satisfactory Satisfactory None/Negative Partly SuccessfulProject 102 1997 2000 POLAND Manufacturing and Services PRIVATE L/E Satisfactory Good Substantial SuccessfulProject 103 1995 2000 ROMANIA Agribusiness PRIVATE L Excellent Excellent Some SuccessfulProject 104 1995 2000 POLAND Property and Tourism PRIVATE L Good Good Some SuccessfulProject 105 1997 2000 ROMANIA Manufacturing and Services PRIVATE L/E Good Excellent Some Partly SuccessfulProject 106 1992 2000 RUSSIAN FEDERATION Natural Resources PRIVATE L Good Excellent Some SuccessfulProject 107 1996 2000 HUNGARY Manufacturing and Services PRIVATE L/E Good Good Some Highly SuccessfulProject 108 1996 2000 BOSNIA AND HERZEGOVINA Bank Equity PRIVATE E Good Satisfactory Some SuccessfulProject 109 1997 2000 <REGIONAL> Municipal & Env Inf PRIVATE L/E Good Excellent Substantial SuccessfulProject 110 1993 2000 <REGIONAL> Non Bank Financial Institutions PRIVATE E Marginal Satisfactory None/Negative UnsuccessfulProject 111 1997 2000 RUSSIAN FEDERATION Manufacturing and Services PRIVATE E Unsatisfactory Good Some UnsuccessfulProject 112 1993 2000 RUSSIAN FEDERATION Natural Resources PRIVATE L/E Satisfactory Satisfactory Substantial SuccessfulProject 113 1994 2000 ARMENIA Transport STATE L Satisfactory Satisfactory Some UnsuccessfulProject 114 1997 2000 BELARUS Bank Equity PRIVATE E Satisfactory Good Some Partly Successful

Appendix 2

Page 11 of 18



Approval



Operation Type1

Transition Impact2


Sponsor3


Change4

Overall Rating5

Project 115 1996 2000 CROATIA Bank Lending PRIVATE L/E Good Satisfactory Some SuccessfulProject 116 1995 2000 <REGIONAL> Bank Equity PRIVATE E Unsatisfactory Satisfactory Some UnsuccessfulProject 117 1997 2000 BOSNIA AND HERZEGOVINA Bank Equity PRIVATE L Good Excellent Some Highly SuccessfulProject 118 1995 2000 RUSSIAN FEDERATION Non Bank Financial Institutions PRIVATE E Negative Marginal None/Negative UnsuccessfulProject 119 1995 2000 UZBEKISTAN Non Bank Financial Institutions PRIVATE E Satisfactory Marginal None/Negative Partly SuccessfulProject 120 1996 2000 CROATIA Bank Equity PRIVATE E Good Satisfactory Some SuccessfulProject 121 1997 2000 HUNGARY Bank Equity PRIVATE E Good Marginal Some SuccessfulProject 122 1997 2000 ROMANIA Bank Equity PRIVATE E Good Marginal Some Highly SuccessfulProject 123 1995 2000 UKRAINE Agribusiness PRIVATE L/E Marginal Marginal None/Negative UnsuccessfulProject 124 1999 2000 SLOVAK REPUBLIC Natural Resources STATE L Satisfactory Excellent None/Negative SuccessfulProject 125 1998 2001 ESTONIA Manufacturing and Services PRIVATE L Satisfactory Good Some SuccessfulProject 126 1997 2001 RUSSIAN FEDERATION Power and Energy PRIVATE L Good Good Substantial SuccessfulProject 127 1999 2001 RUSSIAN FEDERATION Natural Resources PRIVATE L Marginal Satisfactory Some Partly SuccessfulProject 128 1998 2001 LITHUANIA Bank Lending PRIVATE L/E Satisfactory Good Some Partly SuccessfulProject 129 1994 2001 SLOVENIA Manufacturing and Services PRIVATE E Unsatisfactory Unsatisfactory None/Negative UnsuccessfulProject 130 1995 2001 <REGIONAL> Equity Funds PRIVATE E Satisfactory Satisfactory None/Negative Partly SuccessfulProject 131 1995 2001 <REGIONAL> Agribusiness PRIVATE E Good Marginal Some Partly SuccessfulProject 132 1994 2001 SLOVENIA Equity Funds PRIVATE E Good Satisfactory Some Partly SuccessfulProject 133 1999 2001 ROMANIA Bank Equity PRIVATE E Good Good Some SuccessfulProject 134 1997 2001 KYRGYZ REPUBLIC Power and Energy STATE L Good Good None/Negative SuccessfulProject 135 1996 2001 POLAND Agribusiness STATE L Satisfactory Excellent Substantial Partly SuccessfulProject 136 1998 2001 POLAND Equity Funds PRIVATE E Unsatisfactory Satisfactory Some UnsuccessfulProject 137 1996 2001 ROMANIA Transport STATE L Good Satisfactory Some SuccessfulProject 138 1994 2001 BELARUS Bank Lending PRIVATE L Marginal Good Some Partly SuccessfulProject 139 1996 2001 ROMANIA Bank Lending PRIVATE L Good Good Some SuccessfulProject 140 1997 2001 RUSSIAN FEDERATION Agribusiness PRIVATE L/E Unsatisfactory Marginal None/Negative UnsuccessfulProject 141 1999 2001 HUNGARY Manufacturing and Services PRIVATE L Satisfactory Satisfactory Some SuccessfulProject 142 1999 2001 TAJIKISTAN Transport STATE L Good Good Some SuccessfulProject 143 1998 2001 BULGARIA Agribusiness PRIVATE L/E Satisfactory Excellent Some Partly SuccessfulProject 144 1995 2001 FYR MACEDONIA Bank Lending PRIVATE L Good Good Some SuccessfulProject 145 1995 2001 MOLDOVA Municipal & Env Inf STATE L Marginal Unsatisfactory None/Negative UnsuccessfulProject 146 2000 2001 ARMENIA Agribusiness PRIVATE L Good Good Some SuccessfulProject 147 1999 2001 GEORGIA Agribusiness PRIVATE L Good Good Some SuccessfulProject 148 1995 2001 RUSSIAN FEDERATION Bank Lending PRIVATE L Satisfactory Good Some Partly SuccessfulProject 149 1997 2001 LATVIA Bank Lending PRIVATE L Satisfactory Good Some Partly SuccessfulProject 150 1997 2001 SLOVAK REPUBLIC Bank Equity PRIVATE E Unsatisfactory Marginal None/Negative UnsuccessfulProject 151 1996 2001 <REGIONAL> Agribusiness PRIVATE L/E Satisfactory Good Some SuccessfulProject 152 1998 2001 BOSNIA AND HERZEGOVINA Telecoms Informatics & Media STATE L Good Good None/Negative Successful

Appendix 2

Page 12 of 18



Approval



Operation Type1

Transition Impact2


Sponsor3


Change4

Overall Rating5

Project 153 1996 2001 LATVIA Bank Equity PRIVATE L/E Negative Satisfactory Some UnsuccessfulProject 154 1998 2002 MOLDOVA Transport STATE L Good Good Some SuccessfulProject 155 2000 2002 ROMANIA Manufacturing and Services PRIVATE L Marginal Excellent Some UnsuccessfulProject 156 1999 2002 SLOVENIA Bank Lending PRIVATE L Good Satisfactory Some Highly SuccessfulProject 157 1998 2002 SLOVAK REPUBLIC Manufacturing and Services PRIVATE L Marginal Good Some Partly SuccessfulProject 158 1995 2002 ROMANIA Power and Energy STATE L Good Marginal Some Partly SuccessfulProject 159 1998 2002 POLAND Manufacturing and Services PRIVATE L Good Marginal Some Partly SuccessfulProject 160 1997 2002 CZECH REPUBLIC Equity Funds PRIVATE E Unsatisfactory Satisfactory None/Negative UnsuccessfulProject 161 1998 2002 RUSSIAN FEDERATION Natural Resources PRIVATE L Excellent Satisfactory Some Highly SuccessfulProject 162 2000 2002 ALBANIA Bank Equity PRIVATE E Satisfactory Good Some SuccessfulProject 163 2000 2002 POLAND Non Bank Financial Institutions PRIVATE E Satisfactory Good Some Partly SuccessfulProject 164 1999 2002 UKRAINE Bank Lending PRIVATE L Good Satisfactory Some SuccessfulProject 165 1999 2002 LATVIA Transport PRIVATE L Marginal Excellent None/Negative UnsuccessfulProject 166 1997 2002 CROATIA Agribusiness STATE L Satisfactory Good Some Partly SuccessfulProject 167 1995 2002 LATVIA Power and Energy STATE L Satisfactory Good Some SuccessfulProject 168 1997 2002 UKRAINE Agribusiness PRIVATE L Good Good Some SuccessfulProject 169 1999 2002 RUSSIAN FEDERATION Agribusiness PRIVATE L Good Satisfactory Some SuccessfulProject 170 1996 2002 ROMANIA Transport STATE L Satisfactory Excellent Substantial Partly SuccessfulProject 171 1998 2002 ESTONIA Manufacturing and Services PRIVATE L/E Satisfactory Satisfactory Some Partly SuccessfulProject 172 1994 2002 <REGIONAL> Equity Funds PRIVATE E Good Satisfactory Some SuccessfulProject 173 1998 2002 GEORGIA Bank Equity PRIVATE L/E Good Marginal Some SuccessfulProject 174 1999 2002 FYR MACEDONIA Telecoms Informatics & Media PRIVATE L Excellent Marginal None/Negative SuccessfulProject 175 1999 2002 POLAND Bank Lending PRIVATE L Satisfactory Good Some SuccessfulProject 176 2000 2002 CROATIA Bank Lending PRIVATE L Satisfactory Good Some Partly SuccessfulProject 177 1999 2002 BOSNIA AND HERZEGOVINA Bank Lending PRIVATE L Good Good Some SuccessfulProject 178 1996 2002 RUSSIAN FEDERATION Bank Equity PRIVATE L Marginal Marginal None/Negative Partly SuccessfulProject 179 2000 2002 POLAND Property and Tourism PRIVATE E Good Good Some SuccessfulProject 180 1996 2002 GEORGIA Bank Lending PRIVATE L/E Unsatisfactory Marginal Some UnsuccessfulProject 181 1997 2003 AZERBAIJAN Power and Energy STATE L Satisfactory Good Substantial Partly SuccessfulProject 182 2000 2003 KYRGYZ REPUBLIC Bank Equity PRIVATE L/E Good Good Some SuccessfulProject 183 2000 2003 UKRAINE Bank Lending PRIVATE L Good Satisfactory Some SuccessfulProject 184 2001 2003 CROATIA Agribusiness PRIVATE L Good Excellent Substantial SuccessfulProject 185 1998 2003 GEORGIA Manufacturing and Services PRIVATE L Good Satisfactory Some SuccessfulProject 186 2001 2003 KAZAKHSTAN Bank Equity PRIVATE E Marginal Satisfactory Some Partly SuccessfulProject 187 1997 2003 ALBANIA Bank Lending PRIVATE L Satisfactory Marginal None/Negative Partly SuccessfulProject 188 1996 2003 ROMANIA Equity Funds PRIVATE E Good Good Some SuccessfulProject 189 2000 2003 RUSSIAN FEDERATION Telecoms Informatics & Media PRIVATE E Good Good None/Negative SuccessfulProject 190 1998 2003 RUSSIAN FEDERATION Property and Tourism PRIVATE L Good Good None/Negative Successful

Appendix 2

Page 13 of 18



Approval



Operation Type1

Transition Impact2


Sponsor3


Change4

Overall Rating5

Project 191 2001 2003 BULGARIA Manufacturing and Services PRIVATE L Excellent Excellent Some Highly SuccessfulProject 192 1997 2003 RUSSIAN FEDERATION Power and Energy STATE L Good Good Substantial Partly SuccessfulProject 193 2001 2003 SLOVENIA Telecoms Informatics & Media PRIVATE L Good Good None/Negative SuccessfulProject 194 1997 2003 POLAND Equity Funds PRIVATE E Satisfactory Good Some Partly SuccessfulProject 195 1999 2003 RUSSIAN FEDERATION Non Bank Financial Institutions PRIVATE E Unsatisfactory Marginal None/Negative UnsuccessfulProject 196 1996 2003 POLAND Non Bank Financial Institutions PRIVATE E Satisfactory Good Some UnsuccessfulProject 197 2001 2003 CROATIA Bank Lending PRIVATE L Good Good Some SuccessfulProject 198 1996 2003 KYRGYZ REPUBLIC Bank Equity PRIVATE L Good Good Some SuccessfulProject 199 1999 2003 LATVIA Bank Lending PRIVATE L Satisfactory Good Some SuccessfulProject 200 2001 2003 CROATIA Bank Lending PRIVATE L Good Good Some Highly SuccessfulProject 201 2001 2003 RUSSIAN FEDERATION Telecoms Informatics & Media PRIVATE L/E Excellent Excellent None/Negative Highly SuccessfulProject 202 2000 2003 CROATIA Agribusiness PRIVATE L Satisfactory Good Some SuccessfulProject 203 2001 2003 LATVIA Non Bank Financial Institutions PRIVATE E Good Marginal None/Negative SuccessfulProject 204 1996 2003 CROATIA Bank Equity PRIVATE E Good Good Some SuccessfulProject 205 1996 2003 UZBEKISTAN Bank Equity PRIVATE L Marginal Good Some Partly SuccessfulProject 206 1999 2003 BOSNIA AND HERZEGOVINA Agribusiness PRIVATE L Good Marginal None/Negative Partly SuccessfulProject 207 1997 2003 MOLDOVA Municipal & Env Inf STATE L Marginal Good Some Partly SuccessfulProject 208 2000 2003 RUSSIAN FEDERATION Agribusiness PRIVATE L Good Good Some SuccessfulProject 209 1999 2003 POLAND Bank Lending PRIVATE L Satisfactory Good Substantial Partly SuccessfulProject 210 1997 2003 <REGIONAL> Non Bank Financial Institutions PRIVATE E Satisfactory Excellent None/Negative Partly SuccessfulProject 211 2001 2004 POLAND Agribusiness PRIVATE L Excellent Good Some Partly SuccessfulProject 212 2002 2004 CZECH REPUBLIC Telecoms Informatics & Media PRIVATE L Good Good Some SuccessfulProject 213 2000 2004 POLAND Telecoms Informatics & Media PRIVATE L Good Excellent Some SuccessfulProject 214 1999 2004 FYR MACEDONIA Equity Funds PRIVATE E Good Good Some SuccessfulProject 215 2000 2004 SLOVAK REPUBLIC Telecoms Informatics & Media PRIVATE E Marginal Good None/Negative Partly SuccessfulProject 216 2002 2004 ESTONIA Non Bank Financial Institutions PRIVATE E Good Good Some SuccessfulProject 217 2001 2004 TURKMENISTAN Manufacturing and Services PRIVATE L Marginal Good Substantial Partly SuccessfulProject 218 1999 2004 CROATIA Non Bank Financial Institutions PRIVATE E Good Satisfactory None/Negative SuccessfulProject 219 1999 2004 POLAND Telecoms Informatics & Media PRIVATE L Marginal Good None/Negative UnsuccessfulProject 220 1999 2004 POLAND Manufacturing and Services PRIVATE L Good Good Substantial SuccessfulProject 221 1999 2004 UZBEKISTAN Transport STATE L Good Good Some SuccessfulProject 222 1997 2004 HUNGARY Transport STATE L Satisfactory Good None/Negative Partly SuccessfulProject 223 1999 2004 HUNGARY Transport STATE L Good Good None/Negative SuccessfulProject 224 1999 2004 AZERBAIJAN Transport STATE L Unsatisfactory Satisfactory None/Negative UnsuccessfulProject 225 1998 2004 ROMANIA Property and Tourism PRIVATE L/E Good Good None/Negative Highly SuccessfulProject 226 2002 2004 SLOVENIA Bank Equity PRIVATE L/E Good Good Some SuccessfulProject 227 2002 2004 BULGARIA Bank Equity PRIVATE E Good Good Some Highly SuccessfulProject 228 2001 2004 SERBIA Bank Equity PRIVATE E Good Good Some Successful

Appendix 2

Page 14 of 18



Approval



Operation Type1

Transition Impact2


Sponsor3


Change4

Overall Rating5

Project 229 2002 2004 ROMANIA Bank Lending PRIVATE L Good Good Some SuccessfulProject 230 2000 2004 RUSSIAN FEDERATION Bank Lending PRIVATE L/E Good Good Some SuccessfulProject 231 1999 2004 BOSNIA AND HERZEGOVINA Bank Lending PRIVATE L Good Good Some SuccessfulProject 232 1998 2004 UKRAINE Manufacturing and Services PRIVATE L Good Excellent None/Negative SuccessfulProject 233 1996 2004 RUSSIAN FEDERATION Telecoms Informatics & Media PRIVATE L Excellent Excellent Some Highly SuccessfulProject 234 2003 2004 POLAND Telecoms Informatics & Media PRIVATE L Good Good None/Negative Highly SuccessfulProject 235 1999 2004 <REGIONAL> Telecoms Informatics & Media PRIVATE E Good Good None/Negative SuccessfulProject 236 1995 2004 <REGIONAL> Bank Lending PRIVATE L Good Satisfactory None/Negative SuccessfulProject 237 2001 2004 SERBIA Manufacturing and Services PRIVATE L Marginal Satisfactory Some Partly SuccessfulProject 238 2001 2004 RUSSIAN FEDERATION Manufacturing and Services PRIVATE L Good Good Some SuccessfulProject 239 2000 2004 UKRAINE Agribusiness PRIVATE L Good Good Some Partly SuccessfulProject 240 2000 2005 SERBIA Small Business Finance PRIVATE L/E Good Good Some SuccessfulProject 241 2001 2005 BULGARIA Small Business Finance PRIVATE L/E Excellent Excellent Some Highly SuccessfulProject 242 2002 2005 RUSSIAN FEDERATION Non Bank Financial Institutions PRIVATE L Good Excellent Some SuccessfulProject 243 2001 2005 HUNGARY Non Bank Financial Institutions PRIVATE L Good Good Some SuccessfulProject 244 1999 2005 FYR MACEDONIA Bank Equity PRIVATE L Good Good Some SuccessfulProject 245 2002 2005 RUSSIAN FEDERATION Manufacturing and Services PRIVATE L Marginal Excellent Some Partly SuccessfulProject 246 2001 2005 RUSSIAN FEDERATION Natural Resources STATE L Satisfactory Good Substantial SuccessfulProject 247 2002 2005 POLAND Transport STATE L Marginal Satisfactory None/Negative UnsuccessfulProject 248 1996 2005 <REGIONAL> Manufacturing and Services PRIVATE E Satisfactory Good Some Partly SuccessfulProject 249 2001 2005 SLOVAK REPUBLIC Power and Energy STATE L Good Good Some SuccessfulProject 250 1997 2005 <REGIONAL> Telecoms Informatics & Media PRIVATE E Good Marginal Some Partly SuccessfulProject 251 2002 2005 ROMANIA Natural Resources PRIVATE L Excellent Good Substantial Highly SuccessfulProject 252 1999 2005 POLAND Property and Tourism PRIVATE L Satisfactory Satisfactory None/Negative Partly SuccessfulProject 253 2002 2005 RUSSIAN FEDERATION Property and Tourism PRIVATE L Good Excellent Some Highly SuccessfulProject 254 2002 2005 <REGIONAL> Manufacturing and Services PRIVATE L Satisfactory Marginal Some Partly SuccessfulProject 255 2003 2005 RUSSIAN FEDERATION Manufacturing and Services PRIVATE L Good Good Substantial SuccessfulProject 256 2002 2005 BOSNIA AND HERZEGOVINA Bank Equity PRIVATE L/E Excellent Satisfactory Some SuccessfulProject 257 1998 2005 CROATIA Transport STATE L Satisfactory Satisfactory Some Partly SuccessfulProject 258 2002 2005 GEORGIA Transport PRIVATE L Good Good Some Partly SuccessfulProject 259 2003 2005 ROMANIA Manufacturing and Services PRIVATE E Good Excellent Outstanding Highly SuccessfulProject 260 2002 2005 POLAND Non Bank Financial Institutions PRIVATE L Satisfactory Marginal Some Partly SuccessfulProject 261 2000 2005 UZBEKISTAN Natural Resources PRIVATE L Good Excellent Substantial SuccessfulProject 262 2003 2005 UKRAINE Manufacturing and Services PRIVATE L Satisfactory Good Some SuccessfulProject 263 1997 2005 RUSSIAN FEDERATION Property and Tourism PRIVATE L Good Good Some Highly SuccessfulProject 264 2000 2005 POLAND Municipal & Env Inf STATE L Marginal Good Substantial Partly SuccessfulProject 265 2002 2005 RUSSIAN FEDERATION Transport STATE L Marginal Satisfactory Some Partly SuccessfulProject 266 1997 2005 RUSSIAN FEDERATION Agribusiness PRIVATE L Excellent Good Some Highly Successful

Appendix 2

Page 15 of 18



Approval



Operation Type1

Transition Impact2


Sponsor3


Change4

Overall Rating5

Project 267 2002 2005 UKRAINE Agribusiness PRIVATE L Good Good Some SuccessfulProject 268 2001 2005 <REGIONAL> Agribusiness PRIVATE L Good Good Some SuccessfulProject 269 1999 2006 CZECH REPUBLIC Property and Tourism PRIVATE E Marginal Satisfactory None/Negative Partly SuccessfulProject 270 2003 2006 <REGIONAL> Agribusiness PRIVATE L Good Excellent Some Partly SuccessfulProject 271 2003 2006 UZBEKISTAN Agribusiness PRIVATE L Good Excellent Substantial Partly SuccessfulProject 272 2005 2006 KYRGYZ REPUBLIC Property and Tourism PRIVATE L Good Satisfactory Substantial SuccessfulProject 273 2003 2006 <REGIONAL> Municipal & Env Inf PRIVATE E Good Satisfactory None/Negative SuccessfulProject 274 2003 2006 <REGIONAL> Agribusiness PRIVATE L Good Good Substantial Highly SuccessfulProject 275 1999 2006 <REGIONAL> Power and Energy PRIVATE E Satisfactory Good Substantial Partly SuccessfulProject 276 2004 2006 RUSSIAN FEDERATION Agribusiness PRIVATE L/E Satisfactory Satisfactory None/Negative Partly SuccessfulProject 277 2003 2006 BULGARIA Telecoms Informatics & Media PRIVATE L/E Excellent Good Some Highly SuccessfulProject 278 2004 2006 POLAND Manufacturing and Services PRIVATE L Excellent Excellent Substantial Highly SuccessfulProject 279 2003 2006 RUSSIAN FEDERATION Telecoms Informatics & Media PRIVATE L Good Good None/Negative SuccessfulProject 280 1997 2006 <REGIONAL> Agribusiness PRIVATE E Marginal Satisfactory None/Negative UnsuccessfulProject 281 2002 2006 <REGIONAL> Manufacturing and Services PRIVATE L Good Excellent Substantial SuccessfulProject 282 2002 2006 FYR MACEDONIA Small Business Finance PRIVATE L/E Excellent Excellent None/Negative Highly SuccessfulProject 283 2003 2006 ALBANIA Telecoms Informatics & Media PRIVATE L Excellent Excellent Some Highly SuccessfulProject 284 2004 2006 KAZAKHSTAN Agribusiness PRIVATE L Good Satisfactory None/Negative SuccessfulProject 285 1997 2006 UKRAINE Transport STATE L Good Satisfactory None/Negative SuccessfulProject 286 2003 2006 RUSSIAN FEDERATION Manufacturing and Services PRIVATE L Marginal Excellent Some Partly SuccessfulProject 287 2003 2006 ROMANIA Bank Lending PRIVATE L Excellent Good Some Highly SuccessfulProject 288 1998 2006 LATVIA Transport STATE L Good Good Some SuccessfulProject 289 2004 2006 LITHUANIA Municipal & Env Inf STATE L Good Satisfactory Some SuccessfulProject 290 2003 2006 SERBIA Property and Tourism PRIVATE L Good Good None/Negative SuccessfulProject 291 2003 2006 UKRAINE Manufacturing and Services PRIVATE L Marginal Satisfactory Some Partly SuccessfulProject 292 2003 2006 CROATIA Municipal & Env Inf STATE L Excellent Excellent Substantial Highly SuccessfulProject 293 2002 2006 KAZAKHSTAN Non Bank Financial Institutions PRIVATE L Satisfactory Satisfactory None/Negative Partly SuccessfulProject 294 2002 2007 POLAND Municipal & Env Inf STATE L Satisfactory Satisfactory Substantial SuccessfulProject 295 2005 2007 AZERBAIJAN Non Bank Financial Institutions PRIVATE E Good Good None/Negative Partly SuccessfulProject 296 2004 2007 RUSSIAN FEDERATION Bank Lending PRIVATE L Satisfactory Good Some Partly SuccessfulProject 297 1997 2007 <REGIONAL> Equity Funds PRIVATE E Good Good Some Partly SuccessfulProject 298 2000 2007 RUSSIAN FEDERATION Municipal & Env Inf STATE L Marginal Unsatisfactory None/Negative UnsuccessfulProject 299 2004 2007 RUSSIAN FEDERATION Transport PRIVATE L Good Good Some SuccessfulProject 300 2004 2007 RUSSIAN FEDERATION Bank Lending PRIVATE L Excellent Good Some SuccessfulProject 301 2003 2007 RUSSIAN FEDERATION Agribusiness PRIVATE L/E Good Good Some SuccessfulProject 302 2000 2007 POLAND Equity Funds PRIVATE E Good Good Some SuccessfulProject 303 2000 2007 UKRAINE Transport STATE L Satisfactory Satisfactory Substantial SuccessfulProject 304 2005 2007 ROMANIA Agribusiness PRIVATE L Satisfactory Good Substantial Successful

Appendix 2

Page 16 of 18



Approval



Operation Type1

Transition Impact2


Sponsor3


Change4

Overall Rating5

Project 305 2002 2007 ARMENIA Manufacturing and Services PRIVATE L Good Unsatisfactory Some Partly SuccessfulProject 306 2001 2007 UKRAINE Manufacturing and Services PRIVATE L Satisfactory Satisfactory Some Partly SuccessfulProject 307 2004 2007 BULGARIA Natural Resources PRIVATE L Good Excellent Substantial SuccessfulProject 308 2000 2007 ROMANIA Power and Energy STATE L Excellent Excellent Substantial Highly SuccessfulProject 309 2005 2007 HUNGARY Transport PRIVATE L Good Good Some SuccessfulProject 310 1999 2007 ARMENIA Bank Lending PRIVATE L Good Good Some SuccessfulProject 311 2004 2007 UZBEKISTAN Telecoms Informatics & Media PRIVATE E Excellent Excellent None/Negative Highly SuccessfulProject 312 2002 2007 POLAND Manufacturing and Services PRIVATE L Excellent Good Some SuccessfulProject 313 2001 2007 POLAND Municipal & Env Inf STATE L Satisfactory Good Substantial Partly SuccessfulProject 314 2004 2007 MOLDOVA Bank Lending PRIVATE L Satisfactory Good Some Partly SuccessfulProject 315 2002 2007 BULGARIA Power and Energy PRIVATE L Excellent Good Substantial SuccessfulProject 316 2002 2007 RUSSIAN FEDERATION Power and Energy PRIVATE L Good Satisfactory Substantial SuccessfulProject 317 2002 2007 BOSNIA AND HERZEGOVINA Telecoms Informatics & Media STATE L Excellent Marginal None/Negative Partly SuccessfulProject 318 2002 2007 <REGIONAL> Bank Lending PRIVATE L Satisfactory Good Some SuccessfulProject 319 2005 2007 RUSSIAN FEDERATION Non Bank Financial Institutions PRIVATE L Satisfactory Good None/Negative Partly SuccessfulProject 320 2002 2007 SLOVAK REPUBLIC Equity Funds PRIVATE E Negative Unsatisfactory None/Negative UnsuccessfulProject 321 2004 2007 ESTONIA Manufacturing and Services PRIVATE L/E Satisfactory Satisfactory Some Partly SuccessfulProject 322 2002 2007 SERBIA Bank Lending PRIVATE L Satisfactory Good Some SuccessfulProject 323 2003 2007 SERBIA Bank Equity PRIVATE E Satisfactory Good Some Partly SuccessfulProject 324 2006 2008 GEORGIA Small Business Finance PRIVATE L Good Good Some SuccessfulProject 325 2005 2008 GEORGIA Municipal & Env Inf STATE L Good Marginal Some Partly SuccessfulProject 326 2005 2008 MOLDOVA Bank Lending PRIVATE L Good Good Some SuccessfulProject 327 1997 2008 UKRAINE Manufacturing and Services STATE L Good Satisfactory Outstanding Highly SuccessfulProject 328 2005 2008 RUSSIAN FEDERATION Bank Lending PRIVATE L Good Good Some Partly SuccessfulProject 329 2005 2008 LATVIA Bank Lending PRIVATE L Satisfactory Satisfactory Some SuccessfulProject 330 2001 2008 KAZAKHSTAN Transport STATE L Marginal NA NA Partly SuccessfulProject 331 2002 2008 ROMANIA Manufacturing and Services PRIVATE L Satisfactory Satisfactory Some SuccessfulProject 332 2002 2008 RUSSIAN FEDERATION Manufacturing and Services PRIVATE L Satisfactory Satisfactory Some Partly SuccessfulProject 333 2004 2008 RUSSIAN FEDERATION Manufacturing and Services PRIVATE L Satisfactory Good Some Partly SuccessfulProject 334 2005 2008 RUSSIAN FEDERATION Property and Tourism PRIVATE L/E Good Marginal None/Negative SuccessfulProject 335 2001 2008 BOSNIA AND HERZEGOVINA Transport STATE L Good Marginal Some SuccessfulProject 336 2004 2008 HUNGARY Transport PRIVATE L Good Good Some SuccessfulProject 337 2005 2008 POLAND Agribusiness PRIVATE L Satisfactory Satisfactory Some SuccessfulProject 338 2007 2008 ROMANIA Agribusiness PRIVATE L Good Good Some SuccessfulProject 339 2003 2008 RUSSIAN FEDERATION Telecoms Informatics & Media PRIVATE L Good Good None/Negative Partly SuccessfulProject 340 2006 2008 BULGARIA Agribusiness PRIVATE L Good Good Substantial Partly SuccessfulProject 341 2003 2008 BULGARIA Natural Resources PRIVATE L Satisfactory Good Some Partly SuccessfulProject 342 2001 2008 TAJIKISTAN Transport STATE L Marginal Satisfactory None/Negative Partly Successful

Appendix 2

Page 17 of 18



Approval



Operation Type1

Transition Impact2


Sponsor3


Change4

Overall Rating5

Project 343 2005 2008 RUSSIAN FEDERATION Power and Energy PRIVATE L Good Good Substantial SuccessfulProject 344 2001 2008 CROATIA Municipal & Env Inf PRIVATE L Excellent Good Substantial Highly Successful

1 E=Equity; L=Loan; G=Guarantee2 The range is Excellent/Good/Satisfactory/Marginal/Unsatisfactory/Negative3 The range is Excellent/Good/Satisfactory/Marginal/Unsatisfactory/Highly Unsatisfactory4 The range is Outstanding/Substantial/Some/None/Negative5 The range is Highly Successful/Successful/Partly Successful/Unsuccessful

Appendix 2

Page 18of 18

1993-2009 Technical Cooperation Operation Performance Evaluation Reviews (OPERs)

No. Operations Country Sector Industry TC Funds Amt

Type 1 OpsCom Approval

Funding Approved

Project CompletionReport (PCR) Date

OPER Report Date

Overall Rating2

19931 Privatisation Advisory Programme in the Russian Fed Russia State Privatisation 5,044 AS 16-Mar-92 May-92 - Dec-93 Successful2 Telecommunications Master Plan Albania State Telecoms 198 PP 09-Apr-92 May-92 - Dec-93 Partly Successful3 Banking Sector Restructuring Romania State Finance 855 SW/AS 07-Aug-92 Aug-92 - Jan-94 Successful4 Railway Sector Survey Regional State Transport 766 SW 17-Feb-92 Mar-92 22-Jun-93 Feb-94 Successful5 Roads and Road Transport Sector Survey Regional State Transport 409 SW 17-Feb-92 Apr-92 24-Sep-93 Feb-94 Successful

Subtotal 7,2721994

1 Regional Training Programme Regional State Finance 990 T 02-Dec-91 Jan-92 16-Feb-93 Aug-94 Partly Successful2 Tallinn Environment Project Estonia State Environment 158 PP 08-May-92 Oct-92 29-Nov-94 Dec-94 Partly Successful3 Tourism Development for Albania Albania State Tourism 223 AS 09-Apr-92 Apr-92 30-May-94 Jan-95 Partly Successful

Subtotal 1,3711995

1 Wine Sector Investment Programme Moldova State Agribusiness 440 PP/PI 19-Mar-93 Jun-93 21-Dec-94 Jul-95 Successful2 SME Sector Development Project Preparation Belarus State SME 174 AS 09-Jul-93 Dec-93 06-May-95 Jan-96 Successful3 State Railways Restructuring and Rail Modernisation Bulgaria State Transport 583 PP 22-Jun-94 Jul-92 03-Apr-95 Jan-96 Partly Successful

Subtotal 1,1971996

1 Romanian Banking Institute Romania State Finance 435 T 07-Mar-92 Apr-92 25-May-95 Aug-94 Successful2 Bulgarian Investment Bank Bulgaria Private Finance 942 AS/PP 30-Apr-93 Jun-93 11-Sep-95 Dec-94 Successful3 Budapest Wholesale Market Hungary State Agriculture 587 PP 08-May-92 Jul-92 28-Oct-93 Jan-95 Partly Successful

Subtotal 1,9641997

1 Unified Gas Supply System Russia State Energy 4,500 PP/PI 19-Feb-93 Apr-93 19-Jun-96 Jan-98 Successful2 INCAR JSC Enterprise Restructuring Russia State Restructuring 612 PP 15-Aug-93 Dec-93 02-Dec-96 Feb-98 Unsuccessful3 Perm Motors JSC Enterprise Restructuring Russia State Restructuring 862 PP 15-Aug-93 Dec-93 02-Dec-96 Feb-98 Partly Successful

Subtotal 5,9741998

1 Project Preparation TC MEI Investment Programme Croatia State Environment 179 PP 27-Nov-95 Feb-96 11-May-98 Jan-99 Partly Successful2 EC Phare/Tacis Framework Contracts for FIs Regional Private Finance 2,951 PP 18-Feb-94 Aug-94 - Jan-99 Successful3 Environmental Due Diligence of FI Regional Private Environment 3,264 PP/AS/PI 14-Mar-94 Aug-94 06-Mar-98 May-99 Successful4 Privatisation Advisory Programme Ukraine State Privatisation 2,730 PP/AS 03-May-91 Jun-92 28-Feb-95 Sep-98 Partly Successful5 Aktau Port Rehabilitation Kazakhstan State Transport 2,364 PP/PI/SW 28-May-93 Aug-93 20-Jan-94 Aug-98 Partly Successful

Subtotal 11,4891999

1 Mining Privatisation Kazakhstan State Mining 406 PP/AS 20-May-94 May-94 18-Mar-96 Oct-99 Partly Successful2 Municipal Utility Development and Investment Programm Ukraine State Environment 1,042 PP 22-Mar-96 Jun-96 16-Mar-98 Jan-00 Successful/ 3 Telecom Legislative and Regulatory Developmen Lithuania State Telecom 289 AS 02-Feb-96 Nov-96 05-Jan-00 Jan-00 Successful4 Swiss American Micro Enterprise Programme Moldova Private SME 1,078 PP 03-May-96 Aug-96 16-Jul-99 Jan-00 Partly Successful

Subtotal 2,8152000

1 Railways Modernisation Russia State Transport 844 PP/AS 01-Jun-93 Aug-93 19-Apr-96 Jul-00 Partly Successful2 Credit Worthiness of the City of Zagreb Croatia State Environment 184 PP 17-Oct-97 Jan-98 23-Mar-99 Jan-01 Successful3 SME Credit Line I and II Kyrgyz Republic Private SME 2,233 PP/AS/PI 04-Jun-93 Nov-93 01-Jun-95 Jan-01 Successful4 Power Market Twinning Programme Ukraine State Energy 1,297 PP/AS/PI 08-Mar-96 Mar-97 22-Feb-00 Jan-01 Unsuccessful

Subtotal 4,5572001

1 Telecommunications Emergency Reconstruction Projec Bosnia and Herz. State Telecoms 1,870 AS/PI 03-Oct-97 Dec-97 22-Feb-00 Jul-01 Highly Successful2 Mutnovsky Independent Power Plan Russia State Energy 1,319 PP/AS/PI 07-May-93 May-93 16-Sep-94 Dec-01 Partly Successful3 Road Rehabilitation and Upgrading Azerbaijan State Transport 755 PP 19-Apr-96 May-96 03-Aug-99 Dec-01 Unsuccessful4 Creditworthiness Assessment of City of Wroclaw Poland State Energy 481 AS/PI 25-Jul-97 Aug-98 - Jan-02 Successful

Subtotal 4,4252002

1 Turkmenbashi Port Development Projec Turkmenistan State Transport 2,895 AS/PI 19-Sep-95 07-Jul-95 16-Jul-99 Jun-02 Successful2 Enterprise Investment Demonstration Projec Kyrgyz Republic Private Finance 1,405 PP/PI 16-May-97 19-Jun-97 22-Feb-02 Aug-02 Unsuccessful3 Enguri Rehabilitation Project Georgia State Energy 453 PP/PI 04-Aug-95 18-Aug-95 17-Jun-97 Nov-02 Partly Successful4 Emergency Electricity Power Reconstruction Projec Bosnia and Herz. State Energy 2,150 AS/PP/PI/T 19-Jan-96 01-Jul-96 22-Feb-00 Mar-03 Highly/Partly Successful5 Tajikistan Overlay Network Tajikistan State Telecoms 457 AS/PP 06-Oct-95 21-Dec-95 21-Feb-97 Mar-03 Partly Successful6 Energy Efficiency TC Studies Russia State Energy 779 PP 07-Mar-97 01-Apr-97 05-Jun-02 Feb-03 Unsuccessful

Subtotal 8,139


1993-2009 Technical Cooperation Operation Performance Evaluation Reviews (OPERs)

No. Operations Country Sector Industry TC Funds Amt

Type 1 OpsCom Approval

Funding Approved

Project CompletionReport (PCR) Date

OPER Report Date

Overall Rating2

20031 Inst. Dev. & Mgt. of Baku Port Azerbaijan State Transport 991 PI 01-May-98 24-Jul-98 23-Sep-02 Apr-03 Successful2 Norsi Oil Refinery Russia Private Oil & Gas 1,165 PP 25-Jul-97 07-Aug-97 05-Jun-02 Aug-03 Partly Successful3 Env. Support to Budapest Bank Credit Line Hungary Private Finance 281 PP/PI 07-Jun-96 13-Feb-97 05-Jun-02 Dec-03 Partly Successful4 Technical Assistance to Uzbekneftegas Uzbekistan State Oil & Gas 1,443 PP/AS 03-Mar-95 01-Apr-95 05-Jun-02 Oct-03 Partly Successful5 Scoping Study for Railway Restructuring Project Bosnia & Herz. State Transport 199 PP 04-Apr-00 30-Jun-00 05-Jun-02 Mar-04 Successful6 Azeri Multi Bank Framework Financing Facility Azerbaijan Private Finance 3,227 PP/PI/AS/T 12-Jul-96 19-Aug-96 13-Nov-02 Feb-04 Partly Successful

Subtotal 7,3062004

1 Gostomel Glass Factory Ukraine Private Manufacturing 172 PP 30-May-01 07-Aug-01 13-Aug-03 Jul-04 Successful2 Air Navigation System Modernisation Tajikistan State Transport 500 PI 31-Oct-01 29-Oct-02 17-Sep-04 Aug-04 Successful3 KTZ Kazakh Rail TC Kazakhstan State Transport 976 PP/AS 28-Feb-97 14-Mar-97 20-Jan-04 Jan-05 Successful4 Romanian Ports Commercial Enhancement Prog Romania State Transport 320 PP 13-Feb-98 26-Aug-09 05-Jun-02 Jan-05 Partly Successful5 BGZ Pre-Privatisation Poland State Finance 4,161 PP 27-Feb-98 07-Apr-98 28-Jan-04 Feb-05 Successful6 Bydgoszcz Water Supply Poland State Municipal 779 PP/PI 08-Mar-96 16-Jun-00 05-Jun-02 Apr-05 Successful

Subtotal 6,9082005

1 Sakhalinmorneftegaz Russia State Mining 317 PI 13-Jun-01 25-Sep-01 07-Dec-01 Apr-06 Partly Successful2/3 The Mongolian Cooperation Fund2 Mongolia Private Various 6,247 PP/PI/AS/T 30-May-01 14-Jun-01 26-Aug-03 Oct-05 Successful

4 Privatisation of Electricity Distribution Companies Bulgaria State Energy 984 AS 06-Feb-02 07-May-02 26-Jul-05 Apr-06 Highly Successful5 Sofia District Heating Rehabilitation Bulgaria State Energy 1,552 PI 20-Jul-99 22-Nov-01 10-Feb-04 Feb-06 Successful6 Private Sector Road Network Management Poland Private Transport 1,262 PP/AS 23-Nov-99 12-Jun-00 06-Feb-03 Jan-06 Unsuccessful

Subtotal 10,3612006

1 SME/MSE Lines of Credit Belarus Belarus Private SME 721 PI 09-May-00 29-Jun-00 30-Jan-03 Apr-07 Successful2 Road Sector Reform Russia State Transport 1,412 PP/SW 28-Nov-01 08-Jan-02 17-Oct-02 Jan-07 Successful3 Warsaw Metro & PPP Task Force Support Poland State Municipal 1,486 PP 29-May-02 18-Dec-02 14-Mar-05 Jun-06 Partly Successful4 Belgrade Municipal Infrastructure Reconstruction Serbia State Municipal 598 PP/PI 24-Jul-98 30-May-01 05-Jun-02 Aug-07 Successful5 Municipal Environmental Loan Facility Romania State Municipal 654 PI 17-Dec-03 05-Feb-04 NA Mar-07 Successful6 Microfinance Bank of Azerbaijan Azerbaijan Private SME 3,436 PP/PI 04-Jun-01 18-Dec-01 01-Sep-03 May-07 Successful

Subtotal 8,3072007

1 Kombinat Aluminium Podgorica Montenegro Private Manufacturing 359 AS 07-Oct-04 08-Feb-05 ongoing Oct-08 Partly Successful2 Sofia Public Transport Bulgaria State Municipal 241 PP/PI 12-Feb-03 20-Jun-03 07-Oct-05 Jul-08 Partly Successful3 Surgut Municipal Services Development Programm Russia State Municipal 786 PP/PI 05-Sep-01 13-Sep-01 29-Aug-07 Feb-08 Successful4 Port of Dubrovnik Croatia State Transport 345 PI/AS 08-Sep-04 17-Dec-04 13-Mar-06 Feb-08 Successful5 Energy Efficiency and Renewable Energy Credit Line Bulgaria Private Finance/Energy 1,808 PP/PI 20-Nov-03 16-Jun-04 14-Mar-06 Mar-08 Successful6 Prioritisn. of Inv. Needs in Power Generation & Transmission Azerbaijan State Energy 253 PP 07-Aug-02 11-Dec-02 08-Mar-06 Mar-08 Partly Successful

Subtotal 3,7922008

1 ZTP Belgrade Reconstruction Project Serbia State Transport 2,938 AS/PI/PP 01-Aug-01 03-Oct-01 ongoing May-09 Successful2 Kazakhstan Atyrau Airport Project Kazakhstan State Transport 653 PI/PP 22-Jan-03 15-Dec-05 20-Aug-07 Oct-08 Unsuccessful3 Bucharest Multi-Sector Project Romania State Municipal 1,228 AS/PI/PP 05-Feb-02 13-Feb-02 28-Sep-05 Mar-09 Partly Successful4 MBASK Insurance Company Azerbaijan Private Finance 369 AS/PI 28-Oct-04 07-Dec-04 30-Jul-08 Feb-09 Successful5 Komi Municipal Water Services Russian Federation State Municipal 1,673 PI/PP 14-Nov-01 18-Jun-02 19-Dec-08 May-08 Successful6 Environmental Training for Financial Intermediarie Regional Private Finance 476 T 16-Jun-05 12-Dec-05 02-Nov-08 Mar-09 Successful

Subtotal 7,3372009

1 Bosnia & Herzegovina Regional Rail Projec Bosnia & Herzegovina State Transport 320 AS/PP 07-Apr-05 22-Aug-05 27-Aug-09 Sep-09 Partly Successful2 Dnipropetrovsk Municipal Water Corporate Development Ukraine State Municipal 300 PP 07-Oct-04 09-Jun-05 ongoing May-10 Successful3 Uzbekistan Telecommunications Regulatory Developmen Uzbekistan State Tecommunications 457 AS 05-Mar-03 03-Apr-03 05-Mar-07 May-10 Partly Successful4 Tbilisi Public Transport Project Corporate Development Georgia State Municipal 450 PI 12-May-05 13-Feb-06 ongoing May-10 Unsuccessful5 EU/EBRD SME Finance Facility Special Fund Regional Private Finance 55,339 AS/PI 08-Jun-99 18-Nov-99 NA Mar-10 Partly Successful6 Georgian Gas Transmission Pipeline Rehabilitation TC Georgia State Energy 304 PP 01-Feb-06 12-Jul-06 16-May-07 Mar-10 Successful

Subtotal 57,170

1 AS=Advisory Services; PP=Project Preparation; SW=Sector Work; T=Training; PI=Project Implementatio 2 Counts as two OPERs for workprogramme deliveryNote: The totals may not add up to the sum of the component parts due to rounding


1993-2009 Special Studies and Evaluation Progress Reviews

Operation Country Sector Industry EBRD Finance (EUR '000)

TC Funds (EUR '000)

Type 1 Board Approval

Report Publication

Date

Report Type

Russia Small Business Fund I Russia Private SME 1,734 2,851 Line of Credit/TC 26-Jul-93 Jul-94 Mid-Term ReviewRussia Small Business Fund II Russia Private SME 13,818 5,355 Line of Credit/TC 26-Jul-93 Mar-95 Mid-Term ReviewAgribusiness Project Preparation Units Regional State Agribusiness n.a. 4,590 TC 18-May-92 Sep-95 Special StudyRegional Bank Training Centre C.Asia State Finance n.a. 1,704 TC 10-Nov-92 Oct-95 Mid-Term ReviewProject Preparation TCs Regional State Various n.a. 8,349 TC n.a. Dec-95 Special StudyBelarus SME Credit Line Belarus Private Finance 20,806 1,420 Loan/TC 01-Nov-94 Jan-96 Mid-Term ReviewRegional Bank Training Centre TC Uzbekistan State Finance n.a. 1,704 TC 10-May-92 Sep-96 Evaluation Progress ReviewSME Credit Line Project Ukraine Private Finance 84,045 - Loan 29-Nov-94 Dec-96 Mid-Term ReviewWholesale Market Special Study Hungary State Agriculture 43,031 3,455 Loan/TC n.a. Jan-97 Special StudyKyrgyzstan SME Credit Line Kyrgyz Rep Private Finance 4,282 1,888 Line of Credit/TC 11-Nov-94 May-97 Mid-Term ReviewRussia Small Business Fund III Russia Private SME 370,914 32,707 Line of Credit/TC 26-Jul-93 Jul-97 Mid-Term ReviewRegional Venture Funds Russia State SME 269,298 20,814 Equity/TC n.a. Aug-97 Mid-Term ReviewBusiness Advisory Service Baltics Private Finance n.a. 4,196 TC n.a. Sep-97 Mid-Term ReviewFinancial Institutions Development Project Russia Private Finance 29,495 - Loan 23-May-94 Jan-98 Mid-Term ReviewTAM Programme Regional Private Priv/Restr n.a. 11,417 TC n.a. Feb-98 Special StudyRegional Bank Training Centre TC Uzbekistan State Finance n.a. 1,704 TC 10-May-92 Sep-98 Evaluation Progress ReviewSample of PCR Reviews and Assessments Various Private/State Various n.a. 7,377 TC Various Jan-99 Special StudySample of PCR Reviews and Assessments Various Private/State Various n.a. 9,445 TC Various May-00 Special StudyThematic Study on SME Support Various Private SME n.a. n.a. n.a. n.a. Jun-00 Special StudyTechnical Cooperation Funds Programme Various n.a. n.a. n.a. n.a. n.a. n.a. Jul-00 Special StudyScope Paper on Country Strategy Evaluation Kazakhstan n.a. n.a. n.a. n.a. n.a. 04-Oct-00 Aug-00 Scope for Special StudyNuclear Safety Account Various n.a. Energy n.a. n.a. n.a. n.a. Nov-00 Special StudySample of PCR Reviews and Assessments Various Private/State Various n.a. 11,941 TC Various Jan-01 Special StudyEvaluation of Environmental Performance Various n.a. Environment n.a. n.a. n.a. n.a. May-01 Special StudyPost-Privatisation Funds Various Private n.a. 122,424 18,871 Equity /TC n.a. Sep-01 Special StudyLegal Transition Programme Various n.a. Various n.a. 11,624 n.a. n.a. Oct-01 Mid-Term ReviewDirect Investment Facility Various SME Various 20,806 3,029 Equity 24-Feb-98 Nov-01 Mid-Term ReviewEnergy Efficiency of the Bank's Operations Various n.a. n.a. n.a. n.a. n.a. n.a. Feb-02 Special StudySample of PCR Assessments Various Private/State Various n.a. 7,023 TC Various May-02 Special StudyFinancial Institutions Development Programme Russia Private Finance 29,495 1,140 Loan 23-May-94 Aug-02 Special StudyEBRD’s Investments in Equity Funds Various Private SME 1,500,000 n.a. Equity Funds Various Oct-02 Mid-Term ReviewSample of PCR Reviews and Assessments Various Private/State Various n.a. 15,227 TC Various Feb-03 Special StudyRegional Trade Facilitation Programme Various Private Various 300,000 519 Guarantee/Loan/TC 13-Dec-94 Apr-03 Special StudyRussia Small Business Fund IV Russia Private SME 386,466 38,492 Line of Credit/TC 26-Jul-93 Jun-03 Special StudySample of PCR Assessments Various Private/State Various n.a. 5,458 TC Various Mar-04 Special StudyCountry Strategy Evaluation Slovak Rep. Private/State Various n.a. n.a. Equity/Loan/TC n.a. Mar-04 Country Strategy EvaluationTurnAround Management Programme Regional Private/State Various n.a. 4,447 TC Various Apr-04 Special StudyExtractive Industries Regional Private/State Various n.a. n.a. Equity/Loan Various Jul-04 Sector Strategy EvaluationMicrofinance Institutions Various Private SME 20,998 9,355 Equity/Loan/TC n.a. Sep-04 Special Study

Appendix 3Page

3of4

1993-2009 Special Studies and Evaluation Progress Reviews

Operation Country Sector Industry EBRD Finance (EUR '000)

TC Funds (EUR '000)

Type 1 Board Approval

Report Publication

Date

Report Type

Grain Receipts Programme Regional Private Agribusiness 273,525 831 Loan/TC Various Nov-04 Special StudyMSME Delivery Mechanisms Regional Private SME n.a. n.a. Equity/Loan/TC Various Jan-05 Special StudyPower & Energy Sector Strategy Review Regional Private/State Various n.a. n.a. Equity/Loan/TC n.a. Mar-05 Sector Strategy EvaluationCountry Strategy Evaluation Croatia Private/State Various n.a. n.a. Equity/Loan/TC Various Mar-05 Country Strategy EvaluationCountry Strategy Evaluation Azerbaijan Private/State Various n.a. n.a. Equity/Loan/TC n.a. Apr-05 Country Strategy EvaluationSample of PCR Assessments Various Private/State Various n.a. 6,586 TC Various May-05 Special StudyConditionality and Waivers Regional Private/State n.a. n.a. n.a. Equity/Loan/TC n.a. Oct-05 Special StudyDIF Programme Regional Private/State Finance 41,612 1,228 Equity/Loan/TC 24-Feb-98 Apr-06 Special StudyRegional Venture Funds Russia Private Finance 269,298 84,106 Equity/TC Various Apr-06 Special StudySample of PCR Assessments Various Private/State Various n.a. 5,977 TC Various Jul-06 Special StudyTelecommunications Sector Strategy Review Regional Private/State Telecoms n.a. n.a. Equity/Loan/TC n.a. Jul-06 Sector Strategy EvaluationProperty & Tourism Sector Strategy Review Regional Private/State Property &

Tourismn.a. n.a. Loan/TC n.a. Sep-06 Sector Strategy Evaluation

Achieving the Bank's Environmental Mandate through FIs

Regional Private Finance/ Environment

n.a. n.a. Equity/Loan/TC n.a. Nov-06 Special Study

Post-Privatisation Funds Regional Private Finance 122,424 18,871 Equity/TC n.a. Mar-07 Special StudyBusiness Advisory Services Regional Private Finance n.a. 33,470 TC n.a. Apr-07 Special StudySample of PCR Assessments Various Private/State Various n.a. 2,247 TC Various May-07 Special StudyFinancial Sector Operations Policy Regional Private/State Finance n.a. n.a. Equity/Loan/TC n.a. Sep-07 Sector Strategy EvaluationEnvironmental Policy of the Bank Regional Private/State Environment n.a. n.a. Other n.a. Jan-08 Special StudyEarly Transition Countries Fund Regional Private Various n.a. 5,466 TC Nov-04 Feb-08 Special StudySample of PCR Assessments Various Private/State Various n.a. 3,094 TC Various Apr-08 Special StudyAgribusiness Sector Operations Policy Evaluation Regional Private/State Agribusiness n.a. n.a. Equity/Loan/TC n.a. Jun-08 Sector Strategy EvaluationJapan-Europe Cooperation Fund Regional Private/State Various n.a. 5,170 TC Various Jul-08 Special StudyDanube River Basin Regional Private/State Environment n.a. n.a. Equity/Loan/TC n.a. Nov-08 Special StudyDirect Investment Facility Regional Private Various 132,442 2,961 Equity/TC 24-Feb-98 Dec-08 Special StudyInterim Evaluation of the Facility for Medium-Sized ProjRegional Private Various 50,000 n.a. Loan 04-Mar-08 Jun-09 Interim EvaluationImplementation of the CRR3 Regional Private/State Various n.a. n.a. Equity/Loan/TC 10-May-06 Jan-10 Special StudyThe Bank's Small Business Finance Operations Policies Regional Private Finance n.a. n.a. Equity/Loan/TC n.a. Feb-10 Sector Strategy Evaluation


APPENDIX 4a Page 1 of 10

LESSONS FROM INVESTMENT OPERATIONS EVALUATED IN 2009

A. TRANSITION IMPACT, POLICY DIALOGUE AND SECTOR REFORM

1. Transition impact at the sector/country level: policy dialogue and sector reform

Support to FDI in the country beyond the Project’s specific issues. Support to FDI in the country may require actions that go beyond the scope of one specific project. While addressing the specific problems encountered by one particular investor can have a significant demonstration effect, there may still remain barriers that prevent the entry of other players due to the lack of a level playing field. This highlights the importance of the Bank’s involvement in policy dialogue beyond the specific needs of the Project and the potential to achieve TI in the country once a positive relationship has been developed with the relevant authorities to support new projects and entry of new players.

Collaboration with state owned investment companies. State owned investment companies (like AIC), set up with the objective of promoting FDI and diversification of the local economy can become a valid interlocutor to address policy dialogue issues and a source to identify new projects of common interest. While government controlled, these type of entities are expected to operate on more transparent and market driven principles and are usually managed by management with previous private sector background and international experience. Bank’s investments can strengthen reform minded elements in these entities.

A more systematic effort to engage in policy dialogue may use a TC operation (Technical Cooperation) in parallel to the equity investment. The Bank has in the telecom and power sectors sometimes structured TC Operations alongside specific investment operations in order to achieve a higher impact on the sector framework. If the right parties in the target country can be engaged in the definition of the TC Operation this could enhance the probability of a meaningful impact at the sector level.

Electricity sector progress is not a simple linear development and needs constant attention and leveraging of EBRD’s exposure at a high level. EBRD has engaged early on in the electricity sector reform in Russia and may need to review how its relatively large sector and country exposure can facilitate a leveraging of its special status in Russia in the context of a sector dialogue with the highest levels. This may require a new focus of TC Operations in preparation of more fine tuned policy dialogue based on recent positive and negative sector trends. Whilst Bank debt and equity investments in the Russian electricity sector have risen sharply, TC operations should also be used more to contribute to sector transition issues.

Capacity Market issues could develop into serious pitfalls for continued reform progress and should be addressed directly via TC operations. EBRD cannot expect automatic solutions via project impact and sponsor activities. As IFI with the largest exposure in the electricity sector it is possible to focus on issues affecting all generators and contribute to solutions in the spirit of the reform focus.

The Bank should use its influence to broker optimal outcomes with regard to important electricity sector framework issues. The Bank should take a more pro active role in crucial areas which may affect the economics of all generators. The Bank should continue to identify the key political decisions that drive the value of strategic investments supported by the Bank and engage at an early stage with the Government and industry representatives in a policy dialogue that supports the balanced and open discussions and timely making of these decisions.

Addressing government corruption. EBRD correctly identified government corruption and the need for regulatory reform to be part of its additionality, and EBRD was partly brought into the project by the

APPENDIX 4a Page 2 of 10 sponsor for “political risk coverage.” EBRD should have been more willing to join with the EU in taking a serious stand against the past government and its potentially corrupt practices.

Helping to find appropriate regulatory responses to the crisis. Longer term development of the legal framework and financial market structure should include focus on building a local currency market. In the shorter term the Bank could work through policy dialogue to expand the scope of financial sector regulations to refine reserve and provisioning requirements on foreign currency loans, raise creditworthiness standards for foreign currency borrowers and prescribe limits for the open foreign currency positions of firms. More immediately, the Bank could consider buying local currency to fund partner banks for on-lending to the small business sector using appropriate risk mitigation mechanisms without compromising sound banking principles. EBRD should continue its efforts to stimulate consolidation of the banking sector in collaboration with other IFIs.

2. Transition impact at the corporate level: corporate governance

Independent corporate governance ratings can help improve corporate governance. EBRD should consider encouraging its partner banks to improve corporate governance by, among other things, seeking a corporate governance rating from an independent rating agency. Banks that use a rating agency take steps to improve, and to validate improvements, in corporate governance can better merit and attract IFI financing. IFIs could also encourage their partner banks to seek a corporate governance rating. The rating process can enhance the effectiveness, independence, and transparency of EBRD’s “Institution Building” programme with a bank and help lead to more visible, clearly bench-marked results of improved corporate governance. Also, it can provide valuable support to the efforts of EBRD’s minority board nominees to improve corporate governance. Independent supervisory boards, good corporate governance of banks, and sound banking effects. It is difficult for management to restrain growth and limit concentrations in booming markets. This depends on the adoption of sound policies overseen by an active and independent supervisory board. An independent and active supervisory board, dominated by non-executive directors, can be the source of adherence to policies that management may wish to deviate from to seize profit opportunities offered by unsustainable boom markets. The positive results of good corporate governance should become evident in the form of balanced asset and liability portfolios and prudent growth rates based on adherence to sound and adaptive strategies, policies and procedures.

Domination of the supervisory board by independent outside directors improves corporate governance. The supervisory board of a bank must be independent and active. A supervisory board dominated by management can hardly impose limitations on asset and funding growth rates and diversification that constrains management initiative. EBRD should favour active and independent supervisory boards dominated by independent, outside directors There is little evidence from experience that a minority investor’s board nominee, be it EBRD’s or anyone else’s, can impose prudential limits on management where management dominates the supervisory board.

Keep clients focussed on the need to establish and maintain an effective corporate governance framework. EBRD experience with this client has shown that, while principal shareholders and management may be attracted by the idea of establishing effective corporate governance mechanisms, they may regard the issue as being of secondary importance in a period of rapid business growth or in a period of crisis. The Bank must use all possible levers of influence to keep clients focussed on sound corporate governance.

Ensure planned and progressive refreshing of the Board. While recognising the difficulty of identifying suitably experienced and qualified candidates, the Bank should ensure that any term beyond six years for a non-executive director is subject to particularly rigorous review, and should take into account the need for progressive refreshing of the Board. This is likely to be especially relevant where there is a dominant shareholder or long-serving senior management.


Ensure the independence of external auditors. The audit committee of the board should have primary responsibility for making a recommendation on the appointment, reappointment and removal of the external auditors. The Bank should encourage large clients to consider alternatives by inviting tenders for external audit services at least every five yeas. Where there is no change of auditing firm, the Bank should require the client to ensure that the partner in charge of the external audit is rotated after no more than five consecutive years. If the auditor also provides non-audit services, particular care must be taken to ensure that auditor objectivity and independence is safeguarded.

Before consenting to a new Board appointment in a client company, the EBRD should consider carrying out a full background integrity check. EBRD clients should be made aware of the importance the Bank places on independence from potential political influence on its activities and the activities of clients. In particular it should be stressed to clients that the EBRD requires to be advised in full and without delay if the client contemplates appointing a Politically Exposed Person (PEP) to a position of influence.

3. Transition impact at the corporate level: institution building

Institution building programmes and corrective action plans should be designed to meet the strategic objectives and operational and training needs of clients. These programmes should be tailor made to match the client’s identified needs. Strategy development should encompass a detailed business plan with both long-term goals and clearly specified milestones and benchmarks. Consultants should be chosen who understand the environment in which clients operate as well as possessing the requisite professional skills. Planning and goal setting for technical assistance assignments should be a participative process involving EBRD, the partner bank and the consultants.

It is important to support regional banks, including Moscow based banks with a regional network, by providing dedicated long-term funding accompanied with appropriate institution building, financial and operational covenants. In some cases commitment from the client can be more important than a track record in MSME lending. Experience has shown that technical assistance to transfer know-how and standards can be quickly assimilated by committed partner banks. Follow-on funds in many cases will still be additional because of the high risk profile of regional banks in the eyes of the international commercial market.

In case of multiple project exposure in large manufacturing conglomerates, far-reaching systemic changes should realistically be expected. The Bank has had an on-going relationship with Severstal and has introduced multiple EAPs. EvD has previously argued that as new projects are implemented, if the Bank seeks to achieve systemic change, it is necessary to return to prior agreements; assess what has and has not been achieved, and incorporate outstanding obligations into the new project agreement. This project gave the Bank significant leverage to seek multi-project follow-up and systemic change, yet that opportunity was missed.

The organisational structure within a rapidly growing client should evolve to ensure effective management. A rational management structure is essential for the adoption of sound policies and the operation of effective controls, including risk management. It is possible that in the present case deficiencies in the organisational structure contributed to failure to implement necessary changes in policies and processes in a timely manner. When it becomes apparent that the adoption of sound business practices is being impeded, the Bank should encourage the client to review and restructure its management organisation appropriately.

B. ADDITIONALITY

Transparency in definition of Project objectives. Providing political comfort through an equity

APPENDIX 4a Page 4 of 10 partnership with the sponsor to address concerns of operating in difficult political environments is a valid rationale for the Bank’s involvement and can lead to high additionality by enhancing the Bank’s attributes. The Board document should reflect faithfully the real goals of the project and rate the importance of goals to reflect the reality of the project.

Experience has shown subordinated debt to be an effective instrument to provide capital support, in addition to providing comfort to main shareholders and management and demonstrating EBRD commitment to the long term development of a local bank. EBRD subordinated debt facilities play an important role in an environment characterised by a shortage of long term funding and lack of risk appetite in the market for funding of this nature. Subordinated debt facilities are strictly overseen by the regulator who sets the criteria that have to be met to qualify for capital adequacy purposes. The EBRD should continue to maintain constructive policy dialogue with the Central Bank with the aim of achieving a higher degree of convergence with international standards over time.

Limited risk equity structures (portage or quasi portage) may be useful instruments to give the Bank a significant presence in a company from which to conduct policy dialogue in exchange for a limited funds contribution. By structuring the deal in the manner of a defined exit based on a put/call agreement, the Bank can be given a significant share ownership in a company from which to address policy dialogue without incurring either major investment or excessive risk. Neither the amount nor the return need to be particularly spectacular in this type of investment, where the goal of the Bank’s funds is not so much to provide financing for the Company’s capex investment as to acquire a temporary presence in the company to address specifically policy dialogue related issues.

C. SOUND BANKING

1. Role of the Sponsor

Importance of strong strategic investors. The role played by the Sponsor in improving the relationship with the government while maintaining its high operating standards further highlights the importance of dealing with strong and responsible strategic investors found in many of the Bank’s projects and in previous lessons learned. While the Bank’s policy dialogue has supported the Company’s efforts to improve its relationship with the government, the Company’s high standards provided confidence that it would not yield to pressures and cut corners. As an example, the Company did not commit to keeping some wet process kilns operating post commissioning of the dry line.

The selection of a strong lead shareholder with industry experience is a key requisite for start up success. In a start up company it is very important to find a strong lead shareholder with relevant industry experience and the capability to support operations through technical expertise and if required with additional financial commitments. The important (but non-existing) position of the main shareholder (Sponsor) cannot be substituted by a more pro-active Board trying to micro-manage the Company from the Board room (“Too many cooks can spoil the broth…”).


2. Due diligence and appraisal

The onus is on the team to confirm key information received from the sponsor, either by checking with an appropriate source or an independent source. The strong interest in closing a deal with a new client in a new sector and with positive potential TI does not preclude the importance of conducting proper due diligence and requesting information from the client to verify certain requests.

Institutional capacity analyses are a key appraisal due diligence element and need to cover the entirety of a project intervention, i.e. also including envisaged institutional and sector reform agendas. The Client’s commitment to deliver (or intention to deliver) institutional reforms needs to be assessed against its related own capacity and taking into account other key stakeholders with major influence on the Client. Deficiencies noted in this respect need to be addressed through commensurate TC support whereby note needs to be taken of the fact that the implementation of reform agendas might have different requirements than those of the project intervention in the narrower sense.

At project preparation and appraisal operation teams should ensure that they possess full information about material financial obligations within the client’s holding structure. Where relevant, Board documents should include details of the extent of leverage employed to support the acquisition of an entity to which the Bank proposes to take exposure. A leveraged buyout may entail risks to the stability of the client even where the provider of the debt has no recourse to the client. The new owners may be expecting to fund the acquisition debt through dividends from the client, thereby reducing the amount of cash available for business development or to meet other obligations. In case of non-payment or default, a restructuring may result in ownership changes that could be of concern to the Bank. In keeping with the policy of “Know Your Customer”, operation teams should address any risks of this nature and report the results to the Board in the Board document for the proposed operation.

Sensitivity analysis should to be based on long-term energy prices. In calculating IRRs for energy efficiency sub-projects, sound banking dictates that bankers should carry-out the sensitivity analysis based on long-term energy prices to set investment priorities. This is particularly true when energy prices were peaking, as was the case with this project. Some degree of flexibility is required in the design and execution of energy efficiency investment programmes, as investment priorities might change when energy prices change considerably.

Adverse seasonal weather conditions need to be factored into implementation schedules. It is recommended that the contractual durations for future projects are adjusted to the historic set of weather conditions. It is also recommended that the winter season is excluded from the actual construction season. This will enable more realistic contract management and administration of MDB projects in Azerbaijan without time consuming contractual adjustment problems otherwise.

Partner bank features and attributes. In designing and implementing MSME frameworks and operations, the following partner bank features and attributes should be assessed:

• Partner bank shareholders and management are committed to targeting the MSME sector.

• There is evidence of firm strategic commitment of the parent or sponsor where applicable.

• Partner banks are willing to dedicate sufficient staff and resources to maintain continuity.

• Partner banks are willing and have the capability to finance MSME lending with own resources (funds from non-IFI sources) when they have assimilated and internalised the lending methodology.

• There is clear alignment of the operation or framework goals with the partner bank’s strategy and business plan.

APPENDIX 4a Page 6 of 10 The above features and attributes are essential if MSME lending by partner banks is to be sustained beyond the completion of an EBRD framework or operation.

3. Design and Structuring

Importance of clear and transparent sound banking structures. While defined return equity structures can be a very valid investment instrument, especially when the goals of the Project go beyond the mere providing of funding but aim to achieve TI through the Bank’s presence as a shareholder in the invested company, it is advisable that the project structure be based to the best extent possible on transparent and sound banking principles. Project structure may need to respond to the complexities of the negotiation process with the Company/Sponsor, but it should aim to achieve to the possible extent certain degrees of symmetry and risk/reward compensation. In general, it is advisable that limitation of upward return via a cap be compensated with a limitation of the downside risk with a predetermined floor.

Firm equity commitments should be in place from the outset based on a realistic assessment of the start-up period for a new airline. Later fund raisings may become extremely difficult especially in cases where existing shareholders are unwilling to increase their equity investment. The restrictions defined for approving new shareholders should be kept to a minimum to enhance the chances to find additional equity sources.

A start up venture in the airline sector should not move forward until a fully empowered CEO has been appointed. A start up venture depends to a large extent on a strong CEO supported by a senior management team. The Bank should consider the appointment of a CEO acceptable to the Bank as an important pre-requisite for a start up venture prior to releasing draw downs under the Bank facilities.

Resolving shareholders disputes through adequate mitigants. Shareholder disputes in this sector are relatively common and should have been addressed with mitigants in the shareholders agreement. Legal measures alone may not be sufficient and may need to be supported with a stronger effort at the outset to get to know the personalities of the key shareholders much better in order to reduce the risk of surprises.

Sound banking valuation rationale: A defined exit return structure does not preclude a proper company valuation. In structures with a defined return on the Bank’s equity investment, the valuation of the Company and hence the stake in the company given in exchange for a specific equity contribution may not be so relevant from a purely economic point of view; however it is important to have a solid rationale for the valuation of the company and to take it into account for the Bank’s equity contribution as a sound banking and market transparency practice. Evaluation based on different methods (DCF, multiple based,…) should constitute a reference and not just a justification for the ultimately negotiated price and substantial deviations from theoretical value should be properly justified.

Clarity of project phasing. When a project potentially has multiple phases, the Bank should avoid describing and appearing to commit to more than the immediate project. However, if the Bank wishes to take a multi-project approach, then it will need to be able to address follow-on impacts in the context of the initial project, which should be screened and categorized, based on the full anticipated impacts of all phases.

4. Implementation and Monitoring EBRD handling of MSME frameworks. The experience of the present Framework and other operations highlights certain elements of Bank handling that are necessary to ensure effective


completion and continuation of MSME lending when the EBRD supported programme has ended. The following components are necessary in the case of most frameworks:

• A tailor-made corrective action plan and / or institution building programme is agreed with the client.

• The operation team checks periodically that partner banks are internalising and assimilating the procedures, methods and lessons conveyed by technical assistance.

• Where EBRD head office has a role in project implementation or client relations, there is close involvement of and liaison with the EBRD resident office.

• The operation team ensures that partner banks clearly understand the undertakings they give when accepting EBRD conditions such as sub-loan eligibility criteria and environmental reporting requirements.

• The operation team makes appropriate checks to ensure that a partner bank’s systems and procedures are capable of meeting the Bank’s requirements for reporting and other conditions.

• The EBRD must be prepared to allocate significant staff resources to monitoring when working with partner banks. The commitment of additional resources is likely to be needed for monitoring in the case of partner banks that have limited prior experience of the target sector.

EBRD should monitor a partner bank’s commitment to the MSME sector, reconfirming the commitment before proposing additional funding. In the cases under review, the evidence is that the banks concerned intend to develop MSME business further post-crisis when conditions permit. However, instances have been observed where banks, having previously expressed and to some degree demonstrated commitment to serving the MSME sector, have in fact not developed and assimilated appropriate credit procedures and risk mechanisms. Resulting losses have prompted them to turn away from MSME business.

Loan covenants prescribed by the approved operations procedures need to be adhered to, or a prior change of these introduced. The Bank should be careful not to do too loose a structure when prescribing standard loan covenants. It is important to apply the covenants to all loans in accordance with the Bank’s operations procedures, or to seek specific approval when making exceptions. The Credit Department should consider requiring more stringent adherence to the covenant guidelines, especially during boom periods when banks are competing with each other by relaxing their requirements.

In view of the absence of a PIU and the employment of a LE, the project implementation progress and status needs to be assessed regularly by the Bank in detail and underpinned by in-depth visual inspections and documentary evidence. Client reporting is not considered a sufficient substitute for comprehensive and independent monitoring assessments, nor are brief monitoring visits by the Bank which substantially rely on oral client reporting, supplemented by brief investment site visits.

In a multi-financier project participation, the success of loan execution and institutional reform is considerably enhanced through close coordination amongst these stakeholders, eventually with one party taking the lead in the coordination processes. Given the often notable differences between MDB mandates, project loan agreements, pertinent procurement rules and other differences, establishing of a coordinating forum, eventually lead by the party with the highest leverage potential, ensures a better overall project outcome than a fragmented approach by each intervention party individually.

D. ENVIRONMENT

APPENDIX 4a Page 8 of 10 Engagement with the government by ESD on important policy issues, such as improving environmental standards, could result in recognition of shared objectives, thus leading to broad support for additional investments. In the cement industry, the dry process is more energy efficient and produces less pollution. Even with the capex required to build the new plant, the cost of production will be less and economically justifies the investment. The change to dry process production and closing down the wet lines should be based on a simple business case.

Clear definition of the Project expected outcomes and implications. ESD and the project team should coordinate to clearly define and quantify the main project objectives and to provide clarity to the Board as to the realistic expected achievements. If full compliance with EU Best Available Techniques guidance can not be achieved, but the project will still result in very significant positive environmental change, it should be made clear in the Board document what these expected achievements are. Finally, it is important that any final changes made to the environmental sections of the Board document are ultimately reviewed and approved by ESD prior to submission to Board.

For benefit optimisation, energy efficiency investments in manufacturing companies ultimately need to adopt a holistic planning approach. While concentrating on certain pre-conceived energy-wasting facilities may constitute a cost-efficient and strategically necessary first approach - and indeed may help to ‘open the door’ for related further policy dialogue and future investment opportunities – it is not sufficient to allow for benefit optimisation. A holistic approach needs to be adopted to either underpin the pre-conceived choice or to lead to an alternative selection. For complex manufacturing structures, a strategic approach focusing on few investment items with high visibility first, coupled with a policy dialogue, can suffice to generate the necessary demonstration stimulus. Alternative or complementary considerations may be given to the financing of corporate management assistance (“soft assets”) to further the course of energy efficiency savings.

Benefit objectives and their monitoring need strengthening. Where energy efficiency benefits are the main rationale for the Bank’s investment, these benefits need to be independently monitored with the same rigor and at similar frequency as the investment implementation status and progress. Prior to this, benefit targets and achievement schedules would need to be covenanted and thus form the basis for their monitoring.

An integrated approach is a reasonable way forward for large industrial clients. For large industrial clients with existing older and complex operations, it is unreasonable to expect that a single operation will not allow the company to become fully IPPC/BAT compliant. Therefore EvD argues that it is reasonable to undertake a “programmatic approach” with such a client, whereby (1) an initial overall assessment of the client’s operations and needs are made (this has yet to be done for Severstal) and from this a series of projects are developed and prioritized. As each project is successfully completed, the client and the Bank move onto the next project. Such an approach would assure that commitments made under the initial project (in this case the EAP under the working capital facility) are in fact carried forward and fully implemented.

Legacy heavy industry, low costs, and pollution. Experience has shown that legacy assets in EBRD’s countries have been a good source of production capacity that cost little to acquire and operate, and thereby conferred a strategic competitive advantage. The assets, however, impact the environment more than permitted by EU and even local laws. Upgrading the assets to meet higher standards, and to comply with EBRD’s environmental mandate, threatens their cost competitiveness. It is on this basis that asset owners have been able to negotiate compliance holidays for highly polluting assets, even within the EU. The continuing poor environmental performance that is permitted by compliance holidays conflicts with EBRD’s environmental mandate and poses a dilemma when considering projects that support the operation of such assets.

‘Significant improvements’ and EBRD’s Environmental Mandate. Projects that propose to significantly improve the environmental performance of a legacy industrial asset need features that match the ambition. Hindsight has shown that attaching an ESAP to the loan agreement has not resulted


in significant improvements. It seems that the Bank must implement its environmental mandate more forcefully by directly financing investments that reduce environmental impacts to a highly significant degree (as defined by an adequate policy as separately recommended), rather than by relying on ESAP implementation for which the Bank is not providing financing. This is a repeat lesson that the Bank should heed to comply with its environmental mandate.

Engagement with NGOs. In potentially controversial mining projects, early and extensive engagement is necessary to resolve potential conflict and move to best practice standards. Civil society has an important voice in raising issues that are of concern, particularly if they go beyond the immediate project boundaries. Full public disclosure and consultation is a must in mining projects.

Occupational health and safety. The only effective way to address occupational health and safety is to take a harsh approach at the start of a project. Once this chance is missed, it becomes more difficult to impose new standards. Based on this experience the Bank can provide leadership to other sponsors who may find it a challenge to change working culture with respect to occupational health and safety.

E. EBRD ORGANISATIONAL ISSUES

Equity investment deals should be led by a separate equity team or a fund which is less partial than the banking team with the existing exposure and relationship. Leadership of the equity deal structuring should be carried out by a strong private equity team with the full support of the relationship manager or by an independent fund manager. This is particularly important when the Banking Team has a long established lending relationship with the client. A fresh and detached look and different negotiating style are required which can only be delivered by an experienced equity team. An important market test of the agreed transaction structure may in some cases be to include a savvy private equity co-investor alongside the Bank prior to final commitment. Of great importance is also the allocation of substantial staff time for the monitoring of the equity investment. This may be easier for a dedicated fund manager.

FX lending to SMEs and retail borrowers vis banks. EBRD should review in depth its approach to developing local capital markets and take a much more restrained approach to promoting foreign currency funding of retail and SME lending operations. The fact that EBRD has had little other sources of alternative financing schemes should not encourage the Bank to take unwarranted risks for itself, its clients and the final borrowers of these products. The Bank should consider recommitting itself to its founding mandate to establish local capital markets, which is a long term effort that could have born more fruit over the past decade.

Information base about relevant current and past exposure by the Bank needs strengthening. For learning and risk mitigation purposes, the Bank should note in its appraisal documents all current and past exposures to the borrower. This would enable affiliates and strategic shareholders, to the best of the Bank’s knowledge, to relate affirmative statements by the responsible operation team and would enhance the confidence in the Bank’s due diligence process by the decision-maker, i.e. the Board, and the public at large.

A more holistic approach needs to be adopted by the Bank in large-scale manufacturing projects with particular focus on project interfaces. Project preparation and implementation work in the Bank are often pursued by designated banking teams in isolation, although at times conscientiously for process expediency or other reasons. Inherent synergy and cross-project leverage potentials and further reaching developmental requirements should be more systematically explored.

Bank’s integrity due diligence ex ante and during monitoring stages need to be enhanced. The Bank’s standard procedures prescribe “enhanced customer due diligence measures” where so-called politically exposed persons (PEPs) are involved. For Bank investments with clients controlled by so-called ‘business oligarchs’ – integrity-wise both groups appear hardly distinguishable - an independent

APPENDIX 4a Page 10 of 10 integrity investigation should be carried out routinely as part of project due diligence and to safeguard against Bank reputation risk. Complementary, usual project monitoring needs to be accompanied by an independent compliance and reputation risk monitoring mechanism.

APPENDIX 4b Page 1 of 3

LESSONS FROM TC OPERATIONS EVALUATED IN 2009

A. TC PREPARATION AND DESIGN

The Client should always play a leading role in budget preparation and consultant selection. The Client knows its own project and local conditions better than EBRD. Involving the Client closely in the budget preparation and consultant selection is the best way to ensure that the assignment is appropriate to the Client's needs. EBRD should support the Client with guidance, either directly through its own experts or through consultants with expertise in western procurement practices. In the project under consideration, consultant selection was delegated to EBRD for logistical reasons. As the Bank moves further east, logistics will become increasingly challenging. This should not be allowed to create a barrier to the Client's close involvement at all stages of project preparation.

Limiting consultants to an advisory role enhances institutional development. The Consultant in this case was not contracted to run the PIU but instead to serve in an advisory role. The PIU was staffed by existing employees of the water company, who managed the project with the support and advice of the Consultant. As well as saving money on expensive consultancy fees, this approach enhanced the institutional development aspects of the project as the local staff obtained hands-on experience in project management.

Staff the Project Implementation Unit with employees of the company, not with outside specialists. Contracting specialists to staff the PIU for the duration of the project does not assure skills transfer to the company as a whole, nor to other municipal companies in the area. Such specialists are likely to seek out further specific projects elsewhere. Creating a specialist and mobile group of highly skilled consultants in this way inhibits the uptake of new practices in the regular management of the industry. Similarly, existing staff seconded at a much higher salary than normal are unlikely to return to their regular positions and pass on their new skills to colleagues once the project is complete. A better result is obtained if clients allocate internal staff to the project for a limited period and then return them to positions within the company or at other municipal companies in the city.

Consultants should not make assumptions about the level of knowledge already existing in the client company. Consultants engaged in institutional development programmes may waste valuable time and resources if they do not first assess the business practices already followed by the Client at the start of the assignment. Such an assessment may indicate that the Client is competent in modern business practices and has already instigated many management reforms. In this case, the focus of the assignment can be changed from remedial work to a more forward-looking programme better suited to the existing level of expertise of the Client. Pushing ahead with basic training in such a situation not only wastes valuable TC resources, but is likely to alienate the Client and its staff. Design and implementation of technical assistance accompanying MSME frameworks. Banks participating in the present Framework displayed varying degrees of commitment to the technical assistance component of the programme. The following elements should be present in the design and implementation of technical assistance to ensure that partner banks internalise and assimilate the procedures, methods and lessons conveyed by TC consultants: • As in the present case, consultants must be thoroughly familiar with the operating environment. • Benchmarks should be set against which to measure periodically the effectiveness of consultant

performance and the responsiveness of partner banks. • The outcomes expected from training exercises led by consultants should be benchmarked and

assessed in conjunction with the client. • Consultants should report regularly to the operation team and the operation team should take steps


to coordinate the activities of different teams of consultants exchanging information as appropriate. • The requirement for partner banks to report portfolio data and performance when receiving EBRD

funding accompanied by technical assistance should extend beyond ‘graduation’ from the TA component for the full duration of the credit line.

B. TC ADMINISTRATION AND RESOURCE MANAGEMENT

The Bank should consider working towards the adoption of the logical framework approach in TC operations with particular regard to the implementation of multi-annual programmes involving substantial grant funds. The LogFrame should be based on solid baseline data, and include carefully chosen indicators as well as defined means of verification. This would allow for a regular monitoring of the programme’s achievements at different levels (e.g. client institution, sector) and at different points in time.

Active measures should be undertaken to increase competition from consultancy companies in TC assignments, especially if large programmes stretch over several years. In selected cases of evidenced ‘market failures’ brought about by overly dominating consulting firms, the option to suspend those firms from further participation in the framework or individual segments could be discussed by the Banking team together with CSU (subject to the Bank’s Procurement Policies and Rules).

C. TC MONITORING AND SUPERVISION

Do not suspend the Project Completion Report (PCR) from ‘call-off’s of substantial size (e.g. exceeding €200,000) and/ or with significant individual transition impact potential. If, as in this case, other reporting templates are agreed with the donors, the Operation Team should ensure that the typical self-assessment topics, such as client commitment, consultant performance, transition impact and lessons learned are added and/ or attached.

Discuss possibilities of expanding the Bank’s current monitoring procedure for substantial TC programmes. One option would be to review the TI potential separately for call-off contracts at the level of TC Com, in the same way it is done for regular TC standalone projects. Another option is to systematically aggregate the TIMS results of individual call-offs in order to allow for sector-wide conclusions i.e. the impact of the entire framework programme. The discussion of options shall have associated resource requirements in mind, in order to come to an acceptable cost-benefit ratio. EBRD operation teams should monitor the progress of technical assistance operations closely and intercede immediately if it appears that a lack of client commitment may lead to failure of the assignment. The commitment of senior management is essential to the success of TC supported consulting assignments, especially when the aim of the TC is to introduce improvements in client systems and business operations. It is equally important that a senior staff member of the client is nominated as the consultant’s main counterpart during the assignment and that the counterpart has adequate time to devote to the various stages of the assignment and has sufficient back-up support. (Bank Caspian, Kazakhstan, PE09-458/BK)

D. TC AND ITS ROLE IN SECTOR REFORM

Fundamental sector reforms in circumstances marred by high stake ethno-political controversies are unlikely achievable through bilateral MDB investment efforts, let alone TCs. In order to improve prospects for fundamental sector reforms in highly charged political environments the Bank must seek strategic alliances with other powerful co-investors – which it did in this case, but possibly not enough – and also widen its toolkit by a consolidated high-level policy dialogue – which it did not in this case. Commensurately, larger scale investment projects need to


be disaggregated into phases and disbursement progress would need to be linked to clear milestone achievements. TCs can play a crucial role in such phased approach, but this is unlikely in an “under-powered” situation such as the one at stake.

Appendix 5 Page 1 of 5

EVALUATION DATABASE The ECG Good Practice Standards for Private Sector Evaluation require that the AEOR should include an annex profiling the important characteristics of the evaluated sample (e.g., sector, investment size, and percentage of operations affected by specific loss provisions) against the population. This information is presented below. 1. DESCRIPTION OF THE EVALUATION DATABASE 1.1 Selection and size of the database The evaluation database used in Chapter 1 and Appendix 8 consists of evaluations conducted since 1996. Since 2009 it has been selected on a random basis, as described further in Appendix 8, section 1.3. The evaluation database consists of 1,087 operations totalling €16,796 million of EBRD commitments. As some operations have been grouped for evaluation purposes, this results in 679 individually rated projects. The evaluation year 2009 added 61 evaluated operations to the database; 24 evaluated through OPERs, 5 through Special Studies and 32 through XMR Assessments. XMR Reviews were conducted on a further 40 operations which were ready for evaluation but were not selected in the random sample of projects to be evaluated. 1.2 Timing of the evaluations The operations in the evaluation database were self-evaluated by the operation team, on average, 23 months after final disbursement of the Bank's investments and 55 months after Board approval. The Good Practice Standards for Private Sector Evaluation of the Evaluation Cooperation Group specify that the project should have reached "early operating maturity"1 before evaluation takes place. EvD has implemented this requirement since 2004. In the period since then projects in the evaluation database have been self-evaluated by the operation team, on average, 7 months after early operating maturity and EvD has published independent evaluation reports an average of 15 months after early operating maturity. 1.3 Further details on the composition of the evaluated portfolio. The 1,087 operations in the evaluated portfolio are made up of 734 straight debt operations (including portage equity with a fixed return), 277 equity operations and 76 combined debt and equity. Of the straight debt operations, 292 have been repaid, 226 have been fully or partially prepaid and 24 have been fully or partially written off. Of the straight equity operations, 130 have been divested normally and 44 fully or partially written off. Of the 76 combined debt and equity operations, 17 have been repaid and divested normally, 17 have been at least partly written off and 19 have been at least partly prepaid. In terms of investment types, 637 operations are senior debt only, 293 are ordinary shares only (including portage equity) and 66 are a combination of the two. 27 operations consist of subordinated debt only, and 14 are a combination of senior and subordinated debt. 12 are a mix of ordinary shares and preference shares and 10 are guarantees or off-balance sheet items. The remainder are other combinations of these products and other participating interests.

1 An operation has reached early operating maturity when (a) the project financed will have been substantially completed, (b) the project financed will have generated at least 18 months of operating revenues for the company and (c) the MDB (EBRD) will have received at least one set of audited annual financial statements covering at least 12 months of operating revenues generated by the project.

Appendix 5 Page 2 of 5 2. COMPARISON OF THE EVALUATION DATABASE WITH THE EBRD'S CUMULATIVE PORTFOLIO 2.1 Geographic distribution Table 1 overleaf shows the geographic distribution of the evaluation database, compared with all operations ready for evaluation and with the Bank's cumulative portfolio (all signed operations, whether complete or still active). The evaluation database covers all EBRD's countries of operations with the exception of the two most recent additions, Mongolia and Turkey. EvD has evaluated some Technical Cooperation operations in Mongolia and an investment operation in that country has become ready for evaluation in 2010. So far no operations in Turkey are ready for evaluation. Table 1: Geographic distribution of the evaluation database

EBRD country of operations Evaluation Database

Evaluation database %

Ready for Evaluation


%

Bank cumulative portfolio

Bank cumulative portfolio %

<REGIONAL> 43 4% 66 5% 161 6% ALBANIA 11 1% 17 1% 37 1% ARMENIA 30 3% 32 2% 71 3% AZERBAIJAN 24 2% 27 2% 95 3% BELARUS 8 1% 10 1% 26 1% BOSNIA AND HERZEGOVINA 20 2% 28 2% 67 2% BULGARIA 41 4% 56 4% 122 4% CROATIA 38 3% 60 4% 85 3% CZECH REPUBLIC 24 2% 30 2% 53 2% ESTONIA 25 2% 33 2% 46 2% FYR MACEDONIA 19 2% 31 2% 43 2% GEORGIA 33 3% 41 3% 108 4% HUNGARY 52 5% 63 4% 108 4% KAZAKHSTAN 39 4% 52 4% 104 4% KYRGYZ REPUBLIC 15 1% 25 2% 64 2% LATVIA 22 2% 25 2% 32 1% LITHUANIA 21 2% 34 2% 40 1% MOLDOVA 19 2% 26 2% 65 2% MONGOLIA 0 0% 0 0% 21 1% MONTENEGRO 1 0% 6 0% 16 1% POLAND 98 9% 132 9% 205 7% ROMANIA 61 6% 99 7% 198 7% RUSSIAN FEDERATION 260 24% 315 22% 544 19% SERBIA 29 3% 36 2% 94 3% SLOVAK REPUBLIC 32 3% 44 3% 65 2% SLOVENIA 27 2% 33 2% 45 2% TAJIKISTAN 12 1% 15 1% 43 2% TURKEY 0 0% 0 0% 4 0% TURKMENISTAN 6 1% 8 1% 11 0% UKRAINE 50 5% 78 5% 194 7% UZBEKISTAN 27 2% 29 2% 54 2%

Grand Total 1,087 100% 1,451 100% 2,821 100%

Charts 1 to 3 below show the same data grouped into broader regions. They confirm a good level of correspondence between the evaluated sample and the Bank's cumulative portfolio. Compared to the Bank's total signed operations, EvD's evaluated sample slightly over-represents projects in CEB at the expense of the other regions, particularly EEC and SEE. Most of the projects ready for evaluation in the first few years of the bank's existence were in Central Europe


or Russia, reflecting the Bank's portfolio at the time. More recent commitments have a higher share in countries of the CIS and in south-eastern Europe, many of which are not yet ready for evaluation. To date, Turkey accounts for less than 1% of the Bank's signed operations. Chart 1: Cumulative signed Chart 2: All projects ready Chart 3: EvD’s EBRD operations for evaluation evaluated sample

11%

24%

20%6%

19%

20%

0%

CA CEB EECRegional Russia SEETurkey

9%

30%

15%5%

22%

19%0%


9%

31%

15%4%

4%

17%0%


2.2 Sectoral distribution The evaluation database covers 63 of the 77 individual industry classifications used by the Bank's cumulative portfolio. Table 2 shows the distribution of evaluated projects across the Bank's sector teams, and compares this with projects ready for evaluation and all operations signed by the Bank. The evaluation database covers all the sector teams of the EBRD and the distribution of evaluated operations across the sectors corresponds quite closely to the distribution for the cumulative portfolio. Table 2: Sectoral distribution of the evaluation database

Sector Team (SIC) Evaluation Database

Evaluation database %



%

Bank cumulative portfolio

Bank cumulative portfolio %

Agribusiness 118 11% 165 11% 343 12% Bank Equity 83 8% 123 8% 173 6% Bank Lending 167 15% 242 17% 534 19% Equity Funds 57 5% 74 5% 129 5% Insurance & Financial Services 59 5% 73 5% 179 6% Manufacturing and Services 181 17% 228 16% 374 13% Municipal & Env Inf 58 5% 81 6% 192 7% Natural Resources 51 5% 58 4% 101 4% Power and Energy 38 3% 52 4% 106 4% Property and Tourism 36 3% 54 4% 108 4% Small Business Finance 105 10% 123 8% 285 10% Telecoms Informatics & Media 67 6% 81 6% 121 4% Transport 67 6% 97 7% 176 6% Grand Total 1,087 100% 1,451 100% 2,821 100%

Charts 4 to 6 present the comparative sector distribution in terms of the broader sector groups used in Chapter 1 and Appendix 8. In these figures, Equity Funds remain part of the Financial Institutions group, despite a recent internal reorganisation that group that sector with Industry

Appendix 5 Page 4 of 5 and Commerce for management purposes. In the past, Financial Institutions has been clearly under-represented in the number of projects evaluated. The problem was mentioned in the report "Evaluation Coverage in EBRD" (CS/AU/08-36) and has been addressed by making Financial Institutions a priority sector in which EvD aims to achieve a higher coverage ratio than in other sectors. Charts 4 to 6 show that this approach is already having an effect and the situation has now improved substantially. The evaluation database provides a close match to the cumulative portfolio in terms of numbers of projects in each broad sector group. Chart 4: Cumulative signed Chart 5: All projects ready Chart 6: EvD’s EBRD operations for evaluation evaluated sample

46%

34%

7%

13%

Financial Institutions Corporate

Energy Infrastructure

44%

36%

8%

12%



44%

37%

8%

11%



2.3 Facility Risk Rating The following charts present overall portfolio facility risk ratings as at 31 December 2009. There representation here is very good. Chart 11.7: Cumulative signed Chart 11.8: All projects ready Chart 11.9: EvD’s EBRD operations for evaluation evaluated sample

14%

61%

25%

1-4 5, 6 & 6W 7-10

16%

62%

22%

1-4 5, 6 & 6W 7-10

16%

61%

23%

1-4 5, 6 & 6W 7-10


2.4 Conclusion The sample of 679 evaluated investment projects by the end of 2009 provides a good representation of all the projects ready for evaluation. There is also a good representation of the signed portfolio. The evaluated sample will always take a few years to reflect gradual changes to country and sector patterns in the signed portfolio, as it takes time for the more recent projects become ready for evaluation. It has been seen that a previous under-representation of Financial Institutions operations has already begun to be corrected through a focus on this area in 2009.


ASSESSMENT OF STRENGTH OF TRANSITION POTENTIAL & CHECKLIST OF TRANSITION CRITERIA/OBJECTIVES

FOR EX ANTE AND EX POST APPLICATION

ASSESSMENT OF STRENGTH OF TRANSITION POTENTIAL

1. COUNTRY SECTOR AND REGIONAL CONTEXT

a. Current stage of transition (advance transition country or otherwise) b. State of sector reform and development (largely unreformed or otherwise) c. Conditions for market entry and competition (few players versus strong competitive

pressures)

2. THE TRANSITION CHALLENGES FACING SECTOR, COUNTRY AND REGION

a. Market reform objectives in the Bank's country or sector strategy b. Economic priorities facing the country c. Application of the transition indicators (TI Checklist) - Structure and extent of markets - Market organisations, institutions and policies that support markets - Business behaviour and practices

3. THE WAY CHALLENGES ARE ADDRESSED IN THE SELECTION AND DESIGN OF THE PROJECT

a. Consistency with Bank country/sector strategy; b. Key project covenants and undertakings (strong set of transition-related covenants

is likely to be a sufficient sign of transition potential; it is not a necessary condition); c. TC components (TC-funded programmes that can help achieve some of the transition

objectives); d. Policy dialogue

CHECKLIST OF SEVEN TRANSITION CRITERIA/OBJECTIVES

PROJECT CONTRIBUTIONS TO THE STRUCTURE AND EXTENT OF MARKETS

1. GREATER COMPETITIVE PRESSURES Project contributes to greater competition in the project sector: efficiency, innovation

and customer orientation of other suppliers through competitive pressure. To what extent does the project directly improve the competitive environment and/or

extend the use of market-type mechanisms in the economy? (e.g. more rational pricing, significant new entry into the market, setting new quality or technical standards that other firms must follow, trade facilitation, etc.)

2. MARKET EXPANSION VIA LINKAGES TO SUPPLIERS AND CUSTOMERS Stimulation of competitive behaviour through the project entity's interactions with

suppliers (backward/upstream linkages) and clients (forward/downstream linkages); project contributions to the integration of economic activities into the national, regional or international economy, in particular by lowering the cost of transactions.

(a) To what extent does the project change the market behaviour of local suppliers of inputs? (backward linkages);

(b) To what extent does the project change the market behaviour of downstream marketing and/or processing activities of customers? (forward linkages)

APPENDIX 6 Page 2 of 2

CHECKLIST OF TRANSITION CRITERIA/OBJECTIVES (CONT.)

PROJECT CONTRIBUTIONS TO MARKET ORGANISATIONS, INSTITUTIONS

AND POLICIES THAT SUPPORT MARKETS

3. INCREASED PRIVATE SECTOR PARTICIPATION Significant increase or consolidation of private provision of goods and services,

including provision of public goods and services and support for entrepreneurial initiative (e.g. unbundling in infrastructure projects).

To what extent does the project contribute directly to increased private ownership?

4. INSTITUTIONS, LAWS, REGULATIONS AND POLICIES THAT PROMOTE MARKET FUNCTIONING AND EFFICIENCY

Creation/strengthening of public and private institutions that support the efficiency of markets; improvements to the functioning of regulatory entities and practices; contributions to government policy formation and commitment, promoting competition, predictability and transparency; contributions to laws that strengthen the private sector and the open economy. Improved legislation, regulation and legal and regulatory implementation.

To what extent is the project associated with institutional spin-offs effects giving rise to improvements in the functioning of existing institutions or in the establishment of new institutions and practices important for a market-type economy?

PROJECT CONTRIBUTIONS TO BUSINESS BEHAVIOUR AND PRACTICES

5. TRANSFER AND DISPERSION OF SKILLS Project contributes to significant upgrading of technical and managerial skills in the

economy beyond the project entity. To what extent does the project create, upgrade or transfer new skills relevant to a market

economy? (e.g. management, marketing, financial and banking skills, specialised technical skills, etc.)

6. DEMONSTRATION EFFECTS FROM INNOVATION Demonstration of (replicable) products and processes which are new to the economy;

demonstration of ways of successfully restructuring companies and institutions; demonstration to both domestic and foreign financiers of ways and instruments to finance activities. New ways of financing restructuring instruments.

To what extent does the project create a new and easily replicable line of activity? (demonstration effects, e.g. in manufacturing or finance, incl. new modes of financing industrial projects, new products, enterprise restructuring)

7. HIGHER STANDARDS OF CORPORATE GOVERNANCE AND BUSINESS CONDUCT Improved governance standards that are highly visible and invite replication in non-

project entities. To what extent does the project give rise to improvements in corporate governance and/or

the business culture? (incl. fostering entrepreneurship, improving decision-making processes, encouraging innovation and strategic thinking in business)


TRANSITION IMPACT ANALYSIS AIRLINE

TI checklist categories

STEPS OF RATING TRANSITION IMPACT

EX POST

Short-term verified impact

Longer- Term transition impact potential

Risk to potential

TI

STEP I: CHANGE BY THE PROJECT AT CORPORATE LEVEL

Rating1 Rating2 Rating3

3 Private ownership From the outset this was structured as a private company.

N/A N/A N/A

5 Skill transfers CEO was expected to contribute low cost carrier experience, also introduction of IOSA qualification

Satisfactory Good High

6 Demonstration effects Foreign built aircraft, electronic ticketing/low cost carrier features etc.

Good Good High

7 New standards for business conduct Board of international financial investors with one local airline Sponsor. The Bank’s decision to exit highlights the concerns regarding the new majority shareholder.

Marginal Marginal High

STEP II: TRANSITION IMPACT AT THE LEVEL OF THE INDUSTRY AND THE ECONOMY AS A WHOLE

Rating Rating Rating

1 Competition The Company clearly increased the level of competition on its routes and met with very strong price competition from existing carriers

Good Good Medium

2 Market expansion The target clients of the Company are to some degree new groups which had not been flying much before due to costs. Some progress but not to the extent expected. Only a low percentage of locals use air travel.

Satisfactory Satisfactory Medium

3 Private ownership Illustrated that a private airline can be more efficient in some areas.


4 Frameworks for markets Efforts also by the Company led to a change of the rule that an airline has to provide a free meal, now charging extra is legally possible. Conditions in the country are difficult for low cost carriers.


5 Skills transfers The management team built up by the Company has contributed to skills transfers through rotation

Satisfactory Satisfactory Low

6 Demonstration effects The Company demonstrated new ticketing system, more efficient turn around, lower cost base due to adoption of some low cost carrier operating principles, however no stimulation of other low cost carriers given local conditions.


7 New standards for business conduct Relevant international standards were applied for accounting, air safety, Board etc. However, no impact on the other players in the air transport sector.

Unsatisfactory Unsatisfactory

SUMMARY OF VERIFIED, POTENTIAL AND RISK RATINGS Marginal Marginal High

1 This range is: Excellent/Good/Satisfactory/Marginal/Unsatisfactory/Negative. 2 This range is: Excellent/Good/Satisfactory/Marginal/Unsatisfactory/Negative. 3 This range is: Low/Medium/High/Excessive.


OVERALL TRANSITION IMPACT RATING:4

MARGINAL

4 This range is: Excellent/Good/Satisfactory/Marginal/Unsatisfactory/Negative.


CEMENT PLANT



EX POST



Risk to potential

TI



3 Private ownership Although the Bank did not participate in the privatisation process of the Company, the Bank’s support has facilitated the further expansion of a private multinational company in the country.


5 Skill transfers The Sponsor is one of the leading multinational companies in the cement sector. Its acquisition and investment in the Company has brought its technical, managerial and environmental skills to the Company and the country.

Good Good Low

6 Demonstration effects The Sponsor’s operational procedures have set up a reference for the Company’s operations.

Good Good Medium

7 New standards for business conduct It is the Sponsor’s policy to implement its high business conduct standards in its invested subsidiaries worldwide. The presence of the Bank as a shareholder has also contributed to improve transparency through systematic reporting.

Good Good Low/ Medium



1 Competition Support to a fair international market player is expected to attract new investors to a market characterized by unmet demand, increasing competition in the sector. The project did not contemplate capacity increase.


2 Market expansion New downward linkages created with distributors.


3 Private ownership The Bank’s support to the Company is expected to attract new private players to the market.


4 Frameworks for markets The active policy dialogue undertaken by the Bank with the local authorities has materialised in specific legal changes to the charter of the government agency for promoting foreign investments.

Good Good Low/ Medium

5 Skills transfers Technical and environmental improvements brought by the Sponsor to the Company will disseminate to the market as a result of management and workers mobility.

Satisfactory Good Low

6 Demonstration effects The presence of the Bank in the Company and active policy dialogue contributed to solve differences with the government that led to the Company’s decision to undertake the largest FDI in the country outside the oil sector. This has sent a positive message to the market as to the prospect for foreign companies to invest and operate on fair terms in the country.

Excellent Excellent Medium/ High

7 New standards for business conduct The improvements brought by the Sponsor, in particular environmental, have had an effect in upgrading the standards in the cement sector in the country.

Good Good Medium

SUMMARY OF VERIFIED, POTENTIAL AND RISK Good Good Low/


RATINGS Medium

OVERALL TRANSITION IMPACT RATING:

GOOD


EBRD COLLABORATION WITH A MAJOR WESTERN BANK



EX POST



Risk to potential

TI



3 Private ownership Since 2004, EBRD’s projects have indirectly supported the client's acquisition of banks in countries of operation. The client's approach to these banks was to rapidly increase their foreign currency lending, expand the loan to deposit ratio, and FX mortgage lending.

Marginal Marginal High/

Excessive

5 Skill transfers EBRD’s projects helped the client transfer skills into its banking units in countries of operation. In one country, however, the client sold its established greenfield bank rather than merge it with a newly acquired bank, thereby increasing the integration and training and management integration challenges at the newly acquired bank.

Satisfactory Satisfactory High

7 New standards for business conduct EBRD’s projects helped the client grow its foreign exchange lending, especially in the mortgage product, against the advice of the IMF and the BIS. EBRD’s projects signalled consistent approval by EBRD of the client's expansion strategy that was based largely on a rising loan to deposit rate fuelled by cross-border foreign currency loans in a macro environment of unsustainable foreign debt burdens at the country level in the region.

Marginal Unsatisfactory High



1 Competition EBRD’s projects helped the client compete in its markets, complementing the client's strongly competitive approach and rapid expansion in each market and eastwards. This has spurred competition by other banks, mainly in foreign currency lending into a region where such lending was on an unsustainable trend. Competition was fierce as Western banks exported their model of financing mortgages and consumer debt with foreign currency loans.

Marginal UnsatisfactoryHigh/

Excessive

2 Market expansion EBRD’s projects helped the client deepen its banking franchise in each market, benefiting customers (including SMEs) and depositors. This has spurred competition forwards and backwards, but excessively through foreign currency loans to SMEs and retail customers, and at excessively high rates of growth assets and high low/deposit ratios.


3 Private ownership

NA NA NA

4 Frameworks for markets EBRD’s projects helped the client apply its credit and governance policies within its network and in relationships with local borrowers. There was potential to pioneer consolidated regulatory supervision in cooperation with the client's national authorities and the host countries, but this was not pursued. Supervision has been remote and low on on-site inspection. Supervisors allowed great leeway the client to apply the Basle II IRB approach to

Marginal Satisfactory Low


allow it to set its own regulatory capital weightings in volatile emerging market economies showing signs of unsustainably rapid financial system growth and over-reliance on cross-border credit in foreign currency.

5 Skills transfers Former loan officers of the client have moved into other banks.

Good Good Low

6 Demonstration effects EBRD’s support of mortgage lending in foreign currencies ignored IMF and EvD advice and has had adverse consequences for many borrowers.

Unsatis-factory Marginal

High/ Excessiv

e

7

New standards for business conduct EBRD had important integrity issues with a subsidiary of the client that have received an unclear resolution. The client has undertaken some improvements to its anti-corruption measures in coordination with EBRD's Office of the Chief Compliance Officer; this is promising.

Marginal Satisfactory Medium

SUMMARY OF VERIFIED, POTENTIAL AND RISK RATINGS Marginal Marginal High

OVERALL TRANSITION IMPACT RATING: The broad reach of EBRD’s work with the client across the region, which has fuelled rapid asset growth on rising loan to deposit ratios and high dependency on foreign currency lending, in many cases to retail borrowers without foreign currency income, justifies the rating.

MARGINAL


EBRD RELATIONSHIP WITH A LEADING LOCAL BANK



EX POST



Risk to potential

TI



3 Private ownership

N/A N/A N/A

5 Skill transfers Mortgage lending standards were introduced as part of the syndicated loan; however, the loan has not been fully utilised.


6 Demonstration effects N/A N/A N/A 7 New standards for business conduct

Long delayed improvements in corporate governance have gathered pace and the client has strengthened credit procedures which have still to be tested.

Satisfactory Good Medium



1 Competition 2 Market expansion 3 Private ownership 4 Frameworks for markets 5 Skills transfers 6 Demonstration effects 7 New standards for business conduct

A full assessment of realised transition impact will only be possible if and when the bank emerges from the crisis and resumes a prudent development path. For the time being the evaluation team rates realised transition impact as Satisfactory to reflect the successful IPO and loan syndication. Remaining potential is rated Good with High risk, although the risk may become Substantial if the authorities seek to exert undue influence on the bank's operations through the stake held by the national investment fund.

SUMMARY OF VERIFIED, POTENTIAL AND RISK RATINGS

Satisfactory Good High


Satisfactory


EBRD RELATIONSHIP WITH A LOCAL BANK



EX POST



Risk to potential

TI



3 Private ownership NA NA NA 5 Skill transfers

The Projects, especially the securitisation, helped to strengthen the Bank's processes and practices in capital markets and structured finance.

Excellent Excellent Moderate

6 Demonstration effects NA NA NA 7 New standards for business conduct

The Projects, especially the securitisation, helped to strengthen the Bank's processes and practices in transparent business conduct.




1 Competition The projects supported the well-managed and increasingly transparent development of a leading bank in the country, in competition with other major banks.


2 Market expansion The projects supported expansion into SME and retail lending, leading to the recognition of a strategic fit with another local bank that was more developed in those fields.


3 Private ownership The projects demonstrated to others that good corporate governance and transparency help build access to international capital markets and that these markets are available to support private entrepreneurial initiative in the country.


4 Frameworks for markets NA NA NA 5 Skills transfers

The projects had modest skill transfers beyond the firm. Good Good Low

6 Demonstration effects The merger of two local banks helped to demonstrate to others that good corporate governance and transparency help build access to international capital markets and that these markets are available to support private entrepreneurial initiative in the country.


7 New standards for business conduct The projects were among many transactions in the country that helped foster transition to a market economy in banking and capital markets.

Good Good Low

SUMMARY OF VERIFIED, POTENTIAL AND RISK RATINGS Excellent Excellent Moderate


Excellent


HYPERMARKET CHAIN



EX POST



Risk to potential

TI



3 Private ownership From the outset this was structured as a private company.

N/A N/A N/A

5 Skill transfers The Company adopted state of the art models (Walmart) and offered training to employees.

Good Good Low

6 Demonstration effects The Company has contributed to the rollout of new shop formats in the country, first in a major city and then in selected regions

Good Good Low

7 New standards for business conduct The intended corporate governance improvements were delayed due to the fight during two years between the two main shareholders

Marginal Marginal Medium



1 Competition The Company clearly increased the level of competition and offered the consumers more choice

Good Good Medium

2 Market expansion The Company has contributed to market expansion by speeding up the conversion from old style retail to modern hypermarket format in a major city and various regions of the country

Good Good Medium

3 Private ownership To some extent the Company has shown that conflicts between local and international shareholders can impact performance.


4 Frameworks for markets No particular measurable impact. Interaction with local authorities etc..


5 Skills transfers In-house training scheme and high staff rotation contribute to skills transfer to others in this sector


6 Demonstration effects The Company has built up good loyalty scheme for regular clients and built strong customer base.

Good Good Low

7 New standards for business conduct Relevant international standards were applied for accounting etc. Some delays in terms of expected Corporate Governance improvements due to shareholder fight.

Good Good Low

SUMMARY OF VERIFIED, POTENTIAL AND RISK RATINGS Satisfactory Satisfactory Medium


SATISFACTORY


MINING PROJECT

TI checklist

categories STEPS OF RATING TRANSITION IMPACT


Longer- Term

transition impact

Risk to potential

TI

STEP I: CHANGE BY THE PROJECT AT CORPORATE LEVEL (PARTICIPATING BANKS)


3 Private ownership The mine is owned and operated by a western company.

Good Good Low

5

Skill transfer The Sponsor has introduced best practice across the company.

Good Good Low

6 Demonstration effects Neighbouring locally owned mines are copying the company’s approach to occupational health and safety.

Good Satisfactory Low

7 New standards for business conduct The previous owners faced bankruptcy, thus the changes made by the Sponsor are inspirational. There was a real risk that the company cold have been shut down and caught up in litigation.

Good Good Low

STEP II: TRANSITION IMPACT AT THE LEVEL OF THE INDUSTRY (SUB-PROJECTS) AND THE

ECONOMY AS A WHOLE


1 Competition There are neighbouring locally owned mines – mainly copper - in the immediate facility.

Good Good Medium

2 Market expansion Expected development of other mines in the country, either by the Sponsor or by others, has not happened. Accusations of corruption at the level of the central government have had a negative impact on market perceptions of the country.


3 Private ownership Although the primary client is private, the government has negotiated for partial ownership of the processing plant.

Good Satisfactory Medium

4 Frameworks for markets Concerns about the negative role the government has played in the context of this project, has had a negative impact on the market perception of the country.

Good Satisfactory High

5 Skills transfers The Sponsor has introduced a number of best practice procedures which have been copied by neighbouring mines.

Good Good Low

6 Demonstration effects The Project had a positive demonstration effect in terms of introduction of best practice, but the role of government has had a negative demonstration effect.


7 New standards for business conduct The project allowed for full implementation of EU environmental standards.


SUMMARY OF VERIFIED, POTENTIAL AND RISK RATINGS Good Good Medium


GOOD


POWER GENERATION



EX POST



Risk to potential

TI



3 Private ownership First investment supported sale of strategic stake to private sector investor.

Good Good Medium

5 Skill transfers The Sponsor brings in its experience and resources.

Excellent Excellent Low

6 Demonstration effects Setting standards for other generators.

Good Good Medium

7 New standards for business conduct Improved S&P ratings, leading the sector.

Good Good Low



1 Competition The company is an efficient producer with well located plants.

Good Good Medium

2 Market expansion The company contributes to the new sector model

Good Good Medium

3 Private ownership Improved consideration of minority shareholders.

Good Good Medium

4 Frameworks for markets The company maintains dialogue with the relevant authorities. Impact appears limited. The Bank did not attach a specific TC.

Marginal Good Medium

5 Skills transfers The company maintains dialogue with the relevant authorities. Impact appears limited. The Bank did not attach a specific TC.


6 Demonstration effects Co-investment with IFI equity and debt.

Good Good Low

7 New standards for business conduct The Sponsor's standards being introduced.

Good Good Low



Good


POWER GRID COMPANY



EX POST



Risk to potential

TI


Rating5 Rating6 Rating7

3 Private ownership The Company is a monopoly transmission operator for the entire country and the government owns more than 75% of the share capital.

N/A N/A N/A

5 Skill transfers EBRD insisted on public sector procurement rules to be applied and required some changes in procurement rules (consideration of financial strength of tender participants, site visit prior to tender etc.). Overall the Company found this exchange of experience (which caused a delay of about 6 months in procurement) useful and experienced may be used in future procurements. However, no other project elements with a focus on perceived weaknesses in the transmission expansion planning processes and other capabilities of the Company at the corporate level. (TC on pricing regulation is covered below under Framework for Markets)


6 Demonstration effects With the exception of procurement process, no specific project elements with a focus on corporate governance and/or business culture across departments.


7 New standards for business conduct The issues related to the needed upgrade of the transmission expansion organisation within the Company and amended allocation of responsibilities between the various industry participants were not discussed and also not addressed by the Bank Loan and TC Operations.




1 Competition The project-related TC operation recommended a return on assets-based (RAB) pricing model regulation for transmission and distribution and this was promoted in working groups. However, the tariff regulator lacks ability to judge which investment projects suggested by the regulated company are essential and which are not. It lacks a methodology for measuring performance of wires companies. Implementation of RAB regulation appears remote and the evaluation team doubts whether it will be implemented in the near future, if ever.


2 Market expansion For the objective of efficient functioning of the emerging competitive wholesale electricity market in the country, the actual implementation of RAB would be important.


3 Private ownership The Bank Operation did not contain specific elements in this direction. However, in terms of general spirit it was meant as a support signal to the head of the national power company in his efforts regarding the power sector reform programme.

N/A N/A N/A

5 This range is: Excellent/Good/Satisfactory/Marginal/Unsatisfactory/Negative. 6 This range is: Excellent/Good/Satisfactory/Marginal/Unsatisfactory/Negative. 7 This range is: Low/Medium/High/Excessive.


4 Frameworks for markets A relatively small TC attached to this Loan Operation led to the recommendation of RAB -ype regulation of transmission and distribution as recommended by various others. There were also some working groups/meetings between Consultant/EBRD and the Company to broaden support for RAB. Implementation of RAB regulation for the Company appears remote.

Marginal Satisfactory High

5 Skills transfers The limited size and scope of the TC Operation did not have this focus.


6 Demonstration effects The Bank Operation led to the mobilisation of long tenor funding from other commercial banks in local currency. The loan was an unprecedented local currency transaction which was structured and syndicated by the Bank. The Bank views this loan as crucial for the local currency interbank market. Positive financial additionality.

Good Good Medium

7 New standards for business conduct The Bank Operation did not have project elements addressing these areas.


SUMMARY OF VERIFIED, POTENTIAL AND RISK RATINGS Marginal Satisfactory Medium


MARGINAL


POWER MODERNISATION PROJECT (MID-TERM REVIEW)

TI checklis

t categor

STEPS OF RATING TRANSITION IMPACT EX POST


Longer- Term

transition impact

Risk to potential TI



3 Private ownership Not Applicable 5 Skill transfers

The plant appreciated the supplier’s training for operation and maintenance. International tendering and procurement skills were also acquired. IAS-based financial statements enforced the Company to acquire accounting skills.


6 Demonstration effects Not Applicable 7 New standards for business conduct

The Company is certified for ISO 9001 although it was not discernable at the plant.




1 Competition The Company has defended its market position against the threat of the new entrant. It appears progressively difficult.

Satisfactory Marginal High

2 Market expansion Power generation has a single buyer and a single supplier (the state-owned coal mining company), therefore forward and backward linkages are limited. The new boiler could have expanded the market if it had been successful to date.

Not Applicalbe

3 Private ownership Not Applicable 4 Frameworks for markets

The Bank exerted influence on the Government with regard to (i) prohibiting the disposal of assets owned by the state-controlled enterprises; and (ii) including investment surcharge in the payment from the buyer. The Government accepted these.


5 Skills transfers Not Applicable 6 Demonstration effects

The Government, politicians, the Company, and the plant and the Bank expected highly the successful installation and operation of the project boiler as it could generate a considerable replication and could bring various benefits to the country and sector. The installation is still under the test awaiting the positive outcome.

Marginal

To be assessed

after completion

High

7 New standards for business conduct The technology could have been a breakthrough for power plants in the Region. If successfully completed, the plant will be the standard setter.

Marginal

To be assessed

after completion

High

SUMMARY OF VERIFIED, POTENTIAL AND RISK RATINGS Marginal Satisfactory High

OVERALL TRANSITION IMPACT RATING: The Project had large potential for transition. The delayed completion has limited the positive impact on STPP, Donbassenergo and the sector. The completion is still awaited.

MARGINAL


ROAD RECOVERY PROJECT


STEPS OF RATING TRANSITION IMPACT EX POST



Risk to potential

TI STEP I: CHANGE BY THE PROJECT AT

CORPORATE LEVEL Rating Rating Rating

3 Private ownership Not Applicable 5 Skill transfers Not Applicable 6 Demonstration effects Not Applicable 7 New standards for business conduct

The Project introduced IFI procurement procedures for tendering and FIDIC contract terms for works. Other IFI requirements for planning and project management also induced the institutional changes through PIU. However, the transition in the asgency itself is likely to take a longer time.




1 Competition Not Applicable 2 Market expansion

Most of contracts were awarded to local contractors. The Project contributed to the revival of the local construction / road rehabilitation market and strengthened backward linkages. Though the local contractors’ financial capacity appears to be fragile in the challenging crisis time.

Good Good High

3 Private ownership Not Applicable 4 Frameworks for markets

The new Roads Act was enacted and the Roads Recovery Plan was prepared. However, the sustainable maintenance scheme still needs to be established. Motorway tolls have increased significantly in real terms and the domestic and foreign tolls have been equalised in 2009.

Satisfactory Good Low

5 Skills transfers Not Applicable 6 Demonstration effects

The sizeable joint financing (€170 million) for the road sector could increase the economic viability and could catalyse external financing or private sector investors for the highly-trafficked motorways in the future.


7 New standards for business conduct The FIDIC contract terms realised the EU standard design and traffic safety on the project sections, which make traffic flow faster and safer.

Good Good Low

SUMMARY OF VERIFIED, POTENTIAL AND RISK RATINGS Good Good Low

OVERALL TRANSITION IMPACT RATING: The most notable transition impact was to deliver the EU standards in road design and rehabilitation works, in the institutional awareness and practices. The large joint financing amplified the leverages on the borrower, which facilitated the compliance with several transition covenants.

GOOD


ROAD PROJECT

TI

checklist categorie


EX POST


Longer- Term

transition impact

Risk to potential

TI



3 Private ownership. Private ownership was not a target under the Project N/A N/A N/A

5 Skill transfers. Learning about and applying the international procurement rules, as well as achieving with environmental and social compliance, mandated by the Banks, are the key features of the Project

Good Good Medium

6 Demonstration effects. The international procurement rules and procedures, achieving environmental and social compliance became the norm with the PIU and management of the Road Service. The Evaluation team had no evidence whether transfer of new skills took place within the rest of the company.


7 New standards for business conduct. The Project stimulated the competitive behaviour through the Road Service's interaction with the international supply chain in an IFI procurement environment. IFRS accounting and auditing have yet to permeate across the Road Service.




1 Competition. Creating competition was not an explicit target under the Project. N/A N/A N/A

2 Market expansion. Although not envisaged some forward- and backward linkages have been observed, but it is unclear at this point whether the momentum achieved through the Bank’s intervention will be maintained.

N/A N/A N/A

3 Private ownership. Privatisation was not a target under the Project. N/A N/A N/A

4 Frameworks for markets. The reform led to a new Road Law, increase in road sector funding and service level agreement between the Ministry of Transport and the Road Service. Formation of the Roads Advisory Board and improvements to the maintenance setup are yet to be achieved.


5 Skills transfers. Through subcontracting to local contractors, a transfer of skills and expertise is believed to have occurred beyond the Road Service. In consideration of Project specifications, some effects are expected to have happened with the local goods and material suppliers working with the international contractors.

Good Satisfactory Medium

6 Demonstration effects. These were not targeted with the Project N/A N/A N/A

7 New standards for business conduct. These were not targeted with the Project. N/A N/A N/A

SUMMARY OF VERIFIED, POTENTIAL AND RISK RATINGS Satisfactory Satisfactory Medium


SATISFACTORY


STEEL PRODUCER



EX POST



Risk to potential

TI



3 Private ownership The project supported, ex-post, the sustainability of a major privatisation.

Excellent Excellent Medium

5 Skill transfers The sponsor is a major steel group with great expertise in restructuring legacy steel mills.


6 Demonstration effects The borrower appears to have significantly complied with key investments required by the privatisation, investing a large amount in needed capex.

Good Good Low

7 New standards for business conduct The sponsor, consistent with his business interest, has worked to remove non-transparent and corrupt business practices at the plants. Transfer pricing within the Group, however, may be an issue.

Good Good Low



1 Competition The borrower has improved its operations and competes vigorously with other large local and foreign firms. It reportedly also imports coal in order to drive down the prices of locally supplied coal.

Good Good Low

2 Market expansion The company restructuring has created some backward linkages.


3 Private ownership The project supported, ex-post, the sustainability of a major privatisation.


4 Frameworks for markets NA NA NA 5 Skills transfers

The capital investments have called on local suppliers to deliver construction and other services to a higher standard.


6 Demonstration effects The borrower appears to have significantly complied with key investments required by the privatisation, investing a large amount in needed capex.

Good Good Low

7 New standards for business conduct The sponsor, consistent with his business interest, has worked to remove non-transparent and corrupt business practices at the plants. Transfer pricing within the Group, however, may be an issue.

Good Good Low



GOOD


OUTCOME OF PERFORMANCE RATINGS OF THE BANK'S INVESTMENT OPERATIONS

1. POST- EVALUATION OUTCOME

1.4. GENERAL

This Appendix analyses performance ratings of evaluated investment operations. It seeks to draw conclusions and serves as a basis for some recommendations in the main text. Projects for evaluation are selected from all projects considered ready for evaluation.1 Performance evaluations of individual projects are generally only conducted once in their lifetime, normally with no subsequent re-validation. 1.5. EVALUATION COVERAGE IN 2009

During 2008, the Evaluation Department developed a new approach to the selection of projects for evaluation and calculation of the coverage ratio, including a new method of counting operations.2 The new approach is based on random sampling and is fully in line with the ECG's Good Practice Standards (GPS) on private sector evaluation.3 It is expected that the new approach will lead to a falling coverage ratio in the coming years. It was implemented for the first time in the selection of the 2009 work programme. Therefore, for the first time, the AEOR reports on evaluation coverage in terms of the new method of counting and selection. The Evaluation Policy of EBRD requires preparation of evaluations on a random, representative sample of sufficient size to establish, for a combined three-year rolling sample, success rates at the 95% confidence level, with sampling error not exceeding ±5 percentage points, for key performance indicators. In 2009, 115 individual investment operations and 6 large frameworks had reached early operating maturity and were ready for evaluation.4 The Evaluation Department selected a random sample of 61 of the 115 investment operations for evaluation. This random sample forms the basis of the Evaluation Department's conclusions about the Bank's performance. In 2009, 24 of the randomly sampled operations (21% of the total population of projects ready for evaluation) were evaluated through Operation Performance Evaluation Reviews (OPERs) and 5 (4%) through Special Studies. An additional 32 operations (28%) were covered with independent assessment reports by the Evaluation Department on bankers’ expanded monitoring reports (XMR assessments).5 This brought the year's coverage to 61 operations or 53% of ready operations.

1 Investment projects are considered ready for evaluation one and a half years after the last disbursement of loans and two years thereafter in cases of equity or combined equity/loans. At least one year of commercial operations, with at least one year of audited accounts, should normally have passed for all investment projects. 2 Described in the Board documents "Evaluation Coverage in EBRD" (CS/AU/08-36) and "Evaluation Department's Work Programme Final Report for 2009" (BDS09-007) 3 Reference is made to the Third Edition of the GPS on private sector evaluation (18 April 2006) and in particular to Section 2 on "Evaluation Timing, Population, Coverage and Sampling". The EBRD also adopted Best Practice Alternative 1 as described in Standard 2.2.1. In 2009, EvD selected additional operations in ETCs, in Russia and in the Financial Sector as "strategically targeted groups". 4 Originally, 119 individual investment operations were considered ready for evaluation, but some were dropped during the year as described further in the Evaluation Department Work Programme Completion Report for 2009 (BDS10-090) 5 An XMR assessment takes about two-three days work of EvD staff. It does not involve a field mission and is based on a desk-study. It includes: a) study of the XMR (a joint monitoring and self-evaluation report by bankers); b) review of project documents and various industry reports; c) interviews with operation teams, other EBRD staff and sometimes external parties; and d) independent validation of performance ratings and lessons. The performance ratings assigned to projects that are XMR assessed are aggregated in the overall performance rating of all evaluated projects as presented in this report. Lessons from XMR Assessments are included in EvD's Lessons Learned Database (LLD).


Chart 1.1 below shows the actual and projected coverage ratio using the new approach to counting and selecting projects for evaluation. It illustrates clearly the expected fall in coverage over the coming years.

Chart 1.1: Evaluation coverage for investment operations (actual to 2009 and projected)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1993 1995 1997 1999 2001 2003 2005 2007 2009 2011 2013

% o

f inv

estm

ent o

pera

tions

rea

dy fo

r ev

alua

tion

% of investment operations self-assessed by the operation team

% of investment operations covered by XMR Assessments

% of investment operations covered by OPER reports

1.6. SIZE AND REPRESENTATION OF THE SAMPLE OF EVALUATED PROJECTS

Selection of projects for evaluation is described in detail in Appendix 3 of the Evaluation Policy Review of 2004 (BDS04-24 (Rev 1). However, an Update to the Evaluation Policy of the EBRD (BDS10-024) was approved by the Board of Directors on 23 March 2010 and will be the guiding document for selection of projects in future. Appendix 3 of this report presents the new selection process in detail. For the exercise of performance evaluation in this AEOR for 2009, the total sample comprises 679 individually rated projects, evaluated in 1996-2009. These were selected from a total population of 934 standalone operations and small frameworks which became ready for evaluation during the period - a coverage ratio of 73%.6 The annual evaluation coverage was 100 per cent at the end of 1996 and above the 60 per cent target thereafter, as shown in Chart 1.1. The evaluated share of all 1,830 standalone operations signed by the bank since 1991 was lower at 37%, as many of the more recently approved operations had not yet reached the evaluation stage.7 Of the 679 projects in the sample, 284 were evaluated through OPERs, 19 through Special Studies and 376 through XMR assessments. A further 27 projects, constituting 100 per cent of projects ready for evaluation in 1993-1995, are omitted from the results because they were evaluated before a refined and consistent system of evaluation had been introduced. Section 10 of this Appendix presents an analysis of the country, sector representation in the sample as well as and the risk rating distribution of the evaluated

6 The evaluation coverage gap is compensated, in part, by EvD's review of all XMRs. In contrast to OPERs and XMR assessments, XMR reviews do not seek to validate self-evaluation ratings and no editing is made of the lessons. In contrast, the reviews seek to ascertain completeness and clarity in consultation with the teams and report the quality ratings given with EvD's sign-off. The independent OPER reports, XMR assessments and quality-control by XMR reviews, together cover 100 per cent of all operations ready for evaluation. 7 See Appendix 6 for more detailed data.


sample. The sample of projects selected from the groups of operations ready for evaluation continues to be a good representation of the Bank’s portfolio as a whole. 1.6.1. Identification of the population of projects ready for evaluation The sample of projects selected for evaluation each year is drawn from a population consisting of the investments expected to reach early operating maturity (as defined in the Good Practice Standards) during the year. Subject to certain exclusions, specified below, the population includes all disbursed (including partially cancelled) investments - whether still active or already closed (paid-off, sold or written off) - that have reached early operating maturity. The population also includes any investments already closed (i.e. those projects prepaid in the previous year), even if they never reached early operating maturity. Excluded from the population are: • dropped and cancelled investments where no disbursement has been made, • very small investments made under large frameworks (which are generally evaluated on a

programme basis through a Special Study) • certain follow-on operations, such as minor capital increases or investments undertaken to help

finance further expansion or cost overruns on projects previously financed by the EBRD, especially where such follow-on operations did not have separate objectives against which performance could be evaluated.

Projects that are not expected to reach early operating maturity during the year are excluded from the population and rolled forward for inclusion in the population in a future year when they will have reached early operating maturity. Investments are included in the population from which the sample for evaluation is drawn only once, i.e., only for the year in which they will have reached early operating maturity. According to the GPS, operations are deemed to have reached early operating maturity when (a) the project financed will have been substantially completed, (b) the project financed will have generated at least 18 months of operating revenues for the company and (c) the EBRD will have received at least one set of audited annual financial statements covering at least 12 months of operating revenues generated by the project. In practice, the Evaluation Departments prepare a list of projects which will be fully disbursed plus 18 months for loan operations, and 24 months for equity operations, and then discusses these with the Banking Department and others to identify whether they meet the other criteria required for early operating maturity. All projects selected for evaluation and inclusion in the evaluation database had reached early operating maturity, as defined in the GPS, by the time of their evaluation. Some additional projects were subject to Mid-Term Reviews, because they had not reached early operating maturity, but they were excluded from the database for the purposes of reporting on the EBRD's overall performance. 1.6.2. Selection of the sample of projects for evaluation Starting with the 2009 work programme, the selection of projects used to report on success rates for accountability purposes is entirely random. A random sample of the appropriate size is taken from full population for the year. The Chief Evaluator continues to select operations for in-depth evaluation (through OPERs) based on their lessons potential, but the results of these evaluations will not enter the evaluation database unless they also form part of the random sample. Projects forming part of the sample but not selected by the Chief Evaluator to be an OPER are evaluated through XMR Assessments.


Before 2009, EvD used a structured sampling methodology, and projects selected in this manner still form the majority of the total sample of evaluated projects. The Chief Evaluator would select a purposive sample of operations for evaluation through OPERs. The Evaluation Department would then conduct several random samples of the operations not selected as OPERs, and analyse the sector and country coverage and the risk distribution of the samples to find the one which, when combined with the OPERs already selected, provided the best match for EBRD's portfolio as a whole. Under both systems, operations for evaluation through OPERs were selected according to the following criteria: • Lessons learned potential of an operation: the expectation that the evaluation can generate rich

lessons; • Whether a project is high profile: these projects can have important political/transition

connotations or can be flagship operations in a country where the project has high demonstration effects;

• The Bank’s risk in a project, including environmental risks: this can be reputation risks for the Bank or risks due to the size of the investment;

• Whether an operation is under-performing: impaired operations tend to contribute considerably to the crop of lessons learned.

• Likelihood of replication of the operation: lessons from these projects help in enhancing the projects that the Bank is working on at the moment, or will work on in the future.

• Country and sector coverage: it is important to evaluate projects from as many sectors, Banking teams and countries as possible to represent a cross-section of the portfolio;

In this way EvD identifies the projects which have the greatest potential for learning from EBRD's experience. 1.6.3. Standard error of the sample: The new project selection procedure will follow the Update of the Evaluation Policy of the EBRD, whereby the Evaluation Department selects sufficient projects for evaluation to establish, for a combined three-year rolling sample, success rates at the 95% confidence level, with sampling error not exceeding ±5 percentage points. In the three years 2007-2009, there was a combined population of 342 individual standalone operations and investments under frameworks ready for evaluation, excluding investments under large frameworks. Of these, 196 were evaluated by EvD. Thus the overall coverage ratio was 57%. As the selection of projects in 2007-08 was not entirely random, it is necessary to split the population into two strata: • Stratum 1: 85 operations which were purposively sampled for evaluation through OPERs or

Special Studies under the old selection system (25% of 342). There is 100% evaluation coverage of this stratum.

• Stratum 2: The remaining 257 (75% of 342) not selected for evaluation through an OPER report. The random sample of 111 operations for evaluation constitutes 43% of stratum 2.

The random sample error for Stratum 2 is 7.02%, at the 95% confidence level. This exceeds the 5 per cent prescribed in the GPS. In selecting the random samples in 2007-08, EvD ran several samples and selected one which, combined with the projects in Stratum 1, gave the closest possible match to the regional, sector and risk distribution of all the projects signed by the Bank since 1991. In this way, EvD believes that it achieved a greater degree of representativity, and a greater statistical significance, than is suggested by the bare figures. Therefore, EvD believes that the success rates


reported in this appendix and in Chapter 1 can be attributed to the population of EBRD projects ready for evaluation. The new system of selective projects for evaluation will give a lower sample error. If the same number of evaluations had been selected randomly, i.e. 196 out of a population of 342, the sample error would be 4.58%, well within the limit set by the GPS. 1.6.4. Weighting the results: Where stratified sampling has been used, the GPS also require EvD to calculate the weighted average success rates, based on the weight of each stratum in the overall population. As described in section 1.3.3 above, the 342 projects evaluated in 2007-09 are divided into two strata: Stratum 1 consists of 85 operations (25% of the total) purposively selected for evaluation through an OPER in 2007-08, while Stratum 2 consists of the remaining 257 operations (75% of the total), of which a random sample of 111 were evaluated through XMR Assessments in 2007-08 or through OPERs or XMR Assessments in 2009. For weighting purposes, the 111 randomly sampled operations must be given a 75% weighting in the overall results. Table 1.1 below gives the weighted and unweighted outcomes for Overall Performance for 2007-09:

Table 1.1: Outcomes for Overall Performance of projects evaluated in 2007-09

EvD selection type Highly Successful

Successful Partly Successful

Unsuccessful Number of operations

Purposive - stratum 1 4% 53% 30% 13% 85 Random - stratum 2 5% 48% 40% 6% 111 Overall result - Unweighted 5% 50% 37% 8% 196 Overall result - Weighted 5% 49% 38% 8% 342

It can be seen that the overall result is almost identical in both cases. It should be noted that OPER reports show a greater proportion of Unsuccessful projects than XMR Assessments.

2. PERFORMANCE RATING OF EVALUATED PROJECTS

2.4. THE COMPOSITE OVERALL PERFORMANCE RATING OF A PROJECT

The overall performance rating of an evaluated operation builds on several underlying performance ratings, derived from the Bank's mandate. Transition impact is the overriding individual rating for all operations. Environmental performance and change are significant indicators for projects with high environmental risks. The following broad performance dimensions are addressed: a. Transition impact

- transition impact is defined as the effects of a Bank project on businesses, markets or institutions that contribute to the transformation from central planning to a well functioning market economy

b. The environment

- environmental and social performance measures how well the environmental objectives of the project (institutional, emissions control, regulatory compliance, social issues and public participation) were identified and have been met


- environmental change is measure as the difference between the environmental performance

before the project started and its performance at the time of evaluation c. Additionality

- the Bank's additionality in terms of whether the Bank provides financing that could not be mobilised on the same terms by markets and/or whether the Bank can influence the design and functioning of a project to secure transition impact

d. Sound banking principles

- project and company (financial) performance provide the sustainability element to allow transition impact to enfold beyond the project/company, and

- fulfilment of project objectives concerns the extent of verified and expected risk weighted fulfilment potential of the operation’s “process” and “project” objectives (“efficacy”) upon validation of their relevance

e. The Bank's investment performance

- the Bank’s investment performance measures the extent to which the gross contribution of a project is expected to be sufficient to cover its full average transaction cost and contribute during its life to the Bank’s net profit. Unlike the other dimensions, however, it does not represent any impact of the project “on the ground” in the country.

f. Bank handling

- Bank handling assesses the due diligence, structuring and monitoring of the project, as undertaken by all departments and units involved in the operation process, and the Bank as a whole. A judgement is made on the quality of the work and on how effectively the Bank carried out its work during the life of the project. Positive and negative lessons are generated. In case operations are evaluated that are handled by the Corporate Recovery Unit, Bank Handling will also take into account problem recognition, remedial action and recovery efforts.

In the past, multilateral development banks (MDBs) have had different ways of measuring overall performance and performance with respect to their mandates. However, the MDBs have been asked, by their shareholders, to harmonise their evaluation procedures and processes, to ensure their results are more comparable with the outcomes of other MDBs. Therefore, the evaluation departments of the MDBs, through the Evaluation Cooperation Group (ECG), have attempted to harmonise their rating systems so that some comparisons can be made. For the EBRD, this means that the Bank, apart from the presentation of performance evaluation based on all indicators, will also measure transition outcome. Transition outcome combines the ratings that measure “results on the ground” in the respective countries. The composite rating categories for the transition outcome rating are: transition impact; environmental performance and change; project and company financial performance; and fulfilment of project objectives. In the past, EvD has commented on the close relationship between this rating and the overall performance rating (presented in detail in Section 2.2 of this appendix). Starting from 2007, EvD has assigned a transition outcome rating to each of the projects evaluated. The results are shown in Table 1.1 below, where they are compared with the distribution of overall performance ratings. EvD will extend this comparison in future years to build up a time series as for other indicators.


Table 1.1: Transition Outcome, percentage distribution of assigned ratings


Unsuccessful Partly Successful

Sub-total

Successful Highly Successful

Sub- total

No. of evaluations

Transition Outcome 2007 7% 28% 35% 58% 7% 65% 54 2008 11% 37% 48% 45% 7% 52% 44 2009 16% 36% 52% 48% 0% 48% 61 2007-2008 9% 32% 41% 52% 7% 59% 98 2007-2009 12% 33% 46% 50% 4% 54% 159 Overall Performance 2007-2009 8% 37% 45% 50% 5% 55% 159

It can be seen that the overall performance and the transition outcome ratings, when compared, are highly similar. The fall in results between 2007 and 2009 mirrors a similar fall in the ratings for overall performance (see section 2.2 below). The indicators that are not concerned with results on the ground, i.e. the Bank’s additionality, bank handling and investment performance have a limited impact on the overall performance rating. 2.5. OVERALL PERFORMANCE RATINGS 1996-2009

Chart 2.1 presents the assigned overall performance ratings given to all EBRD investment projects evaluated since 1996. Projects evaluated before that date are omitted because EvD introduced a refined and consistent system of evaluation only in 1996. The background numbers behind this chart can be found in Annex 1 to this appendix, Table 1.

Chart 2.1: Overall performance, percentage distribution of assigned ratings


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1996(34)

1997(36)

1998(49)

1999(50)

2000(42)

2001(50)

2002(50)

2003(53)

2004(52)

2005(52)

2006(52)

2007(54)

2008(44)

2009(61)


% o

f eva

luat

ed p

roje

cts


Ratings in the Successful - Highly Successful bracket were achieved by 57 per cent of the operations evaluated in 1996-2009. Throughout the 1990s this share varied around the 50 per cent mark but showed no definite pattern. It rose sharply from 46 per cent in 2001 to 73 per cent in 2004 and has declined steadily since then. In 2009, 51 per cent of projects were rated Successful or Highly Successful. The proportion of projects rated Highly Successful in 2009


was 3 per cent, the lowest figure since 2001 when no evaluated projects achieved the highest rating. The proportion of projects rated Unsuccessful, at 8 per cent, remained lower than in most previous years. In 2009 as in 2008, a high proportion of projects (41%) were rated Partly Successful. In most cases these projects had moderate ratings on all indicators rather than an Excellent rating in one area and Unsatisfactory in another. An exception to this was a group of projects with Unsatisfactory ratings for Project or Company Financial Performance but Good or Excellent ratings for most other indicators. According to the Evaluation Policy, a project which is rated Satisfactory for all the major indicators would be rated Partly Successful overall, and many projects evaluated in 2008 had ratings that were close to this pattern. Across the whole period, nine per cent of the projects scored Highly Successful overall, while rather more were rated Unsuccessful (13 per cent). Projects with the highest Overall Performance rating scored well on Transition Impact and the other performance indicators, while over three quarters of Unsuccessful projects scored Unsatisfactory or Highly Unsatisfactory for Project and Company Financial Performance. This resulted in low sustainability and lost positive external factors in the sector and economy as a whole. A project must necessarily achieve financial sustainability in order achieve transition impact through linkages or positive demonstration effects. 2.6. OVERALL PERFORMANCE RATINGS BY COUNTRY GROUPS8

Chart 2.2 below shows that the highest overall performance ratings have been achieved in South-Eastern Europe (SEE) and Central Europe and the Baltics (CEB), where over 60 per cent of evaluated projects have been rated Successful or better). The corresponding figures for other regions are 55 per cent for Russia (RUS), 46 per cent for Eastern Europe and Caucasus (EEC) and 44 per cent for Central Asia (CA).

8 For an analysis of projects in Early Transition Countries (ETCs), please see section 10 below.


Chart 2.2: Overall performance ratings by country groups

(627 investment operations evaluated 1996-2009)9

4% 10% 3%14%

40%

51%

43%44%

50%

40%

24%38%

29%

30%

16% 14% 16% 15%6%

11%0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

CA (57) CEB (226) EEC (90) RUS (136) SEE (118)


% o

f eva

luat

ed p

roje

cts


Note: 52 regional projects omitted

The average success ratings in the various country groups reflect the more difficult environment the Bank faces in the countries towards the east and south of its region, compared with those in central and south-eastern Europe. This justifies the additional resources EBRD intends to direct towards these countries in the future, but also indicates the challenge facing the Bank as it withdraws from central Europe. The pattern in Chart 2.3 shows the change in Overall performance ratings over time for the main geographical areas. It can be seen that all the regions except Central Asia and Russia show an improvement in recent years over results in the period 2000-2004.

Chart 2.3: Development of overall performance ratings over time for projects evaluated 1996-2009:

presented by region

Central Asia

33%

50%36%

33%

35%46%

33%

10% 14%

5% 4%0%

20%

40%

60%

80%

100%

1996-1999 (9) 2000-2004 (20) 2005-2009 (28)Year of evaluation

% o

f eva

luat

ed p

roje

cts


Central Europe and Baltics

11% 9%

55%45% 55%

19%29%

26%

15% 16% 9%

10%0%

20%

40%

60%

80%

100%


% o

f eva

luat

ed p

roje

cts


9 CA: Kazakhstan, Kyrgyz Republic, Tajikistan, Turkmenistan, Uzbekistan CEB: Croatia, Czech Republic, Estonia, Hungary, Latvia, Lithuania, Poland, Slovak Republic, Slovenia EEC: Armenia, Azerbaijan, Belarus, Georgia, Moldova, Ukraine SEE: Albania, Bosnia & Herzegovina, Bulgaria, FYR Macedonia, Montenegro, Romania, Serbia


Eastern Europe and Caucasus

5%

42% 42% 45%

33% 34%43%

25% 21%8%

0%

20%

40%

60%

80%

100%

1996-1999 (12) 2000-2004 (38) 2005-2009 (40)

Year of evaluation

% o

f eva

luat

ed p

roje

cts


Russia

11% 11% 11%

22%

51% 48%

22% 34%

16%6%

30%

37%

0%

20%

40%

60%

80%

100%


% o

f eva

luat

ed p

roje

cts

Highly Successful Successful Partly Successful Unsuccessful South Eastern Europe

11%23%

48%50%

51%

23%

2%

4%

41% 30%

7% 9%

0%

20%

40%

60%

80%

100%


% o

f eva

luat

ed p

roje

cts

Highly Successful Successful Partly Successful Unsuccessful Note: 52 regional projects omitted

See chart 2.2 for list of countries in each region

Performance in EEC and SEE has improved consistently over the period of evaluation, though in the EEC region the improvement has been moderate. SEE is again the region with the highest overall performance ratings in recent years, with 74 per cent Successful or better in 2005-2009. In CEB there was a drop in performance in 2000-2004, which has since been recovered. In Central Asia and Russia, gains in the period 2000-2004 have dropped off somewhat in recent years. This is a noticeable deterioration compared with the results reported in the AEOR for 2009. That showed that in 2004-2008, 50 per cent of projects in Central Asia were rated Successful or better, compared with only 40 per cent for 2005-2009. In Russia the figures were 66 per cent and 59 per cent respectively. 2.7. OVERALL PERFORMANCE RATINGS BY INDUSTRY SECTORS

Chart 2.4 below shows the sectoral breakdown cumulatively for all projects evaluated since 1996. There is very little difference between the performance of all the sectors. The Corporate sector falls slightly behind the other sectors with 55 per cent of projects rated Successful or Highly Successful, but the difference between the highest and lowest performing sectors is only 6 per cent.


Chart 2.4: Overall performance ratings by sector groups

(679 evaluated investment operations, 1996-2009)10

10% 11% 7%

47% 44% 54% 57%

30% 31%29% 26%

13% 14% 10%

5%

13%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%




% o

f eva

luat

ed p

roje

cts


The time series data in chart 2.5 show that there is little difference between the sectors in recent years. In the AEOR for 2009, Financial Institutions projects in the period 2004-2008 had been rated substantially higher than projects in other sectors, but this is no longer the case with the move to the period 2005-2009. Energy and Infrastructure projects show slightly better results that Financial and Corporate projects, but the difference is not large.

Chart 2.5: Development of overall performance ratings over time for projects evaluated 1996-2008: presented by industry sector

Financial Institutions

11%

45% 46% 48%

25% 30%34%

19% 15% 8%

9%9%0%

20%

40%

60%

80%

100%

1996-1999 (53) 2000-2004 (93) 2005-2009 (85)

Year of evaluation

% o

f eva

luat

ed p

roje

cts


Corporate

42% 46% 43%

29%36%

22% 16% 7%

10% 13%9%

26%

0%

20%

40%

60%

80%

100%


% o

f eva

luat

ed p

roje

cts


10 Corporate = agribusiness, general industry, commercial services, property/tourism, and telecommunications Energy = power and energy, and natural resources Infrastructure = municipal/environment, and transport


Energy

55%52% 56%

30%32% 26%

15%

12% 7%

11%

0%

20%

40%

60%

80%

100%


% o

f eva

luat

ed p

roje

cts


Infrastructure

8%

63%55%

56%

21%

24%29%

8%

5%

11%21%

0%

20%

40%

60%

80%

100%

1996-1999 (19) 2000-2004 (33) 2005-2009 (52)

Year of evaluation

% o

f eva

luat

ed p

roje

cts

Highly Successful Successful Partly Successful Unsuccessful See chart 2.4 for list of industries in each sector

2.8. OVERALL PERFORMANCE RATINGS BY VOLUME

It has been observed over several years that the results are better when considered by volume; that is, larger projects are more successful than small ones. The AEOR for 2009 investigated this phenomenon in some depth and led to a further paper being produced during the year (attached at Appendix 9). This found that larger projects were rated more highly than smaller ones for Overall Performance, Transition Impact and Financial Performance. It also noted that smaller projects were concentrated in the regions on the lower end of transition (CA and EEC) and that business may be more difficult in these countries. In 2009, for the first time in many years, larger projects were not observed to perform better than smaller ones. In 2009, measured by project volume, 49 per cent of projects were rated Successful or better, compared to 51 per cent when measured by the number of evaluations. This appears to be principally affected by Achievement of Objectives ratings, since both Transition Impact and Project/Company Financial Performance still show slightly better performance when it is measured by volume rather than number of operations.


Chart 2.6: Overall performance, percentage distribution of assigned ratings

in terms of volume of projects evaluated

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

Year of evaluation

% o

f eva

luat

ed p

roje

cts


Chart 2.6 above shows the sudden drop in Overall Performance ratings in 2009 in terms of volume of projects evaluated. The variation is at the boundary between Successful and Partly Successful projects. It has been observed in the past that when results are measured in terms of volume rather than number of projects evaluated, there is potential for much greater fluctuation from one year to the next. This is because a few large projects can outweigh the effects of many small ones. Therefore the existence of six large (>EUR 50 million) projects rated Partly Successful in 2009 has had a disproportionate effect on the outcome. These consisted of two road projects, two financial projects and two corporate projects. The Evaluation Department will continue to monitor the relative performance of large and small projects in future years, and expects to find that the result in 2009 was an anomaly as previously seen in 2002 (see chart 2.6 above).

3. THE TRANSITION IMPACT (TI) RATING

3.4. METHODOLOGY

The case presentations in Appendix 7 illustrate the evaluation methodology used after project signing (ex-post). This uses the same framework and indicators as the ex-ante (before project signing) methodology, applied by the Bank during the approval stage of new projects. It should be noted that this methodology includes short-term verified impact, the assessed potential for further transition impact, as well as an assigned risk for the realisation of this potential. From 2000 a six-grade scale was applied for all post-evaluated operations, similar to the scale adopted for ex-ante assessment of TI-potential and attendant risks by OCE. In 2006, EvD revisited projects evaluated 1996-2005 and re-rated them according to the current rating system for transition impact and other indicators. An analysis of the projects which have been rated both ex-ante by OCE and ex-post by EvD can be found in chapter 2. 3.5. TRANSITION IMPACT RATINGS 1996-2009

Transition impact ratings carry a high weight in the overall performance ratings of projects. Chart 3.1 shows the distribution of the ex-post transition impact ratings by EvD since 1996. The detailed figures relating to this chart can be found in Annex 1 to this appendix, Table 2.


The proportion of projects rated Excellent or Good remained high in 2009, at 59% - slightly better than in the previous two years. Similarly, the proportion of projects rated Negative (0%) or Unsatisfactory (3%) was the lowest it has ever been. However, in 2009 a relatively large proportion of the projects was rated Marginal and fewer projects were rated Satisfactory than in recent years. As a result, the proportion of the projects rated Excellent to Satisfactory fell from 87% in 2008 to 75% in 2009. Over the entire period 1996-2009, the share of projects with Good to Excellent ratings is 55 per cent, while 79 per cent of the projects were rated Satisfactory or higher. Eight per cent of the projects during the same period were rated Unsatisfactory or Negative.

Chart 3.1: Transition impact, percentage distribution of assigned ratings in terms of number of projects evaluated

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1996(34)

1997(36)

1998(49)

1999(50)

2000(42)

2001(50)

2002(50)

2003(53)

2004(52)

2005(52)

2006(52)

2007(54)

2008(44)

2009(61)


% o

f eva

luat

ed p

roje

cts


3.6. TRANSITION IMPACT RATINGS BY VOLUME

As for the Overall Performance rating, the proportion of evaluated projects achieving top transition impact ratings has generally been higher when measured by volume. Cumulatively over the period 1996-2009, 63 per cent of projects by volume have been rated Excellent or Good for transition impact, while a further 23 per cent have been rated Satisfactory. Ratings for Transition Impact, unlike those for Overall Performance, maintain this difference in 2009, although the difference is smaller than in previous years. Of the projects evaluated in 2009, 60 per cent by volume were rated Good or Excellent (compared with 59% by number) and a further 20 per cent were rated Satisfactory (16% by number).


Chart 3.2: Transition impact, percentage distribution of assigned ratings

in terms of volume of projects evaluated

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

Year of evaluation

% o

f eva

luat

ed p

roje

cts


It can be seen from Chart 3.2 above that the proportion of projects rated Excellent to Good remained the same as in 2008, while the proportion rated Satisfactory was lower and more projects were rated Marginal. This is similar to the results when counting the number of projects evaluated. 3.7. TRANSITION IMPACT RATINGS BY COUNTRY GROUPS

Chart 3.3 presents the TI rating distribution by country groups of 627 projects evaluated in 1996-2009. Central Asia continues to show markedly lower ratings than the other groups, with only 38 per cent of projects rated Good or Excellent for transition impact. This is a lower result than reported in the AEOR for 2009 (41%), but the proportion of projects rated Satisfactory for this region has risen. In total 35% of projects in Central Asia have been rated Marginal or lower, which is a rather high figure but better than in the past. SEE is again the best performing region with 67 per cent of projects rated Good or Excellent. CEB, EEC and Russia fall between these two extremes, with results in EEC slightly below those in the other two regions.


Chart 3.3: TI ratings of 627 post-evaluated investment operations in 1996-2009 by country groups

5% 10% 3%18%

33%

43%47%

46%

49%

26%

26%23%

21%

22%26%12% 19% 13%

9%6% 9%

10%

5%2%

6%2%4% 3% 2%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

CA (57) CEB (226) EEC (90) RUS (136) SEE (118)


% o

f eva

luat

ed p

roje

cts


Note: 52 regional projects omitted See chart 2.2 for list of countries in each region

Time series analysis indicates that Transition Impact ratings have improved in recent years in most regions. In SEE, the best performing region, ratings at the Excellent to Good level have levelled off, though there is a continued increase in the proportion of projects rated Excellent or Satisfactory. In EEC the proportion of projects rated Excellent to Good has fallen in recent years and the proportion rated Excellent to Satisfactory is level. In the most recent period, 2005-2009, SEE was the best performing region with 72% of projects rated Excellent or Good and a further 25% rated Satisfactory. The worst performing region over the same period was Central Asia, with 43% of projects rated Excellent or Good and a further 25% rated Satisfactory.

Chart 3.4: Development of TI ratings over time for projects evaluated 1996-2009: presented by region

Central Asia

22%35% 36%

44%20%

25%

22% 35% 21%

5%

7%5%

7%4%11%

0%

20%

40%

60%

80%

100%

1996-1999 (9) 2000-2004 (20) 2005-2009 (28)

Year of evaluation

% o

f eva

luat

ed p

roje

cts



13% 13%

44%44%

42%

22%28%

13% 11%9%

5%

28%

11%4%4% 2%4%3%

0%

20%

40%

60%

80%

100%

1996-1999 (91) 2000-2004 (82) 2005-2009 (53)

Year of evaluation

% o

f eva

luat

ed p

roje

cts




53% 43%

33%18% 25%

13%25%

11%

5%

42%

17%

3%8% 0%

0%

20%

40%

60%

80%

100%

1996-1999 (12) 2000-2004 (38) 2005-2009 (40)

Year of evaluation

% o

f eva

luat

ed p

roje

cts


Russia

13% 11%30%

47% 52%

16%25%

11%

11%26%

9%

22%

19%

2%4%4%

0%

20%

40%

60%

80%

100%


% o

f eva

luat

ed p

roje

cts


South Eastern Europe

37%66%

40%

23%

7%11%32%

14%

33%

2%11%19%2% 2%

0%

20%

40%

60%

80%

100%


% o

f eva

luat

ed p

roje

cts



3.8. TRANSITION IMPACT RATINGS BY SECTORS

The sectoral breakdown for this indicator shows very little difference in performance between the four sectors. The Infrastructure sector has performed slightly less well than the other three sectors over the full period, and its results, in terms of projects rated Good or Excellent have fallen further in the most recent period. The proportion of projects rated Satisfactory has increased substantially during this period. The other three sectors have all seen improvements at the Excellent to Good level between the periods 2000-2004 and 2005-2009, although Financial Institutions has fallen slightly at the Excellent to Satisfactory level. In the Financial Institutions, Corporate and Energy sectors, transition impact ratings have improved between the periods 2000-2003 and 2004-2008. These results are shown in charts 3.5 and 3.6 below.


Chart 3.5: TI ratings 1996-2009 by sector groups (679 investment operations)

46% 45%46%

43%

22% 23% 26%29%

14% 12%6%

8%13%9% 4%

15% 16%

1% 5%7% 3%3%1%2%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%




% o

f eva

luat

ed p

roje

cts


See chart 2.6 for description of each sector

Chart 3.6: Development of TI ratings over time for projects evaluated 1996-2009:

presented by sector (679 investment operations)


9%

36% 54%45%

28%23% 18%

19% 10% 15%9%

3%15%

4% 6% 1%2%4%

0%

20%

40%

60%

80%

100%


% o

f eva

luat

ed p

roje

cts


Corporate

14% 13%

34% 47%52%

20%23%

13%8%

10%

26%

16%

3%8%9% 1%2%1%

0%

20%

40%

60%

80%

100%


% o

f eva

luat

ed p

roje

cts


Energy

45%48%

44%

30%24% 26%

15%

15%8%

11%20%4%10%

0%

20%

40%

60%

80%

100%


% o

f eva

luat

ed p

roje

cts


Infrastructure

8%

58%48% 35%

21%18% 38%

17%11%

21%5%6%

2%6%5%

0%

20%

40%

60%

80%

100%

1996-1999 (19) 2000-2004 (33) 2005-2009 (52)

Year of evaluation

% o

f eva

luat

ed p

roje

cts



4. ADDITIONALITY

4.4. METHODOLOGY

Evaluation seeks to assess the Bank's additionality in two ways. The foremost is an assessment of whether the EBRD financing was decisive for the realisation of the project, i.e. that it could not have been carried out for lack of alternative financing from markets. Whether the Bank was additional in terms of influencing the design and functioning of the project is a second consideration: the Bank may for example have requested an equity participation and board representation to improve on corporate governance standards or have conditioned its participation to compliance with higher environmental standards than would have been achieved with market financing only. The weight given to the two aspects of additionality is given in table 4.1 below, taken from Appendix 1 of the Evaluation Policy of the EBRD. The table shows the criteria a project must meet to be rated highly for additionality.

Table 4.1: performance rating benchmarks for additionality

RATING OF ADDITIONALITY

Ratings BENCHMARKS

Verified in all respects

No other financial institutions are willing to provide financing at the same or better condition than the Bank. The terms and conditions are not attractive to other banks and the country risk is still high. The client accepts tough conditionality to secure transition impact.

Verified at large Some competition with market financiers, but the Bank's terms and conditions, although more demanding than competition’s, prevail since sponsors/clients or co-financiers appreciate the Bank's political comfort. In such cases, specific project design and structuring may also be significant for enhanced transition impact. The Bank may also have contributed specific country- or sector knowledge or helped enhance corporate governance standards. Repeat financing to a second phase of a project, may fall into this category.

Verified only in part Competition from commercial financiers is significant and terms and conditions are almost identical, but the Bank's participation (e.g. in a bond issue) may have helped an earlier implementation of the project than would have otherwise been possible. No significant features are added to design and functioning to enhance transition and/or catalyse other financing.

Not verified Competition fully established for financing and the Bank's terms and conditions fail to provide for any material transition impact enhancement and pricing premium to account for the availability of the Bank’s Preferred Creditor Status.

4.5. ADDITIONALITY RATINGS 1996-2009

Of 679 operations evaluated in 1996-2009, 59 per cent had additionality Verified in All Respects, while 30 per cent had additionality Verified at Large. This left 11 per cent of projects with additionality Verified in Part or Not Verified (Table 4.2). Only 14 projects were rated in the latter group between 1996 and 2009, and none since 2007. A number of operations with low additionality were follow-on projects or otherwise linked to other facilities with the same client. Nevertheless, 89 per cent of projects with additionality verified fully or at large supports the EBRD’s additionality requirement under the Bank’s mandate and confirms that there was no crowding out of market financing.


Table 4.2: Additionality, percentage distribution of assigned ratings


Ratings 1996-97

1996-98

1996-99

1996-2000

1996-2001

1996-2002

1996-2003

1996-2004

1996-2005

1996-2006

1996-2007

1996-2008

1996-2009

Verified in All Respects 76% 78% 70% 72% 67% 64% 63% 63% 61% 62% 62% 62% 59% Verified at Large 10% 10% 17% 17% 21% 23% 24% 24% 26% 26% 27% 27% 30% Subtotal 86% 88% 87% 89% 88% 87% 87% 87% 87% 88% 89% 89% 89% Verified in Part 14% 10% 11% 9% 9% 10% 10% 10% 10% 10% 9% 9% 9% Not Verified 0% 2% 2% 2% 3% 3% 3% 3% 3% 2% 2% 2% 2% Subtotal 14% 12% 13% 11% 12% 13% 13% 13% 13% 12% 11% 11% 11% Total (No. of projects) 70 119 169 211 261 311 364 416 468 520 574 618 679

Regarding annual variations of additionality, it is striking that the only significant variation in recent years has been on the boundary between Verified in All Respects and Verified at Large. In 2009 the proportion of projects Verified in All Respects fell dramatically to 31%, its lowest-ever figure. It is difficult to account for this fall, or even to say whether it is anything more than a one-off anomaly. There have been previous years (1999, 2001-20002, 2005) when the proportion of projects Verified in All Respects has been far below the average and then recovered quickly. Rather than try to account in detail for the results of a single year, the Evaluation Department draws the attention of the Board and Management to this result and will watch this indicator closely over the next few years.

Chart 4.1: Additionality, percentage distribution of assigned ratings (679 investment operations evaluated 1996-2009)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1996(34)

1997(36)

1998(49)

1999(50)

2000(42)

2001(50)

2002(50)

2003(53)

2004(52)

2005(52)

2006(52)

2007(54)

2008(44)

2009(61)


% o

f eva

luat

ed p

roje

cts


4.6. ADDITIONALITY RATINGS BY YEAR OF BOARD APPROVAL

Additionality differs from other indicators in that it relates to conditions prevailing at the time of project approval, rather than performance since that date. Therefore it is worthwhile looking at the rating of Additionality by year of Board approval as well as by year of evaluation. Chart 4.2 below shows this.


Chart 4.2: Additionality, percentage distribution of assigned ratings by year of approval

(679 investment operations approved 1991-2007 and evaluated 1996-2009)

0%10%20%30%40%50%60%70%80%90%

100%

1991(4)

1992(23)

1993(43)

1994(52)

1995(58)

1996(58)

1997(70)

1998(34)

1999(57)

2000(53)

2001(41)

2002(46)

2003(44)

2004(26)

2005(40)

2006(22)

2007(8)


% o

f eva

luat

ed p

roje

cts


Only a small number of the evaluated projects were approved in 1991 or in 2007, so the results for those two years should not be considered very reliable. The years 1992-2006 all saw the approval by the Board of more than 20 operations in the evaluation database, so greater weight can be given to those figures. There is a clear decline time in the proportion of projects rated Verified in All Respects for Additionality. Following the Russian financial crisis in 1998, the projects approved in 1998 and 1999 showed relatively high Additionality, as might be expected. Since then, with some variation year-on-year, the proportion of projects rated Verified in All Respects has tended to decline, to around 40% in 2005-07. Of the projects evaluated in 2009, around three-quarters were approved in 2005-07. By chance, projects evaluated in 2008 and 2007 were not only approved earlier, but also over a wider range of years: 75% of projects evaluated in 2007 were approved in 2002-05, while 75% of projects evaluated in 2008 were approved 2001-06. Each year there are a few older projects which have taken longer than normal to become ready for evaluation. Therefore the sharp drop in ratings for Additionality in 2009 simply reflects a process that has been going on for some time and only now come sharply into focus. Chart 4.3 below shows the results for each region separately. As the annual number of operations Board approved in each region is small and fluctuating, a rolling three-year figure has been used. The numbers are still rather small, particularly in Central Asia. With that proviso, this region and SEE most clearly show Additionality falling year on year. The effect is not seen in CEB and barely seen in Russia, which for some time have been the two regions with the lowest Additionality ratings. EEC shows a number of projects rated Verified in Part.


Chart 4.3: Additionality of evaluated projects by date of Board approval: three-year rolling sample

Central Asia

0%10%20%30%40%50%60%70%80%90%

100%

2000-2002 (9)

2001-2003(10)

2002-2004 (9)

2003-2005(12)

2004-2006 (8)

2005-2007 (7)


% o

f eva

luat

ed p

roje

cts

Verified in All Respects Verified at LargeVerified in Part Not Verified

CEB

0%10%20%30%40%50%60%70%80%90%

100%

2000-2002 (44)

2001-2003 (34)

2002-2004 (24)

2003-2005 (16)

2004-2006 (13)


% o

f eva

luat

ed p

roje

cts


EEC

0%10%20%30%40%50%60%70%80%90%

100%

2000-2002(15)

2001-2003(12)

2002-2004(12)

2003-2005(19)

2004-2006(16)

2005-2007(15)


% o

f eva

luat

ed p

roje

cts


Russia

0%10%20%30%40%50%60%70%80%90%

100%

2000-2002(30)

2001-2003(30)

2002-2004(34)

2003-2005(35)

2004-2006(35)

2005-2007(27)


% o

f eva

luat

ed p

roje

cts


SEE

0%10%20%30%40%50%60%70%80%90%

100%

2000-2002(28)

2001-2003(30)

2002-2004(25)

2003-2005(20)

2004-2006(15)

2005-2007(12)


% o

f eva

luat

ed p

roje

cts


Intuitively, the Additionality of the Bank's projects might be expected to be inversely related to two external factors: the economic environment and the total volume of projects approved by the Bank. The reasoning behind the presumed link to the economy is fairly clear: in a difficult economic environment, EBRD's finance is in greater demand, and its conditions more acceptable. Tables 4.2 and 4.3 show Additionality falling during the "boom" years from around 2003 onwards. The argument for a relationship with the volume of approved projects is that if the Bank pursues larger volumes, it may be tempted to relax its requirement for


Additionality in its projects. Chart 4.4 makes a comparison of the proportion of evaluation projects rated Verified in All Respects with the total annual volume of Board-approved projects of the Bank.11

Chart 4.4: Additionality, percentage of projects Verified in All Respects by year of Board approval (679 investment operations approved 1991-2007 and evaluated 1996-2009)

and total annual volume of Board approved projects

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007


% o

f eva

luat

ed p

roje

cts

0

1,000

2,000

3,000

4,000

5,000

6,000

EU

R m

illio

n

% of projects Verified in All Respects (left-hand axis) EBRD's annual Board approved volume (right-hand axis)

Although the inverse relationship is not rigid, there are a number of years where Additionality and volume are clearly moving in opposite directions: 1997-2001 and 2004 onwards. During the period from 1997, only 2002-03 do not show an inverse relationship between these two indicators. The correlation between the two series shown in Chart 4.3 is -0.81 for the full period 1995-2007 for which the figures are available. This is a strong negative correlation. It should not be forgotten, of course, that the proportion of projects rated at least Verified at Large has remained high throughout the period. 4.7. ADDITIONALITY RATINGS BY REGION AND SECTOR

Chart 4.5 shows additionality ratings, by region and over time. All the regions show a falling proportion of projects with Additionality Verified in All Respects across the three periods. Russia and SEE show a rising level of projects Verified at Large, while CA also has a very high proportion of projects achieving this level. Particularly in recent years, most of the projects rated lower than this have been in CEE (17%) of EEC (15%). Russia in particular has improved on this indicator; in the 1990s it was the region with the highest proportion of projects rated Not Verified or Verified in Part.

11 These data are available from 1995 onwards. A comparison using Annual Business Volume and signing date of evaluated projects does not show a strong relationship in the period to 2003, but gives the same clear result for the years from 2004 onwards.


Chart 4.5: Development of additionality ratings over time for projects evaluated 1996-2009:

presented by region12

Central Asia

89%64%

11% 20%29%

80%

4% 4%

0%

20%

40%

60%

80%

100%

1996-1999 (9) 2000-2004 (20) 2005-2009 (28)

Year of evaluation

% o

f eva

luat

ed p

roje

cts



66%42%

21%

35% 42%

45%

13%15%12% 1% 5% 4%

0%

20%

40%

60%

80%

100%

1996-1999 (91) 2000-2004 (82) 2005-2009 (53)

Year of evaluation

% o

f eva

luat

ed p

roje

cts



100%

68%

18%18%

74%

5% 15%3%

0%

20%

40%

60%

80%

100%

1996-1999 (12) 2000-2004 (38) 2005-2009 (40)

Year of evaluation

% o

f eva

luat

ed p

roje

cts


Russia

41%

31% 53%

15%16%

6%

51%63%

11%

11% 2%

0%

20%

40%

60%

80%

100%

1996-1999 (27) 2000-2004 (45) 2005-2009 (64)

Year of evaluation

% o

f eva

luat

ed p

roje

cts


South Eastern Europe

74%62%70%

38%15% 18%

0%9%11% 2%

0%

20%

40%

60%

80%

100%

1996-1999 (27) 2000-2004 (44) 2005-2009 (47)

Year of evaluation

% o

f eva

luat

ed p

roje

cts


Note: 52 regional projects omitted

See chart 2.2 for list of countries in each region Chart 4.6 indicates additionality ratings, by sector and over the same periods as above. The same pattern is seen of a falling proportion of projects with Additionality Verified in All Respects, and the Corporate sector has the lowest result for this in the most recent period. Energy sector projects are all Verified at Large or better, but the other sectors do not show much variation at this level. 12 The unexpectedly high number of projects Verified in Part in the EEC region is caused by a series of operations with a single client, which were not rated as fully additional overall.


Chart 4.6: Development of additionality ratings over time for projects evaluated 1996-2009: presented by region


60% 57% 52%

23% 31% 38%

9%10%8% 9% 1%2%

0%

20%

40%

60%

80%

100%

1996-1999 (53) 2000-2004 (93) 2005-2009 (85)

Year of evaluation

% o

f eva

luat

ed p

roje

cts


Corporate

66%53%

42%

31% 49%

14% 11%

19%

7%4% 1%

0%

20%

40%

60%

80%

100%

1996-1999 (77) 2000-2004 (96) 2005-2009 (99)

Year of evaluation

% o

f eva

luat

ed p

roje

cts


Energy

100%

72% 67%

20% 33%

8%

0%

20%

40%

60%

80%

100%

1996-1999 (20) 2000-2004 (25) 2005-2009 (27)

Year of evaluation

% o

f eva

luat

ed p

roje

cts


Infrastructure

65%

21% 25%

11% 12%

64%84%

5%8%3% 2%

0%

20%

40%

60%

80%

100%

1996-1999 (19) 2000-2004 (33) 2005-2009 (52)

Year of evaluation

% o

f eva

luat

ed p

roje

cts


5. COMPANY AND PROJECT FINANCIAL PERFORMANCE

5.4. COMPANY AND PROJECT FINANCIAL PERFORMANCE RATINGS 1996-2009

The Company and Project Financial Performance ratings reflect whether the Bank financed “sound and economically viable operations”. Sustainable and financially viable projects are a key condition for sustained transition impact. The company performance ratings are based on the profitability, debt-service performance, financial status and prospects of borrowers and investee entities. Project Financial Performance is also assessed using indicators, such as financial internal rates of return (FIRR), which reflect a company’s success and financial strength.13 The financial performance ratings of 668 evaluated operations by the end of 200914 are presented in Charts 5.1 and 5.2:

13 The key financial and economic performance indicators are addressed in each of the respective OPER reports; the macro-economic viability is captured in some types of projects in the economic internal rate of return, EIRR. 14 For this indicator and those following, the number of evaluated projects is reduced by 11 operations. Seven were evaluated through Special Studies and were formally rated only for Overall Performance, Transition Impact and Additionality. Two evaluations related to broad client relationships for which a financial performance rating was not applicable, and two more were not rated for company and project financial performance because the client was not a commercial undertaking in its own right generating revenue.


Chart 5.1: Company performance, percentage distribution of assigned ratings (668 investment operations evaluated 1996-2009)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1996(34)

1997(36)

1998(49)

1999(50)

2000(42)

2001(49)

2002(49)

2003(48)

2004(52)

2005(52)

2006(52)

2007(53)

2008(41)

2009(61)


% o

f eva

luat

ed p

roje

cts


Table 5.1: Company performance, percentage distribution of assigned ratings


1996-2001

1996-2002

1996-2003

1996-2004

1996-2005

1996-2006

1996-2007

1996-2008

1996-2009

Excellent 10% 12% 13% 15% 15% 16% 15% 14% 13% Good 20% 20% 22% 24% 26% 28% 30% 30% 31% Satisfactory 23% 24% 23% 22% 22% 21% 21% 22% 22% Subtotal 53% 56% 58% 61% 63% 65% 66% 66% 66% Marginal 27% 26% 24% 22% 21% 20% 19% 19% 19% Unsatisfactory 11% 9% 10% 10% 9% 9% 8% 9% 9% Highly Unsatisfactory 9% 9% 8% 7% 7% 6% 7% 6% 6% Subtotal 47% 44% 42% 39% 37% 35% 34% 34% 34% Total (No. of projects) 260 309 357 409 461 513 566 607 668

Chart 5.2: Project performance, percentage distribution of assigned ratings


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1996(34)

1997(36)

1998(49)

1999(50)

2000(42)

2001(49)

2002(49)

2003(48)

2004(52)

2005(52)

2006(52)

2007(51)

2008(41)

2009(61)


% o

f eva

luat

ed p

roje

cts



Table 5.2: Project performance, percentage distribution of assigned ratings (668 investment operations evaluated 1996-2009)

1996-2001

1996-2002

1996-2003

1996-2004

1996-2005

1996-2006

1996-2007

1996-2008

1996-2009

Excellent 11% 12% 13% 14% 15% 16% 15% 15% 14% Good 18% 19% 19% 22% 23% 24% 26% 26% 29% Satisfactory 26% 27% 26% 26% 26% 25% 25% 25% 24% Subtotal 55% 58% 58% 62% 64% 65% 66% 66% 67% Marginal 25% 23% 23% 20% 20% 19% 19% 19% 19% Unsatisfactory 10% 9% 10% 10% 9% 9% 8% 8% 8% Highly Unsatisfactory 10% 10% 9% 8% 7% 7% 7% 7% 6% Subtotal 45% 42% 42% 38% 36% 35% 34% 34% 33% Total (No. of projects) 260 309 357 409 461 513 566 607 668

The close correlation between Company and Project Financial Performance reflects the fact that these ratings may be identical (as in greenfield investments) or closely interrelated. The above tables show that results tended to improve after a low period in 1998-2001, although this improvement levelled off after 2004-05 and then fell noticeably in 2008-09. Company Financial Performance in particular saw a substantial fall in 2008. It is not at all surprising to see the current exceptional economic conditions reflected in the outcomes for these two indicators in particular. As discussed in section 9 of this appendix, the Bank's Investment Performance has also declined in the last year. Project Financial Performance has fallen less than Company Financial Performance, which indicates that the specific investment projects financed by the Bank are in many cases doing better than the client company as a whole. 5.5. COMPANY AND PROJECT FINANCIAL PERFORMANCE RATINGS BY COUNTRY GROUPS

Charts 6.3 and 6.4 show the geographical breakdown of evaluated projects. In terms of projects rated Satisfactory or above, there are only minor differences between the different regions, with all regions achieving more than 60% of projects in these categories but with CA and EEC slightly below the other regions. There is a greater difference at the level of projects rated Good or better, and the differences are also more pronounced on company financial performance than project performance. This may indicate that while the financial performance of a company is affected by the local economic environment, a specific investment project can be designed to overcome (at least partly) such regional variations. CEB and Russia have the highest proportion of projects rated Good or Excellent, while the other regions have a higher proportion of projects rated Satisfactory. The proportion of projects rated less than Marginal varies between 10 and 18 per cent, the highest proportion being in Russia. It should also be noted that this indicator, like many others, is rated with reference to the projections made at appraisal, so that an Excellent or Good rating means that the project has outperformed projections. This helps to explain why more advanced transition regions do not always outperform other regions on these indicators: the projections may have been set higher at appraisal.


Chart 5.3 Company performance ratings by country groups


9%18%

6%14% 14%

26%

32%

27%

34%25%

23%

18%

29%

17% 29%

30%16%

16%20%

22%

7% 6% 19% 11%8%

3%3%9%5% 5%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

CA (57) CEB (226) EEC (86) RUS (133) SEE (116)


% o

f eva

luat

ed p

roje

cts



Chart 5.4 Project performance ratings by country groups


11% 16%7%

14% 18%

25%30%

27%

30% 25%

28%23%

27%21% 29%

21% 17%17%

19%19%

11% 5% 15%10%

6%3%5%7%

9%5%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

CA (57) CEB (225) EEC (86) RUS (132) SEE (116)


% o

f eva

luat

ed p

roje

cts



5.6. COMPANY AND PROJECT FINANCIAL PERFORMANCE RATINGS BY SECTOR GROUPS

As shown in chart 5.5, the Energy sector has the greatest number of projects rated Excellent or Good. The other three sectors do not differ much, though the Corporate sector is slightly below Financial Institutions or Infrastructure. At the level of Satisfactory or better ratings, the pattern is similar but more pronounced. The Corporate sector has a large number of projects with relatively extreme ratings, either Excellent (17%) or less than Marginal (17%).


Chart 5.5 Project performance ratings by sector groups


11% 17% 21%7%

32% 23%29%

36%

23%21%

29%31%

18%22%

14%15%

10% 9%4% 6%

3%8%5% 5%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%




% o

f eva

luat

ed p

roje

cts


6. FULFILMENT OF PROJECT OBJECTIVES

Fulfilment and relevance of project objectives is assessed against the objectives submitted at project approval. Chart 6.1 presents ratings achieved by evaluated projects for fulfilment of objectives. Ratings for this indicator have held up fairly well in 2009. The proportion of projects rated Satisfactory or better has continued to rise reached its highest level since 1996. The proportion rated Good or Excellent is rather lower than in 2004-05, but has not deteriorated in 2009. This is a welcome result in the current climate.

Chart 6.1: Fulfilment of objectives, percentage distribution of assigned ratings (672 investment operations evaluated 1996-2009)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1996(34)

1997(36)

1998(49)

1999(50)

2000(42)

2001(49)

2002(49)

2003(48)

2004(52)

2005(52)

2006(52)

2007(54)

2008(44)

2009(61)


% o

f eva

luat

ed p

roje

cts



Table 6.1: Fulfilment of objectives, percentage distribution of assigned ratings


1996-2001

1996-2002

1996-2003

1996-2004

1996-2005

1996-2006

1996-2007

1996-2008

1996-2009

Excellent 21% 20% 19% 19% 20% 20% 19% 18% 17% Good 26% 27% 27% 30% 31% 32% 33% 34% 35% Satisfactory 22% 22% 23% 22% 21% 21% 22% 23% 24% Subtotal 68% 68% 69% 71% 72% 73% 74% 75% 76% Marginal 17% 17% 17% 17% 17% 17% 16% 15% 15% Unsatisfactory 7% 6% 6% 6% 5% 5% 5% 5% 5% Highly Unsatisfactory

8% 8% 7% 7% 6% 5% 5% 5% 4%

Subtotal 32% 32% 31% 29% 28% 27% 26% 25% 24% Total (no. of projects)

260 309 357 409 462 513 567 611 672

7. THE ENVIRONMENT

7.1. THE ENVIRONMENTAL AND SOCIAL REQUIREMENT

Projects are designed and conditioned to fulfil all aspects of the Bank's mandate, including the environmental and social policy of the Bank at the time of appraisal. Environmental and social ratings form part of the overall performance rating. Environmental and social evaluation concerns the physical environment, social environment, as well as occupational health and safety, and issues such as public consultation. The analysis in this Appendix refers to 664 evaluated projects during 1996-200915. 7.2. ENVIRONMENTAL AND SOCIAL RATING SYSTEM

The series from 1996-2009 covers two environmental dimensions: The first dimension concerns environmental and social performance16 of the sponsor, e.g. the preparation and implementation of environmental action plans; compliance with contractual environmental conditions and national and EU statutory regulations etc. The second dimension is the extent of environmental change (positive or negative) brought about by the evaluated operation. Under Bank Handling, EvD also considers environmental bank handling with respect to categorization, environmental due diligence, audits and project monitoring. Starting from 2008, EvD has introduced a new indicator of Environmental Impact on a pilot basis. This combines the two existing indicators in a ratio that is determined by the potential for environmental change at entry. A full description is given in Appendix 5 of the AEOR for 2008 (BDS08-113(F)). 7.3. EVOLUTION OF ENVIRONMENTAL AND SOCIAL RATINGS

The Charts and Tables 7.1 and 7.2 present ratings of environmental performance and of the extent of environmental change as assigned to 664 evaluated projects in 1996-2009.

15 Two projects evaluated through Special Studies were not rated for Environmental Performance or Extent of Environmental Change. A further 13 projects were rated “not applicable” for Environmental Performance and 14 for Environmental Change because they were judged to have no environmental implications. 16 From 2003 onwards, the social elements were incorporated in the new environmental policy. From that time onwards the rating category in fact covers environmental as well as social performance.


Chart 7.1: Environmental and social performance, percentage distribution of assigned ratings


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1996(34)

1997(36)

1998(49)

1999(50)

2000(42)

2001(49)

2002(49)

2003(53)

2004(52)

2005(52)

2006(51)

2007(50)

2008(42)

2009(55)


% o

f eva

luat

ed p

roje

cts


Table 7.1: Environmental and social performance, percentage distribution of assigned

Ratings (664 investment operations evaluated 1996-2009)

1996-97

1996-98

1996-99

1996-2000

1996-2001

1996-2002

1996-2003

1996-2004

1996-2005

1996-2006

1996-2007

1996-2008

1996-2009

Excellent 3% 10% 14% 16% 14% 15% 14% 14% 14% 14% 14% 14% 13% Good 34% 34% 30% 31% 36% 37% 39% 42% 42% 42% 42% 41% 41% Satisfactory 51% 43% 42% 39% 35% 32% 32% 30% 30% 31% 31% 31% 32% Subtotal 89% 87% 86% 85% 85% 84% 85% 86% 86% 87% 87% 86% 86% Marginal 6% 7% 8% 10% 10% 12% 12% 11% 11% 10% 10% 11% 11% Unsatisfactory 6% 6% 6% 5% 5% 4% 3% 3% 3% 3% 3% 3% 3% Highly Unsatisfactory

0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%

Subtotal 11% 13% 14% 15% 15% 16% 15% 14% 14% 13% 13% 14% 14% Total (no. of projects)

70 119 169 211 260 309 362 414 466 517 567 610 664


Chart 7.2: Extent of environmental change, percentage distribution of assigned ratings


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1996(34)

1997(36)

1998(49)

1999(50)

2000(42)

2001(49)

2002(49)

2003(53)

2004(52)

2005(52)

2006(51)

2007(50)

2008(42)

2009(54)


% o

f eva

luat

ed p

roje

cts

Outstanding Substantial Some None/Negative

Table 7.2 Extent of environmental change, percentage distribution of assigned ratings


1996-97

1996-98

1996-99

1996-2000

1996-2001

1996-2002

1996-2003

1996-2004

1996-2005

1996-2006

1996-2007

1996-2008

1996-2009

Outstanding 0% 1% 1% 1% 2% 1% 1% 1% 2% 2% 2% 2% 2% Substantial 29% 32% 26% 24% 22% 21% 20% 20% 20% 21% 21% 22% 22% Subtotal 29% 33% 27% 25% 24% 22% 21% 21% 22% 23% 23% 24% 24% Some 49% 44% 46% 49% 52% 53% 54% 53% 53% 52% 53% 53% 53% None/ Negative 23% 24% 27% 26% 25% 25% 25% 26% 25% 25% 24% 23% 23% Subtotal 71% 67% 73% 75% 76% 78% 79% 79% 78% 77% 77% 76% 76% Total (no. of projects)

70 119 169 211 260 309 362 414 466 517 567 610 663

86 per cent of evaluated operations obtained a Satisfactory or better rating of environmental performance of the sponsor. A total of 11 per cent were rated Marginal in this respect and only 3 per cent were evaluated as having Unsatisfactory performance (no project was rated Highly Unsatisfactory). The ratings confirm that the Bank has generally been successful in improving the environmental performance of projects, with very few exceptions. The extent of environmental change of evaluated projects was rated as Substantial or Outstanding in 24 per cent of the cases, Some for 53 per cent and None/Negative for 23 per cent.17 7.4. ENVIRONMENTAL AND SOCIAL IMPACT RATING

In 2008 the Evaluation Department introduced a new rating for Environmental and Social Impact (ESI) on a trial basis. This takes account of both Environmental Performance and Environmental Change. To date, 54 evaluated projects have been rated for this indicator, and the outcomes are shown in Table 7.3 below. EvD recommends that either the Bank (ESD) begin to fully utilize the new indicator or the pilot be discontinued.

17 The rating system introduced in 2004 no longer distinguishes between None and Negative.


Table 7.3 Environmental and social impact, percentage distribution of assigned ratings


Excellent Good Satisfactory Sub-total

Marginal Unsatisfactory Highly Unsatisfactory

Sub-total

No. of evaluations

2008 8% 32% 28% 68% 28% 4% 0% 32% 25 2009 0% 58% 14% 72% 28% 0% 0% 28% 29 Total 4% 46% 20% 70% 28% 2% 0% 30% 54

8. BANK HANDLING

Bank Handling assesses the due diligence, structuring and monitoring of the project and judges the quality of the work of the Banking Department, in particular the Operation Team, and support departments involved in the operation process, including Environmental Department.

Chart 8.1: Bank Handling, percentage distribution of assigned ratings (671 investment projects evaluated 1996-2009)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1996(34)

1997(36)

1998(49)

1999(50)

2000(42)

2001(49)

2002(49)

2003(48)

2004(52)

2005(52)

2006(52)

2007(54)

2008(43)

2009(61)


% o

f eva

luat

ed p

roje

cts


Table 8.1: Bank Handling, percentage distribution of assigned ratings

(671 investment projects evaluated 1996-2009)

1996-2001

1996-2002

1996-2003

1996-2004

1996-2005

1996-2006

1996-2007

1996-2008

1996-2009

Excellent 13% 13% 15% 15% 17% 18% 18% 18% 17% Good 40% 40% 40% 42% 42% 43% 43% 43% 45% Satisfactory 21% 21% 22% 21% 21% 21% 22% 22% 22% Subtotal 74% 74% 76% 78% 80% 82% 83% 83% 84% Marginal 13% 14% 14% 13% 12% 11% 10% 10% 10% Unsatis-factory 9% 8% 7% 6% 6% 5% 5% 5% 4% Highly Unsatis-factory

4% 4% 3% 3% 2% 2% 2% 2% 2%


260 309 357 409 461 513 567 610 671

The results show that 622 per cent of the operations rated for bank handling have achieved a rating of Good or Excellent and a further 22 per cent Satisfactory. However, nearly one in six


of evaluated projects obtained a less than satisfactory rating. This group of projects, in particular, generated important lessons learned.

9. THE BANK’S INVESTMENT PERFORMANCE

In calculating the Bank's investment performance, EvD uses the model developed by Strategic and Corporate Planning and Budgeting Department and introduced in 2000, which is used by the banking department on projects at the approval stage. EvD inputs actual recorded costs and risk adjustments to recalculate the investment performance at the time of evaluation.

Chart 9.1: Bank Investment Performance, percentage distribution of assigned ratings (289 investment projects evaluated 1996-2009)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1996(14)

1997(15)

1998(19)

1999(21)

2000(18)

2001(20)

2002(22)

2003(18)

2004(23)

2005(23)

2006(26)

2007(24)

2008(22)

2009(24)


% o

f eva

luat

ed p

roje

cts


Table 9.1: The Bank’s investment performance, ratings distribution in OPER reports 1996-2009

1996-2001

1996-2002

1996-2003

1996-2004

1996-2005

1996-2006

1996-2007

1996-2008

1996-2009

Excellent 21% 18% 16% 14% 14% 15% 14% 13% 12% Good 3% 5% 9% 13% 14% 15% 16% 17% 17% Satisfactory 34% 34% 35% 36% 36% 35% 37% 38% 38% Subtotal 58% 57% 60% 63% 64% 65% 67% 68% 67% Marginal 14% 16% 16% 15% 17% 15% 14% 15% 15% Unsatisfactory 12% 11% 10% 9% 8% 8% 7% 6% 7% Highly Unsatisfactory

16% 16% 14% 13% 11% 12% 12% 11% 11%


107 129 147 170 193 219 243 265 289

The sample number is smaller than for other indicators as only OPER reports are rated for this indicator. Starting from 2009, the indicator will be applied to XMR Assessments as well. Over the period, 67 per cent of fully evaluated operations achieved a Satisfactory or better rating for investment performance. The Investment Performance of projects evaluated in 2009 was markedly lower than in previous years. Only 54% of evaluated projects achieved a Satisfactory rating or better, indicating that the project was expected to cover its direct and indirect costs and make a contribution to the Bank's profitability. Only two projects achieved


a Good rating, which means that the project achieved the return required for a Satisfactory rating and performed better than projected at appraisal. A further 25% of projects evaluated in 2009 were rated Marginal, meaning that they were expected to cover direct but not indirect costs. These results are not unexpected in the current economic climate. Given that most operations are still active and repaying at the time of evaluation, they do not necessarily mean that the operations will turn out loss-making in the longer term. The rating of loan operations is affected by factors including specific provisions and risk adjustments made at the facility and country level, while the rating of equity operations will be affected by lower equity valuations.

10. EVALUATION PERFORMANCE RATINGS IN EARLY TRANSITION COUNTRIES

In past years, the AEOR reported on performance ratings in early transition countries (ETCs). Last year a decision was taken to follow the Bank's standard classifications for regions as well as industry sectors, and these do not show ETCs separately. This section was added to allow a brief analysis of ETCs to be included. Chart 10.1 below shows the development of ratings over time for some key performance indicators.

Chart 10.1 Key evaluation indicators in early transition countries: development of ratings over time (83 investment projects evaluated 1996-2009)

Overall performance

27%42% 40%

40%

33%49%

33%

3%

9%24%

0%

20%

40%

60%

80%

100%

1996-1999 (15) 2000-2004 (33) 2005-2009 (35)


% o

f eva

luat

ed p

roje

cts


Transition impact

9%20%

48%43%

53%18% 17%

13%15%

15%29%

3%13%

0%

20%

40%

60%

80%

100%

1996-1999 (15) 2000-2004 (33) 2005-2009 (35)


% o

f eva

luat

ed p

roje

cts


Additionality

71%

7%18%

93%79%

9%

17%3% 3%

0%

20%

40%

60%

80%

100%

1996-1999 (15) 2000-2004 (33) 2005-2009 (35)


% o

f eva

luat

ed p

roje

cts


Project financial performance

9%7%

16%34%

20%

10%

26%

27%

23%

10%

29%

9%

17%

40%

6%7% 13%

0%

20%

40%

60%

80%

100%

1996-1999 (15) 2000-2004 (31) 2005-2009 (35)


% o

f eva

luat

ed p

roje

cts

Excellent Good SatisfactoryMarginal Unsatisfactory Highly Unsatisfactory


The Overall Performance rating is lower than in all other regions except Central Asia. It has shown improvement over time, although the proportion of projects rated Successful or Highly Successful has not changed much over the last two periods. The proportion of projects rated Unsuccessful in the last five years has been only 9% - virtually the same as in the entire population (8% over this period). The performance on Transition Impact has followed a similar pattern, with no recent increase in the proportion rated Satisfactory or better, but a continued fall in the number of projects rated Unsatisfactory. Although progress appears to have stalled to some degree, Transition Impact ratings in the ETC group are better than in the Central Asia and EEC groups, which cover the ETC countries plus some others. This means that ETC countries are performing better than other countries in the same regions in terms of the evaluated Transition Impact of the Bank's projects. Project Financial Performance is still lower in this region than elsewhere, but it has continued to improve strongly over time. This is important because previous AEORs have commented on the necessity for financial sustainability in order for Transition Impact to be achieved. It is surprising that 20% of projects in the period 2005-09 did not have Additionality Verified at Large or Verified in All Respects. This results mostly from a series of operations with a particular bank, which were rated Verified in Part.

Annex 1 to Appendix 8

ADDITIONAL TABLES AND CHARTS RELATING TO APPENDIX 8

Table 1: Overall performance, percentage distribution of assigned ratings (679 investment operations evaluated 1996-2009)

Year of evaluation Unsuccessful Partly Successful

Sub-total Successful Highly Successful

Sub-total No. of evaluations

1996 15% 24% 39% 46% 15% 51% 34 1997 11% 39% 50% 39% 11% 50% 36 1998 22% 21% 43% 53% 4% 57% 49 1999 24% 22% 46% 46% 8% 54% 50 2000 21% 22% 43% 40% 17% 57% 42 2001 24% 30% 54% 46% 0% 46% 50 2002 14% 38% 52% 42% 6% 48% 50 2003 8% 38% 46% 48% 6% 54% 53 2004 10% 17% 27% 60% 13% 73% 52 2005 8% 23% 31% 52% 17% 69% 52 2006 8% 31% 39% 42% 19% 61% 52 2007 9% 29% 38% 56% 6% 62% 54 2008 7% 41% 48% 45% 7% 52% 44 2009 8% 41% 49% 48% 3% 51% 61

1996-97 13% 31% 44% 43% 13% 56% 70 1996-98 17% 27% 44% 47% 9% 54% 119 1996-99 19% 25% 44% 47% 9% 56% 169

1996-2000 19% 25% 44% 46% 10% 56% 211 1996-2001 20% 26% 46% 46% 8% 54% 261 1996-2002 19% 28% 47% 45% 8% 53% 311 1996-2003 18% 29% 47% 45% 8% 53% 364 1996-2004 17% 28% 45% 47% 8% 55% 416 1996-2005 16% 27% 43% 48% 9% 57% 468 1996-2006 15% 28% 43% 47% 10% 57% 520 1996-2007 14% 28% 42% 48% 10% 58% 574 1996-2008 14% 28% 42% 48% 10% 58% 618 1996-2009 13% 29% 42% 48% 10% 58% 679

Table 2: Transition Impact ratings of 679 investment operations evaluated –1996-2009

Year of evaluation

Negative Unsatis-factory

Marginal Subtotal Negative - Marginal

Satisfactory Good Excellent Subtotal Satisfactory -

Excellent

Total projects

evaluated 1996 0% 3% 15% 18% 18% 58% 6% 82% 34 1997 0% 3% 25% 28% 19% 39% 14% 72% 36 1998 6% 6% 14% 26% 27% 37% 10% 74% 49 1999 6% 12% 10% 28% 38% 26% 8% 72% 50 2000 5% 10% 10% 25% 19% 44% 12% 75% 42 2001 4% 16% 14% 34% 24% 42% 0% 66% 50 2002 2% 6% 16% 24% 24% 46% 6% 76% 50 2003 0% 4% 11% 15% 30% 51% 4% 85% 53 2004 2% 2% 13% 17% 10% 63% 10% 83% 52 2005 0% 4% 12% 16% 23% 49% 12% 84% 52 2006 0% 6% 15% 21% 19% 43% 17% 79% 52 2007 4% 2% 4% 10% 35% 35% 20% 90% 54 2008 0% 5% 9% 14% 31% 48% 7% 86% 44 2009 0% 3% 22% 25% 16% 51% 8% 75% 61

1996-1997 0% 3% 20% 23% 19% 48% 10% 77% 70 1996-1998 2% 4% 18% 24% 22% 44% 10% 76% 119 1996-1999 4% 7% 15% 26% 27% 38% 9% 74% 169 1996-2000 4% 7% 14% 25% 25% 40% 10% 75% 211 1996-2001 4% 9% 14% 27% 25% 40% 8% 73% 261 1996-2002 4% 8%% 14% 26% 25% 41% 8% 74% 311 1996-2003 3% 8% 14% 25% 25% 43% 7% 75% 364 1996-2004 3% 7% 14% 24% 24% 45% 7% 76% 416 1996-2005 2% 7% 14% 23% 23% 46% 8% 77% 468 1996-2006 2% 7% 14% 23% 23% 45% 9% 77% 520 1996-2007 2% 6% 13% 21% 24% 45% 10% 79% 574 1996-2008 2% 6% 13% 21% 24% 45% 10% 79% 618 1996-2009 2% 6% 13% 21% 24% 45% 10% 79% 679


ANALYSIS OF FACTORS AFFECTING PERFORMANCE IN EVALUATION REPORTS:

SUPPORTING MATERIAL TO CHAPTER 3 1. THE ORDERED LOGIT MODEL A.1.0 REFERENCES

Train, K.E. “Discrete Choice Methods with Simulation” 2nd Edition, Chapter 7, Cambridge University Press

A.1.1 INTRODUCTION

One of the difficulties faced in this analysis is that the variables that are being investigated in this project are fundamentally unquantifiable and not fully observable. It is difficult to place OPER ratings like project success or transition impact on a continuous scale on the real line and of course any attempt to do so would be highly subjective. Instead, ordered, discrete categorisations are used; a project is “Highly Successful”, “Successful” etc. Thus standard statistical measures, comparing correlations, or the simple regression approaches of ordinary least squares are ineffective.

A.1.2 LATENT UNOBSERVED VARIABLES

The Ordered Logit model takes into account these discrete categorisations and also relies on the fact they are “ordered” - that a “highly successful” project is better than a “successful”.

The underlying idea behind this approach is that there is latent variable on a continuous scale which describes the performance of the project but is unobservable. Intuitively, the higher the latent variable the higher category the particular observation should obtain. For example, if the latent variable is an overall project success we would expect the as the latent variable increases the project will move up the various discrete categories. What is important is to define how to map this latent variable into the discrete categories that are observed. The Ordered Logit approach does this by assuming cut-off points between categories, so as the latent variable crosses a certain cut-off it moves from one category to the next; obviously the number of cut-off points is one less than the number of categories. More formally:

2

A.1.3 STOCHASTIC ELEMENTS

The next step is to describe the process by which the latent variable is formed. A standard linear model is assumed:

Where is the vector of explanatory variables – in this project it would be the factors affecting performance plus any other covariates, is the vector of co-efficients and , a stochastic error term. Related back to the example of project performance, this model is saying the success of the project depends on a weighted sum of the various factors, with the weights determined by , plus some stochastic heterogeneity not captured by or correlated with the factors. This seems a not unreasonable assumption to make.

From here it is possible to calculate the probability of a project being in a certain category given its factors, , the cut-off points and a distributional assumption over the error term. We know that:

)

)

)

Where is the cumulative probability distribution of the error term , i.e . Since this a Logit model the assumption is that the error term follows a

standardised logistic distribution, hence:

Alternative distributional assumptions can be made, for example, that the error term is normally distributed (an Ordered Probit Model); however, for the sample size used in this project the difference in the estimated parameters, once appropriately rescaled, between a Logit and a Probit model are very small.

A.1.4 ESTIMATION

To complete the model, estimates need to be made over the values of and the cut-off points. A maximum likelihood approach is taken; this is equivalent to saying given the assumptions made what values of and cut-off points maximise the chance that the sample

3

observed will occur.1 With a model with as many moving parts as this it is impossible to solve this question analytically, instead the econometrics software uses an iterative procedure to search through various combinations of parameters that maximise the likelihood.

A.1.5CALCULATION OF MARGINAL EFFECTS

The parameters of are not directly interpretable – they reflect the impact that a change in explanatory variable has on an unobservable variable. Instead, the key result to take from these sorts of model are the marginal effects of a change in a explanatory variable has on the probability of a particular category occurring, – for example, what is the impact of an additional positive factor on the probability that a project is rated highly successful. There are two ways to look at the marginal effects. First, is to take the derivative of a particular probability with respect to the explanatory variable in question. The second approach is to consider how the probabilities change given a discrete change in an explanatory variable2. While it is more common to consider the marginal effects in terms of a derivative, the second approach is taken in this project.3 This is because the explanatory variables used are themselves largely binary – there is either a positive factor or there is not. Hence it does not make sense to consider what small changes in the explanatory variables do to the probabilities but instead to ask what happens if there is one more or one less factor.

The calculation of the marginal effects using this approach is rather straight forward. We know that the probability of a project be categorised in a certain way is equivalent to:

If there is now an additional positive factor which has co-efficient , then the new probability equals:

The one issue that needs to be considered when considering the marginal effects is the starting point – i.e. . For example, if a project already has many negative factors an additional negative factor will have a very limited effect on the probability on an “unsuccessful rating”, but this will change as the number of negative factors at the starting point is reduced. However, the cut-off points provide a set of intuitive starting points as they represent the point when a project is on a cusp between the two different categories, thus this when the sensitivity to changes in explanatory variables is largest and hence when the results are most powerful.

1 This is a rather abstract explanation for a simple concept. As a more intuitive example, imagine an unfair die was rolled 100 times and a six is observed 50 times. The maximum likelihood estimator for the probability of a 6 occurring is simply a half. 2 These two approaches become equivalent as the size of the discrete in change in the second approach tends towards zero. 3 If it is of further interest the mathematics behind the first approach are detailed in the references listed at the start of this appendix.

4

2. CHARTS AND TABLES

Table A.2.1. Conversion formulae for transition impact indicators, comparison of scales

New Scale (From 2000) Old Scale (1996-1999) Excellent 6 High 6 Good 5 Medium/High 5 Satisfactory 4 Medium 4.5 Marginal 3 Medium/Low 4 Unsatisfactory 2 Low 3 Negative 1 None 2 No Rating 0 Negative 1 No Rating 0

Table A.2.2. Correlations across performance ratings

Net Factor

Balance

Overall Perform.

Transition Impact

Environ. Perform.

Environ.Chg Comp. Financial Perform.

Project Financial Perform.

Fulfilment of

Objectives

Bank Handling

Investment Perform.

Net Factor Balance — 0.90 0.79 0.37 0.29 0.72 0.74 0.79 0.69 0.52

Overall Perform. 0.90 — 0.85 0.44 0.32 0.72 0.73 0.86 0.73 0.53

Transition Impact 0.79 0.85 — 0.42 0.32 0.65 0.64 0.79 0.76 0.48

Environ. Perform. 0.37 0.44 0.42 — 0.54 0.36 0.33 0.41 0.36 0.23

Environ.Chg 0.29 0.32 0.32 0.54 — 0.20 0.19 0.31 0.31 0.14

Comp. Financial Perform.

0.72 0.72 0.65 0.36 0.20 — 0.91 0.69 0.62 0.58

Project Financial Perform.

0.74 0.73 0.64 0.33 0.19 0.91 — 0.70 0.61 0.57

Fulfilment of

Objectives 0.79 0.86 0.79 0.41 0.31 0.69 0.70 — 0.73 0.50

Bank Handling 0.69 0.73 0.76 0.36 0.31 0.62 0.61 0.73 — 0.49

Investment Perform. 0.52 0.53 0.48 0.23 0.14 0.58 0.57 0.50 0.49 —

5

Chart A.2.1 Breakdown of net factor balances

0

5

10

15

20

25

30

35

40

‐7 ‐6 ‐5 ‐4 ‐3 ‐2 ‐1 0 1 2 3 4 5 6 7 8

Reading

Freq

uency

Average for successful projects

Average for highly successful projects

Average for partly successful projects

Average for unsuccessful projects

Chart A.2.2. Distribution of net factor balances against project performance

‐8

‐6

‐4

‐2

0

2

4

6

8

0 1 2 3Overall Performance

Net Factor Ba

lance

4

APPENDIX 10

ENVIRONMENTAL AND SOCIAL CATEGORIZATION To assist the Bank in the development of the new 2008 Environmental and Social Policy, the Evaluation Department (“EvD”) undertook two special studies, one focused on financial intermediaries (“FIs”) and one on direct investments, to draw lessons learned from past experience so as to contribute to the design of the new Policy.

This work was extremely useful for the Environment and Sustainability Department in its development of the 2008 Environmental and Social Policy which was approved in May 2008. Indeed most of the issues raised by EvD were incorporated. In particular, the change from a focus on “use of proceeds” to “use of funds” as reflected in the new policy and the focus on the “business activities” and “area of influence” as presented in the 2008 ESP as very positive. This is a very constructive change and should eliminate much of the prior disagreements on project categorisation, particularly with respect to C-category projects. In addition, the reorganisation of the ESD in 2009 has focused primarily on improving governance and oversight functions, a key recommendation from EvD. Following discussions in 2008 at the Audit Committee, EvD and ESD agreed to work together on the issue of Categorization and to report back. A joint internal paper was prepared. This section briefly summarizes the findings that emerged from that process. The agreed recommendations are incorporated in Chapter 7 of this AEOR. Observations from Past Evaluations: From past experience and evaluation findings, under the 2003 Environmental Policy (2003 EP), three trends were observed, which are causes for concern:

• Due to an increase in equity, working capital and refinancing projects, there was in increase in Category C projects relative to Category A and B.

• Category A projects historically result in better Environmental Performance and higher Transition Impact than Category B, which outperform Category C projects, and

• In the last 6 years, the Bank’s annual environmental performance outcomes have declined (see section 1.4 of this AEOR).

Figure 1 shows the patterns of environmental and social categorisation throughout the life of the Bank. Ignoring the first three start-up years, these data demonstrated the following:

• There has been significant growth in the Bank’s portfolio after the Russian crisis. • FIs have been a significant portion of the Bank’s portfolio, from a low of 42% in 1999

to a high of 59% in 2006. • Category A level projects have been relatively constant in absolute numbers and

relative percentage, with a high of 16 projects (7%) in 2004. As a percentage of the portfolio Category A projects peaked in 1998 at 11% (13 projects).

• The number of Category B level projects peaked in 2005 at 68, but this only represented 30% of the portfolio. As a percentage the peak was in 1999 at 46%. However, Category B/0 has declined at the expense of Category B/1, which indicates that the Bank made more investments in existing companies.

• In contrast, the number of Category C level projects reached a peak in 2007 at 54 projects, which was also a peak in percentage at 19%, with the growth most

2

noticeable in Category C/1, again, reflecting the investment in companies with existing facilities.

• There has not been a material increase in the proportion of C/0 projects.

Figure 1: Environmental categorization of signed operations (excluding TFP)

0%10%20%30%40%50%60%70%80%90%

100%

1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008

Ye a r o f s ig ning

A B C FI IEE

Focusing on direct investments only, the most conservative way to display these data is as a cumulative percentage graph, i.e. adding year 1 results to year 2, then year 1 and 2 results to year 3 etc. (Figure 2) Each year the additional change is or should be smaller, for example in 2008 year, 8 Category A/0 projects are added to the previous 95. If there is no change in the fundamental structure then the curves should flatten out. In fact, Figure 5 shows that Category C projects were growing as a percentage of the overall population, along with a shift from B/0 to B/1. However, ESD and EvD both agree that, partly as a result of the way categorization has been defined under the 2008 Environmental and Social Policy, the number of Category C projects has significantly decreased and most are now screened as Category B projects. It can be concluded that through 2008, the major growth has been in Category FI projects, followed by a significant growth in Category B/1 and C/1 projects. For direct investments (A,B,C) the nature of the projects has not fundamentally changed to the degree that these data imply, rather, the changes reflect a change in the way projects are reviewed by ESD and presented to the Board by Banking. EvD and ESD attribute the past increase in Category C-level projects to the growth in equity operations.

3

Figure 2: Percent Cumulative Categorization for Direct Investments

0 10 20 30 40 50 60 70 80 90

100

1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 A/0 A/1 B/0 B/1 C/0 C/1

Table of TC Funds

Appendix 11

Donor fundcountry No. % Amount % Amount %Albania 0 0.00% 0 0.00% 40 0.00%Australia 0 0.00% 0 0.00% 80 0.01%Austria 0 0.00% 0 0.00% 16,103 1.25%Belarus 0 0.00% 0 0.00% 431 0.03%Belgium 0 0.00% 0 0.00% 1,532 0.12%Canada 8 2.16% 3,714 2.50% 28,194 2.20%Czech Republic 0 0.00% 0 0.00% 573 0.04%Denmark 4 1.08% 268 0.18% 5,918 0.46%Finland 4 1.08% 429 0.29% 13,522 1.05%France 7 1.89% 1,842 1.24% 20,901 1.63%Germany 7 1.89% 1,237 0.83% 18,236 1.42%Greece 0 0.00% 0 0.00% 1,742 0.14%Iceland 1 0.27% 50 0.03% 202 0.02%Ireland 3 0.81% 88 0.06% 2,793 0.22%Israel 1 0.27% 38 0.03% 305 0.02%Italy 10 2.70% 1,138 0.77% 60,696 4.73%Japan 45 12.13% 12,584 8.47% 122,519 9.55%Luxembourg 1 0.27% 0 0.00% 6,431 0.50%Netherlands 18 4.85% 4,011 2.70% 56,775 4.42%New Zealand 1 0.27% 68 0.05% 175 0.01%Norway 1 0.27% 908 0.61% 6,516 0.51%Portugal 1 0.27% 19 0.01% 786 0.06%Republic of Korea 0 0.00% 0 0.00% 839 0.07%Spain 0 0.00% 0 0.00% 12,577 0.98%Sweden 4 1.08% 554 0.37% 21,033 1.64%Switzerland 10 2.70% 2,458 1.65% 21,397 1.67%Taipei China 15 4.04% 1,628 1.10% 18,239 1.42%Turkey 1 0.27% 105 0.07% 300 0.02%United Kingdom 11 2.96% 1,811 1.22% 38,158 2.97%USA 9 2.43% 2,902 1.95% 61,749 4.81%Other donorsEU 166 44.74% 104,402 70.26% 461,398 35.95%Multi-donor funds1 43 11.59% 8,340 5.61% 213,271 16.62%Financial Sector2 0 0.00% 0 0.00% 2,174 0.17%Various3 0 0.00% 0 0.00% 67,750 5.28%TOTAL 371 100.00% 148,595 100.00% 1,283,357 100.00%

Portfolio data from Funds Reporting December 2009OPER report data from Datawarehouse December 2009

3 Including Alliance Bank, Global Environment Facility, UNDP, EBRD, Chevron Munaigas Inc, BP Exploration, Energy Resources LLC, JSC Arcelormittal Temirtau, World Bank.NB: EvD has also evaluated certain Nuclear Safety Funds, which fall outside the scope of the regular Technical Cooperation Funds Programme.

Covered by OPER reports Portfolio-wide

1 Funds include TAM Nordic Council, EBRD Early Transition Countries Fund, Baltic Fund, RSBF, EBRD TC Special Fund, Balkan Region Fund, Mongolia TC Fund, RVF for North West Russia, EBRD Annual General Meeting 2000, Western Balkans Fund. Donors include the G-7, Nordic countries, Ireland, Luxembourg, Netherlands, Spain, Switzerland and Taipei China.

2 Contributions to these funds consist of technical assistance fees payable by the borrowers under the terms of loan agreements between EBRD and certain financial intermediaries.


EBRD PERFORMANCE EVALUATION BENCHMARKS FOR OVERALL PERFORMANCE AND FOR INDIVIDUAL PERFORMANCE EVALUATION

CATEGORIES 1. OVERALL PERFORMANCE MATRIX Table 1 below shows the weighting process to arrive at the Overall Performance rating. The table gives combinations of ratings applying four major performance indicators (transition impact, project/company financial performance, fulfilment of objectives and environmental and social performance), whereby transition impact gets the highest weight when judging the overall performance of an operation. Apart from these four major indicators, of course the remaining indicators, additionality, bank handling and investment performance, also play a role when assigning the overall performance rating, but to a lesser degree define the overall performance outcome of a project. The table further shows the importance of the performance indicators on sustainability (financial performance and fulfilment of objectives) that help in realising transition impact to enfold during the life of a project.

Table 1 GUIDELINES FOR ASSIGNING THE OVERALL PERFORMANCE RATING FOR COMBINATIONS OF RATINGS ON FOUR MAJOR

PERFORMANCE INDICATORS

OVERALL PERFORMANCE RATING

Transition Impact rating

Project/Company Financial

Performance rating

Fulfilment of Project

Objectives

Environmental and Social

Performance HIGHLY SUCCESSFUL Excellent Excellent Excellent Excellent

Excellent Good Excellent Good Excellent Excellent Good Good Good Excellent Excellent Excellent

SUCCESSFUL Excellent Good Good Good Excellent Marginal Satisfactory Good Good Good Excellent Good Good Good Good Good Good Satisfactory Good Good Good Good Satisfactory Good Good Good Satisfactory Satisfactory Good Marginal Excellent Good Satisfactory Satisfactory Good Good Satisfactory Satisfactory Satisfactory Excellent

PARTLY SUCCESSFUL Good/Excellent Marginal Marginal Satisfactory Good Marginal Good Satisfactory Satisfactory Satisfactory Satisfactory Satisfactory Satisfactory Marginal Satisfactory Satisfactory Marginal Good Good Satisfactory Good Good Good Marginal

UNSUCCESSFUL Marginal Marginal Good Marginal Marginal Marginal Marginal Marginal Unsatisfactory All All All Negative All All All The combinations of ratings for assigning an overall performance rating in the above table are not exhaustive. The combinations listed give an indication of how the weighting process works and gives guidance to Evaluation Staff and Staff in the Banking Department during the subjective process of assigning ratings to overall project performance. However, in assigning ratings of Good or Excellent, etc., it is important to define, as elaborated on in the next section, what are benchmarks to assign these rating categories.

APPENDIX 12 Page 2 of 8 2. BENCHMARKING PERFORMANCE RATINGS

2.1 TRANSITION IMPACT EvD tends to evaluate a project relatively soon after disbursement (18 months as described in Section 2.4.1 of the main text) and the evaluator should be conscious that concrete evidence of the achievement of some transition objectives may not become visible for some time. As presented in the transition impact criteria table in Appendix 2 and transition impact rating table in Appendix 5, the evaluation methodology allows for three ratings on each of the criteria which are relevant for the specific operation: (a) judging the realised transition impact at the time of evaluation; (b) assessing the transition potential that can still be reached, and (c) assigning a risk rating (Low, Medium, High, Excessive) in respect of the likelihood to reach the full transition impact potential over time. A high rating could be appropriate where the transition impact potential in the future is considered substantial. However, if the probability that the transition impact potential can be reached is low due to considerable risk, the evaluator will award a higher 'risk-to-transition-impact' rating and explain the nature of the risk. As explained in section 1.1 above, the transition impact is measured at the industry level and the level of the economy as a whole, including possible regional and cross-border effects. During the evaluation of transition impact EvD concentrates on assessing performance under the “major relevant transition impact objectives” as mentioned in Table 2. They are those objectives (mostly two or three) identified by the Operation Team during project appraisal which are presented in the operation reports to the Board of Directors and monitored through TIMS. EvD also reviews performance under the other transition impact criteria to identify whether any important transition effect might have been missed. Therefore, EvD reviews all seven criteria in the overall assessment of transition impact.

The ratings, as under current practice range from Excellent, Good, Satisfactory, Marginal, and Unsatisfactory to Negative. In assigning these ratings the benchmarks provided in Table 2 below are applied:

Table 2 RATING TRANSITION IMPACT

RATINGS

BENCHMARKS

Excellent

The project achieved significant progress toward all major relevant transition impact objectives. Best practice was achieved in one or more areas.

Good

The project achieved significant progress toward all major relevant transition impact objectives, possibly with minor shortcomings.

Satisfactory

The project achieved acceptable progress toward a majority of the major relevant transition impact objectives, but did not make acceptable progress towards one major objective.

Marginal

The project failed to achieve acceptable progress towards a majority of relevant transition impact objectives. However, progress toward at least one major objective was acceptable.

Unsatisfactory

The project failed to achieve acceptable progress toward any of its major relevant transition impact objectives.

Negative

The project failed to achieve acceptable progress toward any of its major relevant transition impact objectives and even had in some cases a negative effect.

2.2. PROJECT AND COMPANY FINANCIAL PERFORMANCE1

1 Evaluators can, in exceptional cases, take into account local industry performance when judging project and company

financial performance of a project based on initial conditions. Exceptional cases are those whereby the difference in perception of financial performance between the evaluator and the project team differs at least two rating categories.


a. Project financial performance. In the analysis of a non-financial market project financial performance EvD uses an appropriate range of performance indicators in project financing such as: sales figures, net profit, debt service coverage, FIRR and EIRR. Suitable project return analysis will supplement balance sheet and income related indicators. Apart from financial internal rates of return (FIRR) calculation, imperfect markets, significant subsidies or factor price distortions, or externalities justify calculation of the economic internal rate of return (EIRR). Annex 1 to this appendix contains a table with the financial performance indicators used in the evaluation. It should be taken into account that the various performance indicators might somewhat differ per sector, due to specific financial characteristics of the sector. In respect of financial market operations the evaluator has to judge the project portfolio’s profit contribution to the financial intermediary or investment fund. Table 3 gives guidance to assign ratings in respect of project financial performance:

Table 3 RATING PROJECT FINANCIAL PERFORMANCE

RATINGS

BENCHMARKS

Excellent Actual and re-assessed performance indicators are in principle on average 10% better than anticipated at appraisal. Prospects are positive.

Good Actual and re-assessed performance indicators are in principle on average between 0-9.90% better than anticipated at appraisal. Prospects are positive

Satisfactory Indicators are in principle in line with appraisal estimates, but some problems (management, financial, economic, etc.) have been encountered that can influence the prospects of the project negatively.

Marginal Indicators are in principle up to 25% below expectations at approval, but prospects of financial improvement exist.

Unsatisfactory The project shows performance indicators in principle >25% below expectations with limited prospect of improvements in the immediate future.

Highly Unsatisfactory

Complete project failure whereby the Bank loses part or its entire investment.

b. Company financial performance. When a non-financial market company’s financial performance is assessed by EvD it uses an appropriate range of corporate performance indicators: sales figures, net profit, debt/equity position, debt service coverage. As under project financial performance the various performance indicators might somewhat differ per sector, due to specific financial characteristics of the sector in which the company operates. In respect of financial market operations the company performance will be judged by assessing the company’s portfolio credit and equity FIRR performance as well as their liquidity position. Table 4 gives guidance to assign ratings in respect of company financial performance:


Table 4 RATING COMPANY FINANCIAL PERFORMANCE

Ratings

BENCHMARKS

Excellent Actual and re-assessed performance indicators of the company are in principle on average 10% better than anticipated at appraisal. Prospects are positive.

Good Actual and re-assessed performance indicators are in principle on average between 0-9.90% better than anticipated at appraisal. Prospects are positive.

Satisfactory Indicators are in principle in line with appraisal estimates, but some problems (management, financial, economic, etc.) at the level of the company have been encountered that can influence the prospects of the project negatively.

Marginal Indicators are in principle up to 25% below expectations at approval, but prospects of financial improvement exist.

Unsatisfactory The company shows performance indicators in principle >25% below expectations with limited prospect of improvements in the immediate future


Complete company failure that can have dramatic effects on the project and even terminate the project so that the Bank loses all its investments.

2.3 FULFILMENT OF PROJECT OBJECTIVES (EFFICACY) The assessment of fulfilment of objectives concerns verified and risk weighted fulfilment potential of the operation's “process” and “project” objectives upon validation of their relevance. The “project” objectives under review are for instance those related to carrying out an investment plan in respect of plant and equipment and the establishing of a strong management team. In respect of “process” objectives these can be the introduction of an IAS accounting system or for a financial institution the improvement of credit manuals and the training of staff. Fulfilment of project objectives does not incorporate the transition impact objectives which are captured under the transition impact performance rating. Table 5 presented below provides benchmarks for the fulfilment of project objectives:

Table 5 RATING FULFILMENT OF PROJECT OBJECTIVES

RATINGS

BENCHMARKS

Excellent The stated operation objectives at approval are deemed relevant. Early fulfilment or potential

fulfilment, with low risk is verified for all objectives. Plant and equipment are fully operational. A capable management team is effectively in charge and the market built-up is in full swing. The sponsor is fulfilling all its obligations, financial- as well as market-related.

Good Most of the objectives have been fulfilled or are deemed within reach with low applicable risk. Plant and equipment are operational. The management team is functioning adequately. The Sponsor is fulfilling its obligations.

Satisfactory Most of the objectives have been fulfilled or are deemed within reach with some risk to their realisation. Most of plant and equipment are operational, but some delays in installation occurring.. The management team is functioning adequately, though their coming on board saw some delays. The Sponsor is fulfilling its obligations.

Marginal Some of the project objectives have not yet been fulfilled or face a deemed medium-higher risk that they may not be achieved. The sponsor is actively trying to comply with its obligations, but has so far been only partly successful. Some doubts exist about a final positive outcome.

Unsatisfactory The project objectives have not yet been fulfilled with a high risks that many will also not be met later on. Serious doubt exists whether the sponsor is able to fulfil all its obligations. A positive final outcome is doubtful or deemed impossible.


The project objectives have not been fulfilled and the chance of their realisation is practically zero. It is certain that the sponsor is not able to fulfil its obligations in full. A positive final outcome is deemed impossible.

APPENDIX 12 Page 5 of 8 2.4 ENVIRONMENTAL AND SOCIAL PERFORMANCE

2.4.1 Environmental and social performance of the project and the sponsor. Environmental and social performance of projects is measured by assessing the status of the environment in the vicinity of the project and if warranted important wider effects (e.g. captive mines as part of a steel project, the health and safety situation in the project company, the pollution loads and energy efficiency status, the project’s environmental management, social factors2 and the level of public consultation and participation. Table 6 below gives the necessary details of rating categories of the environmental performance of the project and the sponsor.

Table 6 RATING ENVIRONMENTAL AND SOCIAL PERFORMANCE OF THE PROJECT AND THE SPONSOR

RATINGS

BENCHMARKS

Excellent

All appropriate environmental and social (see footnotes 4 and 11) measures are secured and environmental conditionality implemented. No significant outstanding issues. The Sponsor has gone beyond the expectations of the environmental action plan (EAP) and serves as a best practice example.3

Good Appropriate environmental and social (see footnotes 4 and 11) measures are secured and environmental conditionality implemented. The EAP is on or ahead of schedule.

Satisfactory The appropriate environmental and social (see footnotes 4 and 11) risk factors were properly identified and the sponsor is implementing the EAP as prescribed.

Marginal Some environmental and social (see footnotes 4 and 11) measures are secured and only part of environmental and social conditionality was implemented. Several outstanding issues remain. Performance of the sponsor was partly unsatisfactory.

Unsatisfactory Few if any environmental and social (see footnotes 4 and 11) measures were implemented. Significant outstanding issues are experienced. Performance of the sponsor was less than satisfactory.


The project is out of compliance with the objectives as established in the EAP and/or host country or World Bank environmental standards for this type of project; has experienced significant adverse events (spills, deaths, etc.); is an on going risk to the environment; and presents a vulnerability risk to EBRD.

2.4.2 Extent of environmental and social change. An essential part of the environmental and social performance is to identify the extent of environmental and social change, as a result of the project. In view of the large problems of the region with regards to the environmental pollution, Bank projects should address the positive or negative environmental and social (see footnote 4 and 11) effects of projects in an adequate way. It is therefore a very important part of the evaluation exercises to rate the extent of environmental and social change. To do this, it is important to consider both the ex ante and ex post conditions against the stated objectives as defined above. Table 7 below gives details on the rating categories for this.

2 For instance community impacts on indigenous people in the neighbourhood of the project and any resettlement issues. 3 In case a change of environmental policy has occurred between the time of appraisal and evaluation of the project, and

higher standards become applicable, the environmental performance of the project would be rated higher if the project would comply with the new environmental policy.


Table 7 RATING EXTENT OF ENVIRONMENTAL AND SOCIAL CHANGE

RATING

BENCHMARKS

Outstanding

This project will result in significant environmental and social (see footnotes 4 and 11) benefits and/or additionality. The extent of the change is extensive, either because environmental legacies were extensive, or because the project achieves a high level of performance and has excellent potential long-term improvements. Projects which have positive impacts beyond the immediate project (e.g. by positive example lead to new environmental and social standards) should also be considered Outstanding.

Substantial Environmental and social (see footnotes 4 and 11) benefits and/or additionality resulting from the project are significant and have good potential for the future. Beyond the project benefits may also be positive.

Some Some environmental and social (see footnotes 4 and 11) benefits and/or additionality resulting from the project. No measurable benefits beyond the immediate project.

None/Negative No significant environmental and social (see footnotes 4 and 11) benefits associated with the project; or significant adverse (negative) environmental and social impacts associated with the project. Also under this category would be projects that have a negative demonstration effect.

2.5 THE BANK’S ADDITIONALITY The Bank’s additionality in a project is assessed by judging to what extent the client would have been able to secure financing from market financiers on acceptable terms. Another necessary condition is the extent of the Bank's impact on the existence, design or functioning of a project to enhance transition impact. There is a critical level of conditions above which a project becomes and remains additional. In judging additionality at evaluation one tries to verify whether the Bank was additional or not at the time the project was financed by the Bank. Therefore the Bank has introduced the ratings Verified in all respects, Verified at large, Verified only in part and Not verified, as presented in the table below, where the benchmarks for the ratings is given: Benchmarks on rating additionality are presented in Table 8 below.

Table 8 RATING ADDITIONALITY

Ratings

BENCHMARKS

Verified in all respects

No other financial institutions are willing to provide financing at the same or better condition than the Bank. The terms and conditions are not attractive to other banks and the country risk is still high. The client accepts tough conditionality to secure transition impact.

Verified at large Some competition with market financiers, but the Bank's terms and conditions, although more demanding than competition’s, prevail since sponsors/clients or co-financiers appreciate the Bank's political comfort. In such cases, specific project design and structuring may also be significant for enhanced transition impact. The Bank may also have contributed specific country- or sector knowledge or helped enhance corporate governance standards. Repeat financing to a second phase of a project, may fall into this category.

Verified only in part Competition from commercial financiers is significant and terms and conditions are almost identical, but the Bank's participation (e.g. in a bond issue) may have helped an earlier implementation of the project than would have otherwise been possible. No significant features are added to design and functioning to enhance transition and/or catalyse other financing.

Not verified Competition fully established for financing and the Bank's terms and conditions fail to provide for any material transition impact enhancement and pricing premium to account for the availability of the Bank’s Preferred Creditor Status.

APPENDIX 12 Page 7 of 8 2.6 THE BANK'S INVESTMENT PERFORMANCE The Bank’s investment performance in an operation is measured by the Project's net profit contribution. The respective performance rating reflects the extent to which the actual and expected Net Contribution (after risk adjustment) over the life of a Project is sufficient to cover its full transaction cost and to contribute to the Bank's net profit. The rating scale and the profit contribution performance criteria are presented in Table 9 below. The lower end of the scale reflects whether the transaction covers its direct costs and contributes towards general overheads. An operation which makes a satisfactory contribution to overheads achieves a Satisfactory rating. From this level onwards, higher ratings will also need to satisfy comparative tests against performance projections at appraisal.

TABLE 9

THE BANK’S INVESTMENT PERFORMANCE

RATING A LOAN OR EQUITY INVESTMENT'S PROFIT CONTRIBUTION PERFORMANCE

RATINGS BENCHMARKS

Excellent NPVNME4 is equal to or greater than twice Direct Cost and the discounted project contribution after Direct Cost allocation5 is more than 40% higher than the level foreseen at appraisal.

Good NPVNME is equal to or greater than twice Direct Cost and the discounted project contribution after Direct Cost Allocation is more than 10% but not more than 40% higher than the level foreseen at appraisal.

Satisfactory NPVNME is equal to or greater than twice Direct Cost and the discounted project contribution after Direct Cost allocation is not more than 10% higher than the level foreseen at appraisal.

Marginal NPVNME is greater than or equal to Direct Cost but less than twice Direct Cost.

Unsatisfactory NPVNME is less than Direct Cost but greater than or equal to zero (i.e. discounted project contribution after Direct Cost allocation is negative).

Highly Unsatisfactory NPVNME is negative (i.e. discounted project contribution after Direct Cost allocation is negative).

For the purpose of calculating and rating the investment performance of a project EvD uses the financial model that is operated by the Finance Department and that is also used at project appraisal stage. 2.7 BANK HANDLING OF AN OPERATION “Bank handling”, assesses the due diligence, structuring and monitoring of the project and judges the quality of the work of the Banking Department, in particular the Operation Teams, and support departments involved in the operation process, including the Environmental and Sustainability Department. An assessment is made on how effectively the Bank carries out its work during the life of the project. In case operations are evaluated that are handled by the Corporate Recovery Unit, Bank Handling will also take into account problem recognition, remedial action and recovery efforts. Table 10 below presents benchmarks that are used by Evaluation Staff when judging Bank handling in a project:

4 NPVNME (Net Present Value Net Margin Earned): the project's revenue contribution to the Bank's income

statement, net of its financing cost and after risk adjustment to cover the Bank’s expected losses as per the Bank Provisioning Policy, but before recovery of its incremental (direct) transaction cost (for generation and monitoring) or any attributed overheads.

5 Discounted profit contribution after Direct Cost allocation is the same as NPVNME but after deduction of direct transaction costs. This measure is presented at appraisal in the Final Review Memorandum and Board Document, enabling a direct comparison of projections at appraisal and results at evaluation.


Table 10 RATING BANK HANDLING

RATINGS

BENCHMARKS

Excellent Appraisal6 was very well conducted, did not show any gaps and provided an excellent basis to make the investment decision. The Bank structured the operation very well under difficult circumstances thereby securing excellent initial conditions to realise transition impact during the life of the project. Risk to transition was adequately mitigated through a strong conditionality package. Implementation7 was very skilful and contributed to the success of the operation.

Good Appraisal was well conducted, and although not all relevant issues were addressed, provided an adequate basis to make the investment decision. The Bank structured the operation so that adequate initial conditions formed a good basis to realise transition impact during the life of the project. Risk to transition was mitigated through a conditionality package that could have been somewhat stronger. Implementation was skilful and contributed to the success of the operation.

Satisfactory Appraisal could have been better and there is evidence that not all relevant issues were addressed. Nonetheless, it provided a sufficient basis to make the investment decision. Structuring of the operation increased the risk to realise transition impact some important risk mitigating factors were in place. Implementation could have been more skilful and constituted a risk to the project’s success.

Marginal Appraisal was clearly deficient and there is evidence that important issues were not addressed. It did not provide an adequate basis to make a sound investment decision. Deficiencies in the structuring of the operation enhanced the risk to realise transition impact although some important risk mitigating factors were in place. Implementation was deficient, resulting in a high risk of loss for the Bank. Prospects for recovery of the Bank’s investment exist.

Unsatisfactory Appraisal was clearly deficient and there is evidence that important issues were not addressed. It did not provide an adequate basis to make a sound investment decision. A flawed structuring of the operation was an important reason for the complete failure of the project. Transition impact could not be realised. Implementation was deficient resulting in a high chance for the Bank to lose all its investment. Some prospects for recovery of part of the Bank’s money still exist.


Appraisal was clearly deficient and there is evidence that important issues were not addressed. It did not provide an adequate basis to make a sound investment decision. A flawed structuring of the operation was an important reason for the complete failure of the project. Transition impact could not be realised. Implementation was deficient and was partly the cause for losing the entire investment in the operation. No prospects for recovery of part of the Bank’s money exist.

6 Appraisal refers to all handling practices relevant to the pre-approval phase: project and sponsor selection,

project design, due diligence, financial analysis, market analysis, risk analysis, etc. 7 Implementation refers to all handling practices relevant to the post-approval phase: implementation,

documentation and security, syndication, disbursement, monitoring, problem recognition, remedial management, and recovery.

Documents

Annual Evaluation Overview Report for 2010RETIREMENT EVALUATION OVERVIEW REPORT FOR 2011 ii 7. CHIEF EVALUATOR’S ASSESSMENT 51 7.1 Performance of EBRD activities based on evaluation