Annual Evaluation Overview Report 2004

May 2004

ab0cdProject Evaluation Department

DOCUMENT OF THE EUROPEAN BANK FOR

RECONSTRUCTION AND DEVELOPMENT

ANNUAL EVALUATION OVERVIEW REPORT FOR 2004

PROJECT EVALUATION DEPARTMENT

MAY 2004

For Official Use Only

This document has a restricted distribution and may be used by the recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without the Bank's authorisation.

ANNUAL EVALUATION OVERVIEW REPORT 2004

TABLE OF CONTENTS

Page LIST OF ABBREVIATIONS iii EXECUTIVE SUMMARY v 1. PERFORMANCE OF INVESTMENT OPERATIONS ASSESSED AGAINST THE BANK'S MANDATE 1 1.1 Introduction 1 1.2 Performance indicators 1 1.3 Overall Performance Ratings 3 1.4 Transition impact and environmental performance & change 5 1.5 Financial performance of projects 7 1.6 Enhancing the Bank’s additionality in projects 8 1.7 Conclusions based on evaluated projects in 2003 9 2. INSTITUTION BUILDING ACHIEVEMENTS THROUGH PROJECTS IN

CENTRAL ASIA 9 2.1 The context of institutional reforms 9

2.2 Projects as instruments of institutional change 10 2.3 Experience from recent evaluation in Central Asia 10 2.4 Recommendations in respect of institutional change 14 3. KEY FACTORS OF HIGHLY SUCCESSFUL AND UNSUCCESSFUL PROJECTS 15

3.1 Introduction 15 3.2 Main categories of factors affecting performance 16 3.3 Relative importance of determining factors for project performance 16 3.4 Factors more specific to failure/success 18 3.5 Conclusions and recommendations 18 3.6 Best practice and possible starting points as preconditions for appraisal of investment opportunities 19

4. EVALUATION OF TECHNICAL COOPERATION (TC) OPERATIONS 21 4.1 TC Evaluation Coverage 21 4.2 Performance evaluation of TC operations 22 4.3 TC-Related Evaluation Work in 2003 23

5. RECOMMENDATIONS IN EVALUATION REPORTS PREPARED UNDER PED’S WORK PROGRAMME OF 2003 25 5.1 Recommendations from the evaluation special study on the TurnAround Management (TAM) Programme 25 5.2 Recommendations in respect of the evaluation special study on the Russian Small Business Fund (RSBF) Programme 26 5.3 Recommendations from other projects evaluated in 2003 27 6. FINDINGS FROM THE COUNTRY STRATEGY EVALUATION EXERCISE ON THE SLOVAK REPUBLIC 30

6.1 Introduction 30 6.2 Methodology 30 6.3 Scope of the evaluation 31 6.4 Overall assessment 31 6.5 Recommendations for future strategies 32

i

ANNUAL EVALUATION OVERVIEW REPORT 2004

ii

7. REVIEW OF FOLLOW-UP ON RECOMMENDATIONS OF SELECTED OPER REPORTS

AND FROM PREVIOUS AEORS 33 7.1 Recommendations to improve Environmental monitoring 33 7.2 TIMS performance review 33

8. AUDIT COMMITTEE’S REVIEW OF EVALUATION REPORTS 37 8.1 Evaluation reports discussed 37 8.2 Review of recommendations as presented in the minutes of the above-mentioned meetings of the Audit Committee 37 8.3 Other activities in respect of highlighting recommendations from evaluation reports 39 List of appendices

PROJECT EVALUATION DEPARTMENT

ABBREVIATIONS AND DEFINED TERMS AEOR Annual Evaluation Overview Report CEB Countries of central and eastern Europe and the Baltic states CIS Commonwealth of Independent States CHP Combined heat and power CSU Consultancy Services Unit CO2 Carbon Dioxide DTM Deal Tracking Module EAP Environmental action plan EC European Commission EET Energy efficiency team EIRR Economic internal rates of return EMS Environmental management system ESCO Energy service companies ETC Early transition countries EU European Union FDI Foreign direct investment FI Financial institutions business group FIRR Financial internal rates of return FSU Former Soviet Union GDP Gross domestic product GEF Global environmental facility IEA International Energy Agency (of OECD) IFC International Finance Corporation IFI International financial institution IFRS International Financial Reporting Standards issued by the International Accounting Standards Board ISO International Organisation for Standardisation JI Joint Implementation (under Kyoto Protocol) KTC Kazakhtelecom LC Letter of credit LLD Lessons Learned Database MDB Multilateral development bank MEI Municipal and Environmental Infrastructure Team MIS Management information system MoU Memorandum of Understanding MPF Multi-project facility OCE Office of the Chief Economist OCU Official Cofinancing Unit OECD Organisation for Economic Cooperation and Development OGC Office of the General Counsel OM Operations Manual OPER Operation Performance Evaluation Review PB Participating bank PCR Project Completion Report PED Project Evaluation Department PMM Project Monitoring Module

iii

PROJECT EVALUATION DEPARTMENT

iv

PPF Post-privatisation fund RO Resident Office (of EBRD in a country of operation) RSBF Russia Small Business Fund SEE Countries of south east Europe SME Small and medium-sized enterprises Tacis EC Assistance Programme for eastern Europe, the Caucasus and Central Asia TAM Turnaround Management Programme TC Technical Cooperation TCFP Technical Cooperation Funds Programme TI Transition Impact TIMS Transition Impact Monitoring System TMG TurnAround Management Group TOR Terms of Reference UNG National Corporation for Oil and Gas Industry XMR Expanded Monitoring Report (investment operations) XMRA XMR Assessment CAPEX capital expenditure Ex-ante before project signing at project appraisal Ex-post after project signing at post-evaluation

EXECUTIVE SUMMARY OF THE ANNUAL EVALUATION OVERVIEW REPORT FOR 2004

1. PROJECT EVALUATION’S FUNCTION The EBRD’s Project Evaluation Department (PED) synthesises its overall findings, including the Bank’s performance relating to its mandate, in this Annual Evaluation Overview Report (AEOR), thereby complying with its accountability obligations towards the Board of Directors and management. PED also contributes to preserving the corporate memory of the Bank, collecting “lessons learned” during evaluation of projects and carrying out special studies. 2. INVESTMENT PERFORMANCE JUDGED AGAINST THE BANK’S MANDATE Performance ratings of investment operations. The EBRD has continued to select and develop projects that largely meet the Bank’s mandate. During 1996-2003, only 54 per cent of the EBRD’s evaluated projects achieved Successful-Highly Successful overall performance ratings, mainly reflecting the modest financial performance of the projects. By contrast, the Bank’s projects scored much higher on transition impact (TI), with 75 per cent achieving a Satisfactory-Excellent rating for 2000-03. The fact that 26 per cent of the evaluated projects were given a disappointing Negative-Marginal rating reflects the fact that the Bank operates in difficult environments where many obstacles to transition remain. Further analysis of Unsuccessful projects in general industry and the banking sector indicated that poor corporate governance and management skills were recurrent factors, despite the Bank's efforts. Performance of country groups. South-eastern Europe continued to show a relatively stronger performance compared with other regions, in particular during the period 1997-2003. It is now getting closer to Central Europe and the Baltic states whose performance has deteriorated, partly because of lower verified additionality (whereby it does not replace private sector financing). It is also noticeable that operations in the early transition countries (ETCs) recorded lower performance ratings than in other regions. The seven ETCs have suffered from political instability and face difficult transition challenges, mainly due to the small size of domestic and export markets, underdeveloped financial systems and public governance issues. There are, however, some encouraging signs of improved performance of evaluated projects in ETCs in recent years. Performance of sectors. The large disparity in performance between different industry sectors, seen in earlier AEORs, appears to be decreasing. Infrastructure operations still receive better overall success ratings, in comparison with projects in financial institutions and general industry. However, these differences are not as marked as in previous years. 2. INSTITUTION BUILDING ACHIEVEMENTS THROUGH PROJECTS IN CENTRAL ASIA The Bank’s Transition Reports show that institutional reforms advance more slowly than the initial phase of trade and price reforms. The main reasons for this are the uncertainty concerning the outcome of the reforms, and the limited capacity within the state to implement them. While there are understandable reasons for the delays, the continuing lack of properly functioning institutions can create major business obstacles and reduce private investment opportunities. It is therefore essential that the project, being the main source of the Bank’s intervention, carries institutional changes among its transition objectives. The report presents the outcomes of several evaluated projects in Central Asia, in which important institutional development experience was gained. A number of more generic conclusions are drawn with the purpose of enhancing the Bank’s activities and improving project quality in early transition environments through: (a) better positioning of the project within other sector

activities; (b) improving the design of future projects through enhanced conditionality; (c) identifying relevant transition objectives; and (d) better identifying bottlenecks at sector level, especially in the area of institution building. 3. KEY FACTORS BEHIND HIGHLY SUCCESSFUL AND UNSUCCESSFUL PROJECTS PED has not to date conducted a formal statistical analysis of why projects either do very well or are unsuccessful. Short of relying on a formal list of factors, and matching their presence or absence in dedicated sections of evaluation reports on investment operations, the exercise undertaken and presented in this report is based on what has been de facto revealed in evaluation reports as the most recurrent causes of success and failures. The analysis focuses on recurrent performance factors of which three are internal to the project (financial, commercial, and institutional), and two others are external (effect of the business cycle and government). Bank handling of the process is added as a main factor both in project design and implementation. The main conclusion from this exercise is that the causes of success or failure are more internal to the project than external. This offers scope for the Bank to increase its likelihood of success and reduce the possibility of failure. The report presents recommendations and lists critical success factors for project preparation, which can be of use when preparing financing for business opportunities in early transition countries.

4. EVALUATION OF TECHNICAL COOPERATION (TC) OPERATIONS

TC activities facilitate the EBRD’s core investment operations and enhance the fulfilment of its transition impact mandate. In compliance with its responsibility towards the contributors to its Technical Cooperation Funds Programme, the Bank must focus equally on TC projects as it does on investments funded from the Bank’s own resources. Accordingly, TC projects are subject to a mandatory self-evaluation process, in the form of Project Completion Reports (PCRs), and to an independent evaluation process on a sample basis. Since 1993, when PED started TC evaluation work, it has conducted 46 OPERs and 19 special studies on sectors and themes, covering numerous TC operations. The total volume of evaluated TC operations based on an OPER report, as a percentage of the volume of TC operations with a completed PCR increased to 21.8 per cent in 2003, from 18.9 per cent in 1998, when the PCR review and assessment work was introduced. If groups of TC commitments covered in special studies are included, the coverage ratio rises to 57.2 per cent. The experience gained from TC evaluation shows that the Bank has been improving the preparation and use of TC donor funds. This can be attributed in part to an enhanced internal process of preparing, approving and monitoring TC operations. The report also outlines the findings of the latest special study on PCR assessments. 5. RECOMMENDATIONS FROM EVALUATION REPORTS Chapter 5 presents recommendations from evaluation special studies on the EBRD’s Turn Around Management (TAM) Programme, as well as on the Bank’s Russia Small Business Fund. The chapter also highlights the recommendations from OPERs on investment operations prepared under the 2003 Work Programme. In 2003 the Audit Committee began to review evaluation reports on investment operations and evaluation special studies on a more systemic basis. It was believed that by regularly discussing evaluation reports in the Audit Committee, more attention could be given to the follow-up of recommendations by management, and it would also strengthen the lessons

vi

vii

learned uptake on behalf of the Board. It would further strengthen the accountability function of evaluation and provide an important challenge for PED to secure a high quality for its evaluation products. Chapter 8 highlights specific recommendations that were selected by the Audit Committee during discussions in 2003 on five OPER reports on investment operations. Comments by management on these recommendations are also included. Chapter 7 reviews the follow-up on recommendations from selected OPER reports and from previous AEORs. The chapter revisits recommendations from previous AEORs on environmental monitoring and notes positive follow-up actions taken by management in the past years, as well as including an additional recommendation on environmental monitoring.

Having suggested the establishment of a transition impact monitoring system in the AEOR of 1999, the Transition Impact Monitoring System (TIMS) was approved in December 2002 and began to be implemented in January 2003. After just 15 months of implementation, it may seem premature to evaluate the performance of the new system, but it was nevertheless considered to be useful to assess the actions already taken, while the TIMS continues to expand and deepen on a daily basis. Based on participation in review meetings and the analysis of TIMS draft documents supporting the process, the preliminary evaluation findings of PED staff are positive. Already implemented to the maximum extent within the existing staff and time constraints of the Office of the Chief Economist (OCE), the process is well on its way to helping improve the structure of the projects (TI targets, covenants, risk mitigation) and addressing the problems as and when they arise, as well as raising the awareness of the Bank. To further improve the process, more time should be allocated by OCE staff to respond to the additional demands from TIMS and the Banking Department should give more emphasis to transition impact monitoring within this new framework. 6. FINDINGS FROM THE COUNTRY STRATEGY EVALUATION EXERCISE ON THE SLOVAK

REPUBLIC As highlighted in Chapter 6, the transition challenges facing the Slovak Republic have changed considerably over the past decade. They required continuous strategic monitoring and the establishment of new medium-term strategic directions by the EBRD. The strategies were in general well adapted to the way the business environment and private sector government policies evolved in the Slovak Republic. The strategic changes were also implemented in a timely way, as highlighted by evolving priorities in the portfolio for the Slovak Republic from one strategy to the next. The level of transition impact achieved, however, was mixed during the 1990s. The corporations that the Bank supported for privatisation did not benefit from a proper protective and preventive legal framework for corporate governance, solvency and bankruptcy before mid-2000. The banking sector was also restructured with minimal banking supervision regulation and other prudential safeguards. The coordination with other IFIs on policy dialogue and structural reforms was constrained by the political conditions in the country. The lessons learned from this evaluation range from strategy design to implementation and impact. They lead to the key recommendations that the strategy documents should be even more focused on priorities for immediate interventions, rather than longer-term intended initiatives, and that they should be better designed to effectively involve policy dialogue at sector and national levels, in coordination with other IFIs.

1. PERFORMANCE OF INVESTMENT OPERATIONS ASSESSED AGAINST

THE EBRD’S MANDATE 1.1 INTRODUCTION The EBRD’s Project Evaluation Department (PED) helps preserve the corporate memory of the Bank by evaluating projects, carrying out special studies and identifying “lessons learned”. PED also synthesises its overall findings - including the Bank’s performance on its mandate in this Annual Evaluation Overview Report (AEOR), thereby complying with its accountability obligations towards the Board of Directors and management. To ensure optimal use of this corporate memory, PED assists the banking teams and others during the early stages of project preparation to use the relevant “lessons learned” from past experience. This process ensures that past experience is applied to the selection and design of future projects. The experience gained from the Bank’s past performance and the generic lessons and recommendations presented in this report are thus available for the Bank’s future strategic orientation. During 2003 an update of the evaluation policy was prepared and presented to the Board at the beginning of 2004. In addition, decisions were taken by management regarding the independence of the evaluation function. The box below gives further details.

Evaluation Policy Review of 2004 and further enhancements of PED’s independence The Evaluation Policy Review of 2004 which was approved by the Board in February 2004 reviews the evaluation practices and procedures as they have evolved over the years in the Bank and presents the necessary updates. The report is based on: (1) the integration of experience gained by PED during the six years of project evaluation since the last update; (2) more than 10 years of the Bank’s learning experience; (3) changes in the Bank's modus operandi and its organisation; and, (4) enhanced harmonisation efforts in recent years with the other multilateral development banks (MDBs). Particular attention is given to: defining project evaluation distinctly from other functions, and establishing an approach to project evaluation which places critical importance on independent accountability and transparency and learning. The Review implements the good practice standards prepared by the MDBs’ evaluation cooperation group (ECG) on private sector evaluation and thereby contributes to the harmonisation process among MDBs. The independence of the evaluation function was further enhanced in 2003 through the implementation of a reporting structure in which the Corporate Director for Evaluation reports directly to the President while maintaining the reporting line to the Audit Committee of the Board of Directors. To maintain good contact with management on lessons learned the Secretary General is consulted by the Corporate Director for Evaluation on evaluation reports of projects and special studies. The Corporate Director also discusses with the Secretary General a draft of the AEOR and PED’s Work Programme before finalisation of these reports.

1.2 PERFORMANCE INDICATORS By the end of 2003, 508 investment projects of the Bank’s total portfolio of 1017 projects had reached a stage where they were ready for evaluation. Since the start of the Bank in 1991, PED has evaluated a total of 385 (or 76 per cent) of these investment projects. A well balanced sector and country coverage in the sample of evaluated projects has secured a broad representation of the overall portfolio of the Bank. Appendix 8, Section 10 gives further evidence of the size and representation of the sample of evaluated projects.

2

The evaluation performance indicators, which allow PED to assign the overall performance rating, are primarily based on the Bank's mandate to foster transition in its countries of operations. The relevant indicators consist of the following:

Evaluation performance indicators1

Mandate-related

indicators Sound banking

principle-related indicators

Bank effectiveness-related indicators

Performance on transition impact

Project and company financial

performance

The Bank's investment

performance

Environmental performance and

change

Fulfilment of project

objectives

Bank handling

The Bank's additionality

Overall performance rating

The indicator boxes that are presented above in blue make up the indicators that define “results on the ground” and as such make up the “transition outcome” rating.2 The evaluation of transition impact focuses on the broader effects that the project has on the sector and economy at large. Seven transition impact indicators, as used by the Bank during the screening and approval of projects, cover privatisation, competition, linkages to other sectors and skills transfer as well as the development of frameworks for markets, demonstration effects and corporate governance standards. PED assesses the short-term transition impact of a project that can be verified at the post-evaluation stage, as well as the longer-term transition impact potential that can still be realised. PED then reviews the risk of the project to realise its full transition potential. Appendix 7.1 presents the list of transition objectives that is used by PED and OCE when assessing transition impact ex-ante (before project signing) and ex-post (after project signing at post-evaluation). The transition matrices highlighted in Appendix 7.2 for each project evaluated in 2003 illustrate how PED deals with measuring ex-post realised transition impact, the longer-term transition potential still remaining and with the risk that full transition potential is realised during the life of the project. Appendix 8 gives details on the overall scores and the seven underlying performance rating categories for all evaluated projects. The evaluated operations referred to in this AEOR are, on the one hand, based on the post-evaluation of a sample of evaluated projects undergoing an operation performance evaluation

1 Details on EBRD’s Operation Performance Rating System at Post-Evaluation, with details on the benchmarks for each of the rating

criteria are presented in Appendix 1 of EBRD’s “Evaluation Policy Review of 2004”, which is posted on EBRD’s website: www.ebrd.com.

2 Presenting evaluation findings based on “results on the ground” , i.e. “transition outcome”, makes the findings more comparable with other multilateral development banks (MDBs). See further details in Appendix 8, section 1.

3

review (OPER) and, on the other hand, on the assessment of expanded monitoring reports (XMRs), the self-evaluation reports prepared by operational staff. The OPER exercises are normally undertaken by PED after the investment has been realised, i.e. one year (from 2004 onwards, 18 months) after last disbursement of the loan and two years after last disbursement of equity. Another stipulation is that at least one year of commercial operation must have occurred with one year of audited financial accounts.3 In 2003, a total of 18 projects ready for evaluation4 were selected for an OPER exercise, based on a purposive sample5 comprising 26 per cent of operations ready for evaluation. The XMR assessments (XMRAs) carried out by PED comprised a total of 30 projects (or 43.5 per cent of ready operations) and are selected on a random basis. Evaluation, therefore, covered a total of 69.5 per cent of projects ready for evaluation in 2003 (see Appendix 8, section 1). PED believes that the self-evaluation process, whereby operational staff prepare the XMRs, work well. This view is based on the quality of the XMRs prepared by the EBRD’s banking department on all projects ready for evaluation. To further enhance this process, PED has started to organise XMR training sessions for operational staff who are set to prepare XMRs in the near future. Once a project has been evaluated, either through an OPER exercise or through an XMR assessment, there is no regular re-assessment of a project’s performance in light of later events, although thematic and sector-oriented special studies may revisit some post-evaluated projects. A system of monitoring transition impact during project execution, recommended in AEORs of previous years and recently introduced by the office of the Chief Economist (OCE) is now in place and a process review of the transition impact monitoring System (TIMS) is presented in Section 7.3 of this report. 1.3 OVERALL PERFORMANCE RATINGS During 1996-2003, 54 per cent of evaluated operations were given Successful or Highly Successful ratings,6 as shown in Table 1.1 with the overall performance ratings. A positive development was evident in 2003 when the Successful and Highly Successful rated projects reached 60 per cent. Cumulatively the outcomes remained relatively stable at 54 per cent during this period, as in 2001 and 2002 a score below 50 per cent was observed. Eight per cent of the projects scored Highly Successful overall, while 17 per cent were rated Unsuccessful.7

54 per cent of evaluated operations were given Successful or Highly Successful ratings on overall performance for the period in 1996-2003.

The overall performance outcome may seem modest. However, many parts of the region in which the Bank operates remain risky from an investment perspective, and this continues to be true even for the more recent EBRD projects developed in advanced transition economies. In particular in these countries, where the Bank needs to secure its additionality in projects, it

3 Appendix 10 includes a flow chart of the evaluation process relating to the evaluation of investment operations. 4 In total, 19 OPER reports were produced. However, one of these was not an evaluation of a ready project but of the relationship

between EBRD and one of its major clients. Therefore it does not count towards the coverage ration of projects ready for evaluation. 5 Projects ready for evaluation on which an OPER report is prepared are selected on a purposive basis, i.e. projects are selected based on lessons-learned potential, risk for the Bank, a project’s high profile, etc. 6 Weighting by volume of investment yields better results with 64 per cent Successful or higher, 23 per cent Partly Successful, while the

Unsuccessful ratings share is 13 per cent. 7 While in the AEOR for 2001 no projects scored Highly Successful, in 2002 and 2003 6 per cent of the evaluated projects scored Highly

Successful. In addition, the number or projects in 2002 with an Unsuccessful rating dropped to 14 per cent compared with 22 per cent in 2001. This rating declined further in 2003 to 9 per cent.

4

means that it must sometimes accept relatively high-risk projects as sponsors for lower-risk projects in these countries can readily attract financing from the market. In recent years the EBRD has further deepened the transition process in more challenging areas where unemployment is high and financial intermediation limited, thereby accepting additional credit risk. The average score on transition impact Satisfactory to Excellent is much higher than the above score on the overall performance (see Section 1.4). The relatively lower average rating on financial performance partly drives the outcomes on overall performance (see Section 1.5). In addition, experience has shown that projects scoring Satisfactory on transition impact tend to be associated with a rating of Partly Successful in the overall rating.

Table 1.1: Overall performance ratings distribution of evaluated projects, 1996-2003

Year of evaluation

Unsuccessful 1

Partly Successful

2

Subtotal

1+2

Successful

3

Highly Successful

4

Subtotal

3+4

No. of evaluations

1996 15% 29% 44% 44% 12% 56% 34 1997 11% 42% 53% 36% 11% 47% 36 1998 22% 20% 43% 53% 4% 57% 49 1999 24% 22% 46% 46% 8% 54% 50 2000 19% 24% 43% 40% 17% 57% 42 2001 22% 31% 53% 47% 0% 47% 49 2002 14% 38% 52% 42% 6% 48% 50 2003 9% 31% 40% 54% 6% 60% 48

1996-97 13% 36% 49% 40% 11% 51% 70 1996-98 17% 29% 46% 45% 9% 54% 119 1996-99 19% 27% 46% 46% 8% 54% 169

1996-2000 19% 27% 46% 44% 10% 54% 211 1996-2001 20% 27% 47% 45% 8% 53% 260 1996-2002 19% 29% 48% 44% 8% 52% 310 1996-2003 17% 29% 46% 46% 8% 54% 358

Chart 1.1 below shows the breakdown of overall performance ratings by country groups, for all investment operations evaluated since 1993. South eastern Europe shows a relatively stronger performance compared with other regions, in particular during the period 1997-2003 (see also Appendix 8, section 2.3). It is getting closer to Central Europe and the Baltic states whose performance has deteriorated, partly because of lower verified additionality (see section 1.6)

Chart 1.1: Percentage distribution of overall performance ratings on 369 post-evaluated investment operations in 1993-2003 by country groups

10% 8% 9% 4%

49% 49%33%

38%

52%

24% 34%

31% 36%

36%

17%8%

27% 26%

8%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

CE Europe& Baltics

(177)

SE Europe(61)

Russia (64) ETCs (42) Other CIS(25)

Region

% o

f eva

luat

ed p

roje

cts Unsuccess-

ful

PartlySuccessful

Successful

HighlySuccessful

Note: 16 regional projects are omitted

5

Chart 1.1 also shows that operations in the early transition countries (ETCs)8, have lower performance ratings than in other regions. These seven countries have suffered from political instability and face difficult transition challenges, mainly due to the small size of domestic and export markets, underdeveloped financial systems and public governance issues. This very poor investment climate results in a low level of foreign investment and creates major challenges for EBRD investments. The relatively poor past performance of projects in early transition environments is discussed in Chapter 2 through selected case studies. The analysis in Appendix 8 shows, however, some encouraging signs of improved performance for ETCs in recent years. The above mentioned low average performance is mainly due to projects in the region evaluated before 2000, when only 21 per cent were rated Successful or better. Since then the overall performance of projects in the ETCs has risen to 52 per cent with these ratings, getting closer to the average across all regions. 1.4 TRANSITION IMPACT AND ENVIRONMENTAL PERFORMANCE & CHANGE 1.4.1 Performance on transition impact Table 1.2 presents the performance ratings on the transition impact, applying the six-point rating scale9 that was introduced in 1999. Of a total of 189 projects evaluated in 2000-03, 75 per cent achieved Satisfactory-Excellent ratings.

75 per cent of evaluated operations obtained Satisfactory-Excellent ratings on transition impact for the period 2000-2003.

This score is a very important accomplishment which confirms the Bank’s mandate compliance. However, 25 per cent of the evaluated projects obtained a rating of Negative-Marginal which shows that the Bank operates in difficult environments where many obstacles to transition remain.

Table 1.2: Transition impact of post evaluated projects in 2000-03

Evaluation Year/ Period

Negative 1

Unsatis- Factory

2

Marginal 3

Subtotal

1+2+3

Satisfactory

4

Good

5

Excellent 6

Subtotal

4+5+6

No. of evaluated projects

2000 5% 10% 7% 22% 24% 40% 14% 78% 42

2001 4% 14% 14% 32% 25% 43% 0% 68% 49

2002 2% 6% 16% 24% 26% 44% 6% 76% 50

2003 0% 4% 15% 19% 29% 48% 4% 81% 48

2000-01 4% 12% 12% 28% 23% 42% 7% 72% 91

2000-02 4% 10% 13% 27% 24% 43% 6% 73% 141

2000-2003 3% 8% 14% 25% 25% 44% 6% 75% 189

8 ETCs: Armenia, Azerbaijan, Georgia, the Kyrgyz Republic, Moldova, Tajikistan and Uzbekistan 9 In April 1999 the five-point rating scale (High, Medium, Low, None, Negative) on measuring transition impact was changed to a six-

point rating scale (Excellent, Good, Satisfactory, Marginal, Unsatisfactory, Negative). The same six-point rating scale is used ex-ante as well as ex-post (see Appendix 8, Section 3.2). As such, in 2000 PED started with a new time series in rating projects on transition impact. The trend observed over the past three years is comparable to the trend shown over the period 1996-1999 when the five point rating scale was used.

6

Chart 1.2 presents the transition impact (TI) rating distribution by country groups of 177 projects10 evaluated in 2000-03. Again, performance in the ETCs and other CIS countries (excluding Russia)11 lags behind other regions, though not by as much as for the overall performance rating. The time-sequence analysis in Appendix 8, Section 3.3 shows that transition impact in ETCs has remained steady with 65-68 per cent of projects rated Satisfactory or above for transition impact. This suggests that the projects introduced transition reforms, but that their sustainability and hence overall performance was affected by

e economic and political climate. The region with the highest transition impact ratings rem

Chart 1.2: Percentage distribution of transition impact ratings on 177 post-evaluated investment operations in 2000-03 by country groups

thains south-eastern Europe, as in last year’s AEOR.

4% 9% 12%

37%

56%42%

48% 39%

34%

22%

15%15% 22%

9%

9%

15% 22%28%

10% 9% 15%

6%

6%3%4% 6%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

CE Europe& Baltics

(67)

SE Europe(32)

Russia (33) ETCs (27) Other CIS(18)

Region

% o

f eva

luat

ed p

roje

cts Negative

Unsatis-factoryMarginal

Satis-factory

Good

Excellent

Note: Twelve regional operations are omitted.

Chart 1.3 shows assigned TI ratings by sector. It seems that differences in results between sectors, observed in past years, have become much less apparent. This implies a reduction over time in the TI ratings for operations in infrastructure and financial institutions, and an imp

Chart 1.3: Percentage distribution of transition impact ratings on 189 post-evaluated investment operations in 2000-03 by sector groups

rovement for operations in industry and commerce.

8%

46% 48% 39%

58%

28% 16% 26%

25%11% 26% 14%

10% 11%

3%3%17%

3%3%3%3%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

% o

f eva

luat

ed p

roje

cts

Negative

Unsatis-factoryMarginal

Satis-factory

Good

Excellent

Infrastructure

Financial Infrastructure Industry excl. Telecoms (12)Institutions (72) (31) Telecoms (74)

Sector

- Municipal, power, energy efficiency and transport, excluding shipping; Industry and commerce - agribusiness, general industry, natural resources, property/tourism and telecommunications

10 Twelve regional projects are omitted from the total of 189 evaluated projects for transition impact. 11 Other CIS countries, excluding Russia, are: Belarus, Kazakhstan, Turkmenistan and Ukraine.

7

1.4.2 Environmental performance and environmental change The EBRD was established with a specific environmental mandate. Article 2, clause (vii) of the Agreement Establishing the EBRD encourages the Bank “to promote in the full range of its activities environmentally sound and sustainable development”.

Environmental performance12, which is not rated during project appraisal, is included in the ex-post assessments. Environmental performance in 1996-2003 was rated Satisfactory or better in 80 per cent of cases. By major sectors, in terms of all past evaluated data, environmental performance was rated Satisfactory or better in 89 per cent of the cases for infrastructure, 78 per cent for industry and commerce (excluding telecoms) and 76 per cent for financial institutions (FI). Over the period 1996-2003, only 3 per cent of the projects evaluated have been rated Unsatisfactory in respect of environmental performance. Non-financial institutions tend to receive greater environmental focus during due diligence and result in better outcomes, even though these projects are held to a higher performance standard.13

Ioiawaee Pce 1 Tfo

1

1

1

1

1

80 per cent of evaluated operations obtained ratings of Satisfactory or better onenvironmental performance for the period 1996-2003.

n respect of environmental change14, 20 per cent of the evaluated projects rated Substantial r Outstanding, while 54 per cent achieved Some environmental change. The EBRD achieved ts most impressive results for environmental change in Category A15 projects (which on entry re judged to represent significant environmental liability), with 58 per cent of the projects ere being rated as Substantial or Outstanding. This implies that the EBRD is best able to

chieve its environmental mandate in projects that present the largest amount of nvironmental risk. Environmental change is a non-quantitative measure of the Bank’s nvironmental footprint.

ED considers these ratings to be positive, in particular, as the potential for environmental hange is not a consideration at the time of project selection. Further details on nvironmental performance and change are presented in Appendix 8, Section 7.

.5 FINANCIAL PERFORMANCE OF PROJECTS

he share of investment operations that achieved ratings of Good-Excellent on project inancial performance remained fairly stable and was at 56 per cent in 1993-2003.16 Almost ne-fifth of all project financial performance ratings, however, remain Unsatisfactory.

The project financial performance was rated Good or Excellent in 56 per cent of the cases for the period 1993-2003.

2 Environmental performance of projects is measured by accumulating the environmental and health and safety performance indicators:

environment being the status of the environment in the vicinity; health and safety: the way in which health and safety and respective risk assessment systems are effectively applied and the extent of compliance in this respect; pollution loads and energy efficiency: the extent to which the emissions are significantly lower that the regulatory limits; environmental management: the level of compliance with the agreed environmental action plan; public consultation and participation: whether the public consultation and participation has been carefully planned and organised with a responsible person in charge.

3 The performance standards for FI sub-projects are largely local laws while non-FI projects are measured against EU or WB standards. 4 The extent of environmental change (environmental impact) is measured as the difference between the environmental performance

before the project started and its performance at the time of evaluation. 5 The EBRD’s environmental categories are defined on the web: http://www.ebrd.com/about/policies/enviro/procedur/procedur.pdf.

Category A projects are those projects presenting the greatest environmental risk. 6 The project financial performance by volume of investments is Good-Excellent for 66 per cent of the evaluated projects.

8

Evaluation data support the fact that financially sustainable projects are more difficult to achieve in the early and intermediate transition countries, including in Russia, where the systemic constraints and risks are high. The percentage of Good and Excellent is as low as 29 per cent of the cases in the ETCs (Appendix 8, section 5), showing the need to focus attention on the systemic constraints at sector level that are likely to directly affect the financial performance of the project (see Chapter 2).

The risk was relatively high on average in general industry and in financial institutions, in particular in complex reform projects without a strong strategic sponsor. Positive financial performance, where the sustainability of a project is not threatened, is a necessary condition for transition impact to unfold, but it is not a sufficient condition as some financially successful projects can still score low on transition impact. Chapter 3 provides further analysis on factors affecting financial performance, as well as other aspects of project performance, for highly successful and unsuccessful projects. 1.6 ENHANCING THE BANK’S ADDITIONALITY IN PROJECTS Since the establishment of the EBRD, the Bank’s additionality in projects has been very good in the majority of cases.17 On a cumulative basis, from 1993 to 2003, 62 per cent of the projects evaluated rated the Bank’s additionality as Verified in All Respects, 28 per cent Largely Verified and only 10 per cent Verified in Part or Not Verified.

The Bank’s additionality in projects was rated Verified in All Respects and Largely Verified in 90 per cent of the cases for the period 1993-2003.

A closer analysis, however, shows deterioration over time, with the latter two ratings rising from below five per cent in the mid-1990s to 14 per cent in 2001, and as high as 24 per cent in 2002. While there was a slight improvement in 2003 (see Chart 1.6), the figures of recent years indicate that the Bank should improve its additionality in projects.

Chart 1.6: Ratings on the Bank’s additionality (1993-2003)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003

Year of evaluation

% o

f eva

luat

ed p

roje

cts Not

Verified

Verified inPart

Verified atLarge

Verified inAllRespects

17 The Bank’s additionality in projects is verified in terms of whether the Bank provides financing that could not be mobilised on the same terms by markets and/or whether the Bank can influence the design and functioning of a project to secure transition impact.

9

Further analysis in Section 4 of Appendix 8 shows that evaluated projects in central and eastern Europe and the Baltic states score amongst the lowest in respect of the Bank’s additionality in the portfolio. This reflects the difficulty of identifying projects in which the Bank is additional in the most advanced transition countries. 1.7 CONCLUSIONS BASED ON EVALUATED PROJECTS IN 2003

• In total 54 per cent of the evaluated projects in 1996-2003 achieved Successful-Highly Successful overall performance ratings mainly reflecting the more modest financial performance of projects. Of evaluated projects in 2000-03 a total of 75 per cent scored positively on transition impact. This positive outcome on transition impact leads PED to the conclusion that the Bank continues to maintain a portfolio which largely meets the Bank’s mandate.

• Regarding transition impact, the performance of the evaluated projects in the south-

eastern Europe (SEE) group of countries remains ahead of central eastern Europe and the Baltic states (CEB) and other groups, with a 65 per cent share of projects in the Good or Excellent bracket.

• Renewed attention should be given to additionality requirements in the design and

implementation of projects in the regions advanced in transition, so that high scores on additionality maintains the overall project performance at the desired level.

• Continued focus should also be given to project financial performance in the ETC

group to ensure that the current momentum of improvement in overall project performance is maintained. It is important to enable this group of countries to reach and maintain a level of performance comparable to the higher levels seen in other regions (see also Chapter 2).

2. INSTITUTION BUILDING ACHIEVEMENTS THROUGH PROJECTS IN

CENTRAL ASIA

2.1 THE CONTEXT OF INSTITUTIONAL REFORMS The EBRD’s Transition Reports distinguish between an initial phase of reforms which take priority in the early years of transition - mainly price and trade liberalisation - and a second phase which focuses on institutional reforms. Such reforms comprise competition policy and other regulatory policies, restructuring and privatisation of large-scale enterprises, the development of market based financial institutions, and reforms in the infrastructure sector. These reforms tend to be considerably more complex than initial reforms, require significant implementation capacity from the state, and can meet resistance from various interest groups.18 The Transition Reports show that institutional reforms advance more slowly than the initial phase of the trade and price reforms. The main reasons for this are the uncertainty concerning the outcome of the reforms and the limited capacity within the state to implement them.19 While there are understandable reasons for the delays, the continuing lack of properly

18 See Transition Report 2002, chapter 2. 19 See Transition Report 2001, chapter 2.

10

functioning institutions can create major business obstacles and reduce private investment opportunities. Institutional reforms in advanced countries are often prompted by external factors such as direct exposure to foreign markets or prospects of accession into the European Union. These factors are usually absent in early transition economies which have often remained inward oriented, relying on a few primary goods to generate export revenues. In addition, their domestic markets of early transition economies tend to be small, the local infrastructure is underdeveloped and there are often widespread deficiencies in governance, both public and private.20 This is why proper use of Bank’s instruments to successfully accomplish institution building in early transition economies has become so important. The projects should be carefully designed and selected to reflect the reality of weak institutions and poor business environment.21 2.2 PROJECTS AS INSTRUMENTS OF INSTITUTIONAL CHANGE The Bank’s mandate is to support the transition to a fully operating market economy in countries which are committed to the principles of Article 1 of the Agreement Establishing the Bank. At all stages of transition, one can find a need to build institutional components into projects. It is essential that the project, being the main source of the Bank’s intervention, carries institutional changes among its transition objectives. In this Chapter the outcomes of several evaluated projects in which important institutional development experience was gained are presented. A number of more generic conclusions will be drawn with the purpose of enhancing the Bank’s activities in early transition environments and improving project quality in that region. 2.3 EXPERIENCE FROM RECENT EVALUATIONS IN CENTRAL ASIA 2.3.1 Introduction. Among the evaluation exercises that PED has undertaken in early transition economies over the last two years, the following investment operations in early transition environments in Central Asia provide important lessons in respect of institution building:

• Uzbekistan - Refinery project rated Successful. The authorities were reluctant to change the organisational (hierarchical) structure and related vested interests in the sector. The technical cooperation funding helped to bring in international financial rules and standards (IFRS) and to expose the staff of the refinery to modern management practices, but only little of what was learned could be used, as the company was not ultimately privatised.

• Uzbekistan - Power project rated Partly Successful. The Bank financed a few

generators but added unrealistic institution building requirements. The leverage resulting from the amount of investment was not sufficient to trigger institutional changes.

• Kazakhstan – Telecoms project rated Successful. The authorities created an

independent regulatory body, they undertook to rebalance tariffs, but they did not privatise the company, contrary to expectations.

20 See ‘A revised Approach and Action Plan for Early Transition Countries, EBRD SGS04-48, 26 Feb 2004 page 4. 21 Same as above, page 10

11

• Kazakhstan - Aktau Port rated Partly Successful. The harbour administration was

expected to act as an autonomous entity and manage its assets according to commercial principles. The central government, however, imposed the subletting of lucrative petroleum loading facilities to another state entity on unfavourable terms. This policy endangers the port’s ability to self-finance necessary future improvement and extension work, and possibly its debt servicing to the Bank (from port revenues) as well.

2.3.2 Evaluation findings and lessons learned. For each of the above mentioned projects, PED has selected the main lesson on institution building (reported in this section), together with a more detailed description of project experience and highlights of key evaluation findings that leads to the lesson. LESSON 1: The choice of the project within the sector, while aiming to achieve commercialisation and transition targets, is very important and can contribute to the fulfilment of the key institutional development target. Simply participating in some relatively minor improvements of an outdated (old Russian design) refinery situated in the most difficult location - concerning feedstock sources and product sales - may not give the Bank much leverage regarding sector reform targets in a difficult low transition environment. The Bank’s investment in the natural resources sector in Uzbekistan took place in January 1997 in the form of a US$ 90 million loan to the National Corporation for Oil and Gas Industry (UNG), guaranteed by the Republic of Uzbekistan. In a parallel financing, JEXIM provided a similar loan amount for this project. These loans were used to install a desulphurisation unit (HDS plant) at the Fergana refinery to cope with high sulphur domestic oil, together with a number of other refinery modifications to improve the environmental and safety standard and increase the yield of the refinery. The project achieved some progress at the company level (reduction of environmental pollution, upgrade of technology, reduction of sulphur content, safety training and motivation of staff and management etc.) but the impact on the sector has proved to be limited and there is a high risk that the transition impact will not materialise due to the centralised decision making process.

Major evaluation finding from the refinery project: The evaluation of this project concluded that the commercialisation and privatisation covenant of the Bank was not well conceived. There are almost no circumstances where a stand alone refinery could be successfully commercialised and ultimately privatised, due to the inherent lack of interest from the global oil companies and the difficulty of structuring a refinery as a financially viable stand-alone project. This is particularly the case in remote frontier locations with difficult access to both feedstock and to key markets for petroleum products.

LESSON 2: The size of the project, illustrating to a degree the Bank’s commitment to the sector, needs to be in proportion to the envisaged transition impact targets. Specific targets must also be supported by a well structured technical cooperation component with timely delivery of the goods in close coordination with the project implementation schedule.

12

In Uzbekistan the Bank became the first international financial institution enter the power sector, financing a relatively small project (US$ 27.8 million) for the upgrading of two gas turbines within a large power plant with 10 installed units and a total capacity of 3,000 MW. The project consisted of two key components:

• A physical/construction component, including the rehabilitation/modernisation of low-pressure steam turbines, the rehabilitation of the water cooling systems, the installation of additional desalination units in the water treatment plant and the provision of engineering/consultancy services. These measures were designed to reduce specific fuel consumption and pollutant emissions, improve the performance of condensers, maintain plant output and improve the quality of raw/treated water.

• An institutional donor funded the development component, including a study on

electricity tariffs and a programme for introducing a modern accounting system and advanced financial management. This aspect had the aim of supporting reforms targeting corporatisation, commercialisation and market-based sector performance.

The project execution was hampered by delays in the delivery of the crucial technical cooperation components and disputes regarding the award of the contract to the preferred supplier. The project achieved progress at the company level but the impact on the sector was limited and there is a high risk attached to achieving the expected transition impact, given the still highly centralised political decision making process. This illustrates the difficulty of trying to change the sector from the bottom up when there is a lack of desire from the government to truly liberalise and commercialise the power sector.

•

•

Major evaluation findings from the power project: The technical cooperation operation and the advisory and engineering services provided by the procurement consultant helped to develop expertise and skills with respect to financing, accounting, project preparation, tendering, procurement, contract negotiations and project supervision. The corporatisation of the power station increased the scope for managerial autonomy and enhanced the awareness of financial constraints and commercial needs, but there is still a long way to go in transforming the power station into a cost-efficient and competitive generation company likely to attract private investors.

The sector is still run by a vertically integrated entity vested with monopolistic power (operating the transmission system and acting as a single buyer). The old management and other insiders remain in control of the sector’s assets and the government continues to interfere with tariff matters and investment decisions. And the regulatory authority that was established by presidential decree only has limitedresponsibilities confined to technical issues.

LESSON 3: When a project in the public sector aims at sector-wide institution building (in the form of privatisation, tariff reform and/or a new regulatory framework) involving the company as well as regulatory and higher ministerial authorities, precise project conditions should be agreed in advance with each relevant party. Covenants made with the company should be linked to conditions in an agreement with supervision and higher government authorities. Binding progress on privatisation and other reforms should set as a condition for loan disbursements. Enforcement rules should be fully built into both types of agreement with a realistic timing to help ensure that all concerned partners will join efforts to successfully implement the project.

13

In Kazakhstan the Bank focused its resources on the telecommunications sector with an integrated investment package comprising three components:

• Privatisation of Kazakhtelecom (KTC) through a US$ 70 million equity investment representing up to 30 per cent of the company’s capital to be sold later to a foreign investor. This was structured as portage equity with a put option to the government. The investment would enhance KTC’s commercial independence and corporate governance, and contribute to the transparency of the privatisation to a potential foreign investor.

• A component to physically upgrade Kazakhtelecom financed by a term loan to the

company for US$ 50 million to enhance earnings, competitiveness and service quality, and make the firm more attractive to a foreign investor; and

• An institutional development component. The technical cooperation programme

would focus on regulations, models for tariff rebalancing, and ways to provide a universal service.

The government eventually turned down the equity investment, despite the fact that it had clearly stated in a Memorandum of Understanding its intention to proceed with privatisation. In addition, the loan was not conditional on the successful completion of the equity investment and was disbursed after the equity deal had collapsed. Nevertheless, the project achieved some progress at the company and sector levels.

•

•

Major evaluation findings from the telecoms project: The corporate loan, without the benefit of the equity investment, mainly enhanced the physical infrastructure, but it also stimulated improvements to the financial management and reporting. However, staff levels and costs have subsequently grown faster than revenues and remain high by regional benchmarks. Most users that can afford the service can now access it, but customers and competitors have complained about monopolistic practices.

Privatisation to a foreign strategic investor was not achieved and Kazakhtelecom still dominates the market. The government has proposed restricting foreign ownership to 49 per cent of a telecommunications firm, which means that further privatisation of KTC, if it occurs, is unlikely to involve a foreign strategic investor. The government has signalled enhanced commitment to sector reform by establishing an independent regulator and announcing a liberalisation plan. The 2003 follow-on loan by the Bank, contains covenants to implement the technical cooperation-guided reforms, particularly on tariffs. Nevertheless, the state still has to implement them.

LESSON 4: In situations of unwanted government interference, the Bank must react speedily to avoid potential damage such as giving the appearance of endorsing the disruption. In April 1996 the Bank approved a sovereign loan equivalent of US$ 54 million for the Republic of Kazakhstan to reconstruct the Port of Aktau which was threatened from rising Caspian Sea levels. The project included:

• A physical component: the Bank’s project focused mainly on the dry cargo berth and associated facilities (renting out of port facilities being subject to the Bank’s approval).

14

• A technical cooperation component: Supervisory services relating to procurement and construction as well as to institution building for the port authority.

The physical investment objectives were fulfilled. The level of the general cargo berths was raised, and the workmanship implemented is regarded as high quality. The improvement works and the installation of modern equipment on the general cargo berths were carried out.

Major evaluation findings from the port project: The money spent through the technical cooperation component, was largely wasted due to the fact that the port management’s ability to act as a commercial-oriented entity was restricted by frequent government intervention. In addition, the progressive deterioration of the port’s finances has been exacerbated by the government imposing the leasing out of three out of the port’s four oil berths, the main revenue providers. This was done at the time without adequate compensation and without the Bank’s official consent, although the Bank is still seeking adequate remedy.

2.4 RECOMMENDATIONS IN RESPECT OF INSTITUTIONAL CHANGE In the cases described above, which all refer to different policy environments, the operations did not contribute to institution building as had been hoped, mainly because the government and the high level public administration authorities did not seem to be genuinely on board to enhance this aspect of transition impact. Possible actions of the Bank to enhance the realisation of the transition impact potential could be: •

•

•

•

•

•

Make it clear that the reforms will not go forward if the government and senior civil servants are not committed, or do not have the capacity to implement the reforms.22 Find ways to ensure commitment by the government and the civil servants who need to be involved in project implementation (linked to Lessons 3 and 4 above). Indicate more clearly where the responsibility lies, so that the government can alter its approach (Lesson 3) and secure adequate conditionality upfront. The delivery of the Bank's entire package of engagements from a country strategy should be linked to progress on key institution building objectives in the relevant sectors (Lesson 3). Make signing of any project in the country subject to progress on these key reform initiatives within agreed timetables (Lessons 3 and 4). The Bank’s conditionalities must be progressive from one investment to the next, and each time be realistically anchored to the level of commitment and the capacity of the government to implement reforms (Lessons 2, 3 and 4). Country strategies should provide a full and realistic description of the business and policy environment, and on that basis, give clear guidance on how to structure projects in order to maximise transition impact (Lesson 4).

More specifically, recommendations for future projects would be the following:

• Provide a better positioning of the project within other sector activities23 and identify the next steps to be taken within a broader institutional building programme;

22 or the reforms will not be accepted by the public, as they may lead to job losses and higher unemployment. 23 Sector event lines are stylised sequences of changes that are observed as sectors move from a situation of being centrally planned to a state whereby the market is fully functioning.

15

• Improve design of future projects by: (a) stating conditionalities to be fulfilled at inception of the project through specific covenants and Memorandum of Understanding (MoU); (b) selecting the exact relevant transition objectives and monitoring them throughout the life of the project;24 and (c) better identifying bottlenecks at sector level, especially in the area of institution building (governance) and policies, and, for example, price reforms (see Lessons 1, 2, 3).

• Enhance policy dialogue at high levels of decision making. This will facilitate, among other things, the establishment of more binding MoUs (Lesson 3).

• Seek support from other IFIs, and if the opportunity arises, to build alliances to improve the Bank’s policy leverage with the aim of triggering institution building. Other IFI loans may contain covenants and commitments to institutional reforms in a sector. Coordinated actions with the other IFIs and EBRD investments linked to the same type of reforms (privatisation for example) would have a greater impact on transition than separate initiatives (Lessons 3 and 4).

3. KEY FACTORS BEHIND HIGHLY SUCCESSFUL AND UNSUCCESSFUL

PROJECTS25 3.1 INTRODUCTION As illustrated in Chapter 1, the evaluation of project performance is broken down into financial performance, company performance, transition impact, environmental performance, application of sound banking and additionality principles, and ‘Bank handling’. To make a judgement on the overall performance, the evaluation components are combined into an overall assessment. The factors affecting project performance, in both private sector projects and in corporatised state sector operations are present in every one of the above project components, but are not always systematically set out in OPER reports. This is because the focus of OPER reports is on the performance of each component, not necessarily on the factors affecting it. Statements on lessons of experience can benefit by being more explicit in terms of these factors, since they tend to pinpoint the causes of success or failure. PED has not to date conducted a formal statistical analysis of why projects do very well or why projects are unsuccessful. Short of relying on a formal list of factors, and matching their presence or absence in dedicated sections of OPERs, the exercise undertaken to derive the findings in this chapter is based on what has been de facto revealed in OPERs as the most recurrent causes of success and failures. The analysis focuses on the 10 identified recurrent categories that could be regrouped into four more aggregated categories that are known from evaluation practice to affect performance. The scope of the analysis in this chapter covers two groups of projects situated at each end of the performance evaluation range: the highly successful and unsuccessful ones. Given the above mentioned difficulty of identifying factors in a systematic way, it seems sensible to use the extreme cases as they would logically offer a greater chance of narrowing down the factors affecting performance. Any conclusions could then be extended to the entire range of successful/unsuccessful projects. 24 This is all the more feasible now with the implementation of TIMS. 25 Of the 18 Highly Successful projects one was a state sector operation. Of the 34 Unsuccessful rated projects four projects were state sector operations. In addition the state sector operations were mostly run as separate corporations.

16

3.2 MAIN CATEGORIES OF FACTORS AFFECTING PERFORMANCE During the analysis, 10 recurring factors (called secondary factors) were identified from the ex post reviews of investment operations rated as Highly Successful and Unsuccessful. The factors were then classified into five broad categories, the so-called main factors (see Table 3.1). Three of the main performance factors are internal to the project: the financial, commercial, and institutional. The two others are external to the project: the effect of the business cycle and the role of government. Bank handling of the process is added as a main factor both in project design and implementation. Each of the secondary factors is further defined in Table 3.1, as is Bank handling.

Table 3.1: Recurrent factors affecting performance from OPERs

Main factors Secondary factors Further definition of secondary factors Financial analysis Quality of financial analysis as part of project appraisal and

during implementation Financial

Cost performance Actual costs versus expected costs at appraisal Sales performance Actual sales of the client company versus projected sales Market analysis Understanding of demand and competition, as part of project

analysis at appraisal

Commercial

Competitiveness On both quality and price of the product Sponsor commitment Local or international sponsor Management skills Including senior management skills, experience and

entrepreneurship

Institutional

Corporate governance Quality of corporate governance at Board level External Business cycle Includes financial crises Government behaviour Positive or negative government interference with client’s

implementation of the project Bank Handling EBRD’s management of the project at appraisal and

implementation, including quality of relations with the client The classification of the main factors has a number of limitations. The boundaries of each factor remain somewhat undefined and, as a consequence, there may be a degree of overlap. Some categories (commercial, institutional) may include more factors than others and therefore carry different weights in the aggregation of the results simply for this reason. Seeking greater separation and achieving a better balance in grouping factors into broader categories will be easier in the future when the exercise is applied to a larger number of highly successful and unsuccessful projects. 3.3 RELATIVE IMPORTANCE OF DETERMINING FACTORS FOR PROJECT PERFORMANCE Among the OPER reports issued to date, 18 are rated highly successful, and 34 are rated unsuccessful. As can be seen in Tables 3.2 and 3.3, in both successful and unsuccessful cases, there is a common series of factors causing either success or failure in the industry and finance sectors which are dominant in the Bank’s portfolio. (see Tables 3.2 and 3.3).

17

Table 3.2: Highly successful projects (% of factor occurrence)

Sector

Categories determining outcomes

Finance (7 projects)

General Industry

(4 projects)

Specialised Industry

(5 projects)

Average All Sectors

(18 projects) Financial Analysis 0% 25% 0% 6% Financial Factor

Cost Performance 14% 25% 60% 28%

Sales Performance 86% 75% 80% 78%

Market Analysis 86% 50% 40% 56%

Commercial Factor

Competitiveness 14% 50% 100% 44%

Sponsor Commitment 43% 75% 100% 61% Management Skills 86% 50% 60% 72%

Institutional Factor

Corporate Governance 0% 75% 40% 33%

Business cycle 0% 0% 0% 0% External Factor

Government behaviour 14% 50% 40% 28%

Bank handling 71% 100% 40% 72% Note: further details in Appendix 6.1 Note 1: The figures in the table represent the percentage of the projects in each sector for which the specified secondary factor was significant. For example, in one out of the four OPERs on General Industry operations (25 per cent), Cost Performance was a significant factor in the project’s success.

Note 2: The average of all sectors includes Finance sector, General Industry, Specialised Industry, Energy and the Infrastructure sector. Data on the latter two sectors are found in Appendix 6.1.

The percentage of occurrences in which the management factor is cited as a cause of success/failure in the projects evaluated is 72 per cent for highly successful projects and 71 per cent in the case of the unsuccessful ones. It is the highest occurrence in both cases. The degree of sponsor commitment and the quality (or lack of) corporate governance are also relatively high in both cases. The frequency of occurrence in the case of sponsor commitment is 61 per cent for highly successful projects and 38 per cent for unsuccessful projects. The percentages are reversed in case of governance. Government behaviour (positive or negative), although less frequent than the previously mentioned factors is still significant in the two groups: 28 per cent for highly successful projects and 24 per cent in the case of failed projects.

Table 3.3: Unsuccessful projects (% of unsuccessful occurrence)

Sector Categories determining outcomes

Finance (8 projects)

General Industry

(11 projects)

Specialised Industry (8 projects)

Average All Sectors

(34 projects) Financial Analysis 50% 36% 50% 47% Financial Factor

Cost Performance 13% 27% 50% 32%

Sales Performance 13% 64% 63% 41%

Market Analysis 25% 36% 50% 38%

Commercial Factor

Competitiveness 25% 36% 38% 29% Sponsor Commitment 13% 36% 75% 38%

Management Skills 75% 64% 63% 71% Institutional Factor

Corporate Governance 88% 64% 50% 59% Business cycle 50% 27% 13% 26% External Factor

Government behaviour 0% 27% 13% 24% Bank handling 25% 55% 38% 41% Note: see details in: Appendix 6.2

18

Note 1: The percentage in each sector is calculated as the ratio of the number of cases where the secondary factor appears to be significant over the total number of cases investigated in that particular sector (see numerical example in Note 1 of Table 2). The same method applies to the last column ‘all sectors’. Note 2: The average of all sectors includes Finance sector, General Industry, Specialised Industry, Energy and the Infrastructure sector. Data on the latter two sectors are found in Appendix 6.2. In terms of Corporate Governance, it should be noted that a well functioning Board of directors, with a committed sponsor, which (a) correctly sets the strategic priorities with a solid majority, and (b) is supported by a competent and motivated staff of middle and senior management, is more likely to result in a project being highly successful or unsuccessful if the above conditions are not met. 3.4 FACTORS MORE SPECIFIC TO FAILURE/SUCCESS Beyond the core of common elements leading to success and failure depending on how they apply, there are a number of specific characteristics that have been identified as leading to project success or failure. In industry and finance, we observe a heavy concentration of occurrences of very good marketing analysis (56 per cent) and sales performance (78 per cent) for projects to be highly successful (Table 2). Bank handling is also a more frequent contributing factor in case of success (72 per cent) as compared with failure (41 per cent). The key occurrences in project failure are more variable than those linked with project success (see Table 3). Beyond the core group of client management and governance previously identified, sales performance (lack of) has also an important role. But other factors appear relatively frequently that do not show so strongly in project success: financial analysis (47 per cent), cost performance (32 per cent) and business cycle (36 per cent). The frequency of occurrence of the above mentioned factors tends to indicate where the borderline is between highly successful and unsuccessful projects. The successful ones are free from public sector interference and are very focused on commercial performance. The latter are hampered by some form of government or public sector interference and cannot develop commercially as expected at appraisal. Although these factors are in principle well known and must be quite obvious to project teams, it may be that the teams do not automatically focus fully on them in project preparation and monitoring. 3.5 CONCLUSION AND RECOMMENDATIONS The main conclusion from this exercise is that the causes of success/failure are more internal to the project than external (business cycle, government). This offers scope for the Bank to increase its chances of success and reduce the possibilities of failure, and leads to the following recommendations:

• The Bank staff should try to get upfront agreements on the best enterprise Board majority configuration that would ensure the success of the project and company financial performance.

• Appointing adequate management and training of management possibly through a

well designed technical cooperation operation can be vital to help managers reach new markets and tackle new competition successfully.

• The exact role of the government, or senior decision makers in relevant ministries,

19

should be carefully reckoned with during project preparation, and in cases where the borderline between the public and private sectors is unclear, upfront agreements with major parties in a project or associated with a project, particularly in the industry sector, should be concluded.

• The Bank staff maintain their focus on market analysis and sales promotion in

industry projects as they often appear to be key elements of successful projects.

• Highly experience Bank staff should be used in the preparation of a complex project in industry and finance. Beyond the sophisticated work of project preparation, such staff should have a good grasp of what it really takes on the client side to expand markets, effectively manage corporations and implement a sound governance structure in uncertain business environments.

• In general industry, for successful as well as unsuccessful projects, sales

performance, i.e. actual sales of the client company versus projected sales, as well as corporate governance, i.e. quality of corporate governance at Board level, are critical factors. Therefore, these factors should get extra attention during project preparation.

3.6 BEST PRACTICE AND POSSIBLE STARTING POINTS AS PRECONDITIONS FOR APPRAISAL

OF INVESTMENT OPPORTUNITIES In addition to the above findings and taking into account the lessons learned presented in last year’s AEOR the box below gives a list of best practice situations and related starting points as preconditions for appraisal. The list refers to critical success factors for the Bank’s prospects particularly when preparing projects with local sponsors. The list can be of use when preparing financing for business opportunities in early transition countries.

CRITICAL SUCCESS FACTORS FOR PROJECT PREPARATION

1. Strong manager/management team Best practice: A strong management team with an insight into needs for reform, transparency and good corporate governance must be in place. Dominant managers from the past era should be avoided. Precondition for appraisal: The manager must be strong, experienced and dedicated but he might still work on completing or upgrading the existing management team. Good prospects of success must be present. Some management consultancy services might be required.

2. Strong line of business Best practice: The company must have a strong product range or deliver key services with obvious competitive advantage. Precondition for appraisal: The company must have at least one successful product on the basis of which a wider product range can be built or a package of services which with at least one dominant service which can be over time be expanded with other services.

3. Strong market position free from political intervention Best practice: The company must have a strong position within reasonably stable domestic markets, with no political intervention and low regulatory risk. Precondition for appraisal: The company must be familiar with the local market which might see considerable fluctuations over time. Sufficient risk mitigation measures should be in place to prevent political intervention (seek external advisory services or involvement of local investment forum, etc.). Regulatory risk should be compensated by reducing dependence on goods and service where such risk plays an important role.

4. Avoid foreign adventures Best practice: A company with established profitable exports may be seen as a strong investment, but investors should ensure that foreign international expansion strategies are avoided. Precondition for appraisal: Investors should be cautious regarding the development of new export markets. Avoid investing large sums and build up strong trade relations first. Expert advice through technical cooperation consultancy services might be required.

20

5. Financial strength of the sponsor Best practice: The sponsors should demonstrate financial strength. Precondition for appraisal: The sponsor must have shown in the past that it can manage a business and generate profit. It must have sufficient support from local banks as a result of several years of good relationships. Its financial condition should be balanced and it must have a good reputation in the local market.

6. The company should not be in an acute crisis, financial, environmental or any other Best practice: No acute crisis, financially, or pending environmental or other liabilities should exist, as this might jeopardize the project from the start. Precondition for appraisal: The company may be or have been in some sort of financial or other crisis, but this must be solved first before considering the investment. The sponsor must have adequate financial support locally (private or local bank) or other guarantees of some sort to prevent the crisis reappearing. Financial consultancy services from technical cooperation resources may help in solving such problems up front.

7. Restructuring challenges requires a strong strategic investor. Best practice: Restructuring challenges that lie ahead for the company should not involve fundamental enterprise and sector constraints demanding resources beyond reasonable reach without the involvement of a strong strategic investor. Precondition for appraisal: A local sponsor that has a proven track record in its line of business (production/marketing) might require technical advice from consultants to push through restructuring initiatives. Consultant support funded by technical cooperation is therefore required.

8. Adequate organisation and legal structure of the company Best practice: The due diligence must clarify that the company has a simple and transparent organisation. Unnecessarily complex structures should be avoided. Precondition for appraisal: Good prospects exist to make the organisation and legal structure of the company more transparent and easier to operate. Management should have taken steps that signal a strong commitment to simplifying the structure. Good legal advice through technical cooperation might be required.

9. The importance of shedding non-core and unprofitable businesses as part of restructuring exercise Best practice: Company management should give evidence that shedding of non-core and unprofitable lines of business is well underway. Precondition for appraisal: Management must have prepared a good plan to shed non-core and unprofitable lines of business. Agreement must be reached at the start on the timing of such activities. Consultant support is most likely required.

10. Manufacturing operations should have adequate structure and technology. Best practice: The structure and technology within the industry and returns to scale in marketing development and manufacturing should reasonably enable the local enterprise to co-exist and compete with other local, regional or global companies. Precondition for appraisal: The structure and technology in the industry and returns to scale in marketing development and manufacturing could have some flaws, but through a concise restructuring process with the necessary consultancy support, improvements should be possible within a reasonable amount of time. Technical cooperation support is required.

11. Information systems, accounts and audits at least approaching Western standards are essential. Best practice: Adequate monitoring should be allowed for. The company must have in place information systems, accounts and audits at least approaching Western standards. Precondition for appraisal: Unless the local sponsor is financially very strong, take a “corporate recovery focus” to project monitoring so that future problems can be spotted early on and action be taken immediately. Remain critical of the company’s financial statements even in the case of audited accounts.

21

4. EVALUATION OF TECHNICAL COOPERATION (TC) OPERATIONS

4.1 TC EVALUATION COVERAGE 4.1.1 Introduction TC activities are primarily used to facilitate the EBRD’s core investment operations and enhance the fulfilment of its transition impact mandate. In compliance with its responsibility towards the contributors to its Technical Cooperation Funds Programme (TCFP), the Bank is obliged to exercise the same attention for TC projects as it does for investments funded from the Bank’s own resources. Accordingly, on completion, TC projects are subject to a mandatory self-evaluation process, the results of which are documented in Project Completion Reports (PCRs), and to an independent evaluation process on a sample basis. Independent TC evaluation work falls broadly into two categories: (a) in-depth evaluations of individual or a group of TC operations in the form of an operation performance evaluation review (OPER) which involves a field visit and occasionally consultant input (from 2002 onwards, six TC OPERs per year); and (b) a desk-study-type PCR Assessment of 20 cases per year. Given their restricted scope and depth, the evaluation value of the PCR Assessment work is understandably less than can be expected from fully-fledged OPER work on TC operations.26 4.1.2 TC evaluation coverage by PED Since 1993, when PED started TC evaluation work, 46 OPERs and 19 special studies on sectors and themes have been carried out covering many TC operations. In addition, since 1998 a total of five PCR Reviews and Assessment synthesis exercises and one PCR Assessment synthesis study in 2003 have been completed.27 Overall, these reports, though very different in scope and evaluation focus, have covered over 1,000 TC-funded consultant assignments, involving approximately €280 million of funding from some 40 donor sources under the EBRD's TCFP. This represents about 32 per cent of total TC funding commitment or 27 per cent of cumulative TCFP funding mobilisation.28

The total volume of evaluated TC operations based on an OPER report, as a percentage of the volume of TC operations with a completed PCR (see Table 5.1) has increased from 18.9 per cent in 1998, when the PCR review and assessment work was introduced, to 21.8 per cent in 2003. If groups of TC commitments covered in special studies are included, the coverage ratio rises to 57.2 per cent.29

26 During the PCR Assessment process PED evaluates the project results and impact achievements (reliability and accuracy of report

content) in greater depth. See Appendix 10 for an overview of the evaluation system of TC operations. 27 Starting from 2003, OCU has taken on the PCR review function, concentrating on reviewing report quality and fulfilment of objectives. 28 As at 31 December 2003 the Bank was successful in mobilising €1,031.4 million TC funding of which €882.4 million or 85 per cent

was committed. 29 As PED concentrates on the evaluation of larger TC operations the total numbers of TC operations evaluated is relatively small. In

total 46 OPER reports on investment operations have been prepared since 1993, comprising 180 TC commitments. This amounts to 5 per cent of the Bank’s 3,450 TC commitments.

22

Table 5.1: Technical cooperation evaluation coverage status in 1991-2003 (€ million)

TC completion and PCR coverage

1991-1995

1991-1996

1991-1997

1991-1998

1991-1999

1991-2000

1991-2001

1991-2002

1991-2003

a. PCRs completed 73.1 107.5 134.0 188.8 236.5 302.8 364.8 424.4 491.8

b. TC operations evaluated through OPER reports

9.8 11.8 17.8 29.3 32.1 36.6 41.1 49.2 56.5

c. PCR assessments by PED - - - 3.4 8.7 13.9 18.9 26.1 31.1 d. PCR reviews by PED - - - 3.1 6.3 13.0 13.0 19.4 19.4 e. Total TC operations (b+c+d) 9.8 11.8 17.8 35.8 47.1 63.5 73.0 94.7 107.1

f. Evaluation coverage (b+c+d)/a (%)

13.5% 11.0% 13.3% 18.9% 19.9% 21.0% 20.0% 22.3% 21.8%

g. TC operations related to Evaluation Special Studies

22.4 24.7 93.3 93.3 93.3 106.6 122.3 160.0 174.2

h. Total TC operations evaluated (b+c+d+e)

32.2 36.5 111.1 129.1 140.4 170.1 195.3 254.7 281.3

i. Evaluation coverage (b+c+d+e)/a (%)

44.1% 33.9% 82.9% 68.4% 59.4% 56.2% 53.5% 60.0% 57.2%

Chart 5.1 presents the information from Table 5.1 in graphic form. It shows clearly the trends of TC evaluation coverage since 1995. It should be noted that PED, through the evaluation of TC-supported investment operations, provides further important assessments to TC donors.

Chart 5.1: Evaluation coverage of technical cooperation commitments for 1995-2003 (€ million)

0

100

200

300

400

500

600

1995 1996 1997 1998 1999 2000 2001 2002 2003Year (cumulative)

TC

com

mitm

ents

(ME

UR

O)

Completed TCs not covered by PEDSpecial StudiesCovered by PED (excluding Special Studies)

21.8%

57.2%

100%

With the selection of TC operations for evaluation, PED also takes into account TCFP funding sources, sector distribution of evaluation work in general and lessons learned potential of TC operations. Appendix 9 highlights the contributions of donors to TC operations that have been evaluated by PED through an OPER exercise. It shows that most of the countries with relatively high contributions to the Bank’s TCFP are adequately represented. 4.2 PERFORMANCE EVALUATION OF TC OPERATIONS TC operations do not lend themselves to aggregation of overall evaluation outcomes in the same way as investment operations. This is because PED selects TC operations with which an OPER report will be produced on the basis of size and lessons learned relevance, which differs from the methodology for selecting investment operations for evaluation. PED’s TC evaluation experience, nevertheless leads to the conclusion that the Bank has improved the preparation of TC operations. In recent years, the Bank has been paying greater attention to the preparation and use of TC donor funds. This can be attributed in part to the TC Review Committee that reviews and approves all acceptable TC funding requests. In the Committee,

23

which is chaired by the Vice President of Risk Management, all relevant Departments participate. In addition, the contribution of the Official Co-financing Unit (OCU) to the quality of TC operations in general should be mentioned. This has been possible through the strengthening of the monitoring and self-evaluation capacity of the Unit. In addition, the assistance provided by the Consultancy Services Unit (CSU) in terms of reviewing the terms of reference and consultant section has enhanced quality at entry. Recommendation: Management could organise additional staff training in the areas of preparation, implementation and monitoring of TC operations thereby helping Bank staff to further enhance the quality of TC operations. In this respect the dissemination of lessons learned on TC operations by OCU and PED can contribute to this training process. 4.3 TC-RELATED EVALUATION WORK IN 2003 4.3.1 TC OPERs In 2003 six TC projects were subjected to an in-depth OPER exercise and received the following overall ratings: • Institutional development and financial management of Baku port (Azerbaijan):

Successful • Norsi oil refinery TC studies (Russia): Partly Successful • Environmental support to a general purpose credit line to Budapest Bank (Hungary):

Partly Successful • Technical assistance to Uzbekneftegas (Uzbekistan): Partly Successful • Scoping study for railway restructuring project (Bosnia and Herzegovina): Successful • Azeri multi bank framework financing facility (Azerbaijan): Partly Successful These TC operations were funded by two donor contributions from Japan, one from Sweden and one from the UK, as well as two operations by EC Tacis and three operations by EC Phare. These operations were approved between 1994 and 2002 and cover the following sectors: chemical manufacturing and processing, finance-insurance, oil and gas production, ports and inland waterways, and transport services. By TC type, they involved advisory services, project implementation, project preparation and training. They were linked to investment operations representing €114.4 million.

4.3.2 Special study on PCR assessments PED carried out a special study on the Project Completion Reports (PCRs), the self-evaluation reports developed for TC operations. The assessment looked into a population of 129 PCRs submitted by Operations Leaders (OL) from the Banking Department (BD) over the past year from which 22 TCs were included in a sample for in-depth assessment. Each TC project was evaluated in the areas of Bank performance, client commitment, consultant performance, TC contribution and Bank investment, and transition impact. In general, the Banking Department has evaluated the TC projects well, with PED agreeing with 82 per cent of the overall ratings given. Most of the 129 PCRs submitted by the OLs during the past year recorded either a successful or highly successful performance (70 per cent). In these operations, client commitment, consulting work and the Bank’s performance is largely rated Good or Excellent. In the sample of 22 TCs, PED downgraded/upgraded three TCs rated Successful or Highly Successful, yet still kept them within the successful range.

24

Less successful TC projects in the overall population show weaknesses in the same areas. In the sample of 22 projects, PED downgraded one less successful TC from Partly Successful to Unsuccessful, which failed on all three counts, i.e., client commitment, consultant performance, and the Bank’s performance. The relatively poor results of the less successful TCs - that simultaneously exhibited a good evaluation of client commitment, consultant performance and bank performance (26 per cent) - seem to be largely driven by an unfavourable economic environment and/or by political interference. While these factors are often external factors, they have to be considered carefully in TC design and monitoring. The evaluation found that 65 per cent of the sample of 22 TC projects designed to generate EBRD investment actually led to an investment. All of those had an overall TC rating of Successful or above, and had Excellent or Good client commitment and consultant performance. OL interviews led to the conclusion that these follow-on investments would not have gone ahead without the TCs. The balance, about 35 per cent, did not lead to an investment because of a combination of factors, including insufficient client commitment, poor consultant performance, political interference, and a generally weak ‘enabling environment’. In 72 per cent of the sample of 22 TCs, transition impact (TI) was rated Medium or High, which coincided with a high client commitment and consultant performance. Cases where TI is rated High are mainly those where it is in the immediate client’s interest to work closely with the consultant. Those cases with low transition impact also tend to be characterised by low TC success overall. Interviews with OLs revealed deficiencies in the lesson learned dissemination process, but also interesting comments on how to practically incorporate the lessons learned into bank practices. A few patterns emerged concerning how/why Clients/Consultants fail to show commitment/deliver results, and why Bank preparation and monitoring sometimes fail. A table in Section III of Appendix 4 shows the lessons learned and gives examples of best practice to deal with those issues.

25

5. RECOMMENDATIONS IN EVALUATION REPORTS PREPARED UNDER PED’S 2003WORK PROGRAMME

In this chapter the key recommendations are presented from evaluation special studies and OPER reports on investment operations prepared under PED’s 2003Work Programme. It is hoped that this presentation of the recommendations together in one chapter will assist management with monitoring follow-up of recommendations. 5.1 RECOMMENDATIONS FROM THE EVALUATION SPECIAL STUDY ON THE TURNAROUND

MANAGEMENT (TAM) PROGRAMME The TAM Programme which is managed by the Bank through the TurnAround Management Group (TMG), seeks “to enhance the knowledge, confidence and capabilities of the senior management of potentially viable enterprises in Central and Eastern Europe and the Former Soviet Union and to assist them to lead these enterprises into sustainable economic success in a market oriented economy and, where appropriate, in their transition to privatisation.” The evaluation special study captures the operations of the TAM programme in the period 1998-2003. The TAM Programme has been highly successful. Nearly all the companies visited have acted on TAM’s advice and have made significant changes in their businesses. Nevertheless, TAM is totally dependent on donor funding, and the unreliability of this funding threatens TAM’s sustainability, constrains its ability to meet the demand for its services, and reduces its efficiency. Recommendations:

•

•

•

•

•

•

•

TMG should focus its efforts primarily on companies in early to intermediate transition countries, companies located outside the capital cities, companies entirely owned by local investors, and smaller enterprises. TMG and the Bank’s Official Cofinancing Unit, with active support from the Bank’s senior Management, should jointly continue to give top priority to arranging a three- to five-year multi-donor facility that would provide funding usable for a wide range of countries and not tied to the use of advisors from specific donor countries.30 The Bank should consider sharing in the costs of the TAM Programme as part of its normal operating budget. TMG should develop a framework for continuing to provide substantial subsidies (albeit not necessarily at the current level) for first assignments for small, locally owned companies and for phasing out these subsidies for follow-up assignments, assignments for larger companies (or companies owned by large companies) and companies with foreign shareholders. TMG should review annually the standard project budgets and allowances for TMG’s administrative costs. TMG should consider its Supervisory Board as a source of advice and guidance that can contribute to improving TMG’s operations and should seek to use Board meetings as a forum for discussing policy and strategic issues. TMG to date has used its Supervisory Board meetings essentially to provide information and respond to questions. It should seek to use these meetings rather as a source of advice and guidance. The supervisory Board may wish to discuss, among other things, the relevant recommendations of this special study. The EBRD should appoint a senior executive, other than a member of TMG’s management team, as chairman of TMG’s Supervisory Board.

30 Funds required for TAM activities in respect of ETCs would be funded under the ETC Initiative.

26

•

•

•

TMG should ensure that environmental issues are systematically addressed by its advisory teams. Although the Evaluation Team does not dispute the findings of TMG’s evaluation efforts, it nevertheless has several recommendations for improvement. Although the recent reintegration of TMG into the Banking Vice-Presidency may contribute to improved collaboration between TAM and the Bank, further steps are needed.

5.2 RECOMMENDATIONS REGARDING THE EVALUATION SPECIAL STUDY ON THE

RUSSIA SMALL BUSINESS FUND (RSBF) PROGRAMME An evaluation special study was carried out in 2003 on the RSBF Programme in Russia. The Special Study identifies a number of key issues and proposes recommendations aimed at helping the Programme achieve its intended objectives and maximise transition impact. The principal issues and recommendations are the following: 5.2.1 Key challenges: • reducing dependency on a single consultant; • exploring changes to the current methodology and the potential for additional distribution

channels; • embedding MSE lending in downscaling banks, and encouraging counterpart finance; • securing additional donor funding for TC. The Special Study recommends a reappraisal and restatement of the programme’s strategic purpose and the development of an action plan to achieve the strategic goals. The action plan should include targets for achieving institution building goals in each downscaling bank, in addition to loan numbers and volumes. 5.2.2 Recommendation on programme management

It is recommended that the EBRD reviews the range of tasks and functions currently outsourced to consultants in order to identify those tasks and functions that could be returned directly to the Bank’s control. The Bank should also consider a full review of the management resources allocated to the programme. It is important to note that a member of the Bank’s staff has recently been assigned to coordinate policy dialogue relating to the MSE sector in Russia.

5.2.3 Recommendation on monitoring of use of grant funds for technical cooperation The absence of records of time spent by consultants with individual banks and branches is a deficiency in management information that hinders comparison of the performance of individual banks and branches within banks, as well as comparison of the performance of individual consultants. As such, the Bank is recommended to encourage consultants to record the time spent with RSBF partner banks so that the EBRD can enhance performance monitoring (of consultants and partner banks). It is noted that, whether one consulting firm is working with several banks or branches, or whether more than one firm is involved, the monitoring of consultant time input is essential for performance assessment and costing purposes.

27

5.2.4 Recommendation on diversifying methodologies and improving systems •

•

•

•

The programme’s lending methodology should be reviewed with a view to incorporating changes to attract a greater proportion of sound borrowers engaged in production activities. The EBRD should continue dialogue with NGOs operating in Russia to enable the Bank to identify reliable NGO partners with whom it may be possible to work to expand the range of distribution channels for the programme. The participating banks (PBs) within the programme should be permitted to tailor MSE loan products to meet the requirements of borrowers and the local competitive environment, while continuing to observe sound banking principles. The RSBF should, instead of imposing software on PBs, define the reporting requirements of programme management that PBs will need to satisfy. Necessary additional technical assistance should be offered to PBs on a cost-sharing basis to enable them to integrate RSBF reporting with their existing systems or to select a suitable alternative MIS product. It is noted that this may, given time, lessen the degree of dependency on a single consulting firm and facilitate the introduction of other suitable consultants when identified.

5.2.5 Recommendation on commercial sustainability To date, the performance of the downscaling programmes in PBs has been measured primarily in terms of loan numbers and volumes. Attention should be given to assessing the profitability of MSE lending operations in PBs on a full cost basis taking account of the cost of TC inputs. The scope of TC should be extended to include targeted technical assistance to enable PBs to cost out their product lines. The introduction of cost sharing with PBs should be considered for targeted assistance of this nature. To assist in gauging PB progress towards sustainability of MSE lending, pro forma financials should be developed to track the earnings of downscaling banks on their RSBF portfolios, offset by the full cost of the in-kind subsidy represented by TC including consultant input. 5.3 RECOMMENDATIONS FROM OTHER PROJECTS EVALUATED IN 2003 5.3.1 Evaluation of Sakhalin II, Phase 1 (a) In 2003, PED completed an evaluation of Sakhalin II Phase 1, which highlighted the

following key recommendations: • to encourage the development of SMEs in Sakhalin as subcontractors of large Russian,

foreign and joint venture contractors. • to arrange technical cooperation to SMEs for the development of business plans and

implementation support combined with the extension of micro-finance should be part of this plan to foster backward local linkages and sustainable development.

• to provide strong support to the local administration for an integrated sustainable development plan and the establishment of an appropriate development foundation/entity should become a focal point for multilateral development bank involvement.

(b) Drawing from the lessons of Sakhalin II, Phase 1 project, important environmental

issues for Phase 2 include: • need to resolve the on-going lack of an appropriate SEIC solid and hazardous waste

plan for its operations on the Island; and resolve warehousing concerns as identified by the lenders’ environmental auditors;

• establish an ecosystem monitoring programme (key indicator species) to be carried out over the life of the project so as to monitor ecosystem health and possible changes;

28

• develop a single standardised annual reporting format acceptable to the lenders, the project sponsor, and government authorities.

(c) Broader issues for future Sakhalin Island development to be addressed by all

stakeholders include: • the need to consider cumulative environmental and social impacts of all oil and gas

projects on the Island; particularly as they might affect the Okhotsk-Korean grey whales, migratory fish and migratory birds;

• establishing a Public Information Centre where all relevant environmental and social documents are available for review by interested stakeholders;

• providing institutional support to local and regional authorities to carry out environmental, health and safety, and social impacts monitoring;

• providing assistance to the government to address Tier II and III oil spill response capacity, improved navigation aids etc, to reduce the risk of a tanker related spills which are beyond the company’s responsibility.

• giving assistance in setting up a fund for future generations and investments into the sustainable sectors of the Island's economy beyond the lifetime of the oil and gas projects is important for Russia as a whole, and as a topic for the future dialogue between the Bank and the Russian authorities.

• helping with a co-ordinated effort to find complementary investment opportunities should be encouraged in an appropriate way within the operating objectives of the Bank as well as other IFIs and donors, given the dramatic development opportunities and challenges facing this frontier region.

5.3.2 Recommendations from other projects evaluated in 2003 are the following: • The Banking/Environmental team should review in cooperation with Ispat-Sidex and the

current local government to potentially develop a Phase II environmental action plan (EAP) that will continue to move the company towards achieving full EU IPPC compliance (in the context of EU accession), while recognising the limitations imposed by the plant’s existing operations.

• Implement a Corporate Governance Monitoring System when making equity

investments. Corporate governance through equity investments requires clear and measurable objectives. Based on the evaluation of the Bank’s investment in the heavy industrial firm Raba in Hungary, it was concluded that the Bank should consider implementing a Corporate Governance Monitoring System, run by the Office of the Chief Economist within its Transition Impact Monitoring System (TIMS) process, to assess and monitor governance performance of the Bank’s equity investments.

• Financing IFRS Compliance Investment Projects. Based on the evaluation of the

fabricated metal product manufacturing company Skoda Kovarny in the Czech Republic PED concluded that the Bank should consider using corporate loans to finance investment in IFRS compliance by industrial groups. Achieving compliance with IFRS requires investment in accounting skills and systems and motivates strategic business changes and even disposals as the firm attempts achieve reliable and presentable accounts. IFRS compliance projects may be large and long-term in nature, but they are as worthy to finance as any investment in real assets. Investing in IFRS compliance requires substantial resources applied in discrete project steps that offer clear benchmarks for progress payments and strong conditionality. Successful IFRS compliance increases the likelihood

29

of business success, thereby improving cash flow and decreasing credit risk, and will therefore attract resources to the firm from suppliers, banks and capital markets.

• The Bank, being a development finance institution, should focus more attention on

the provision of a universal service for telecommunications operations. The World Bank group, as with many bilateral development agencies, is relatively active in the area of universal services by contributing to the development of cost effective delivery mechanisms. The Bank should include a statement regarding its position on universal services in its next update of the Telecommunications, Informatics and Media Operations Policy and may wish to consider trustee roles in universal service funds to ensure transparency and arms-length dealing on behalf of the government. The Bank may also focus some TC to strengthen the regulator in particular regarding strong universal service funds. This recommendation is based on the evaluation of Mobifon, a private mobile telephone operator in Romania and on other evaluated telecommunications projects.

•

•

Centralising and deepening risk analysis and management of a multi-project client to complement centralised banking management. The evaluation of the a commercial bank with extensive interests in the bank’s countries of operation showed the importance of centralising and deepening risk analysis and management of a multi-project client within the Bank. Taking this commercial bank as an example, the rapid regional growth of this client warrants a more centralized risk analysis and management of the EBRD’s exceptionally large exposure to this bank. There is a need for a consolidated risk management approach to large multi-project clients that would ensure that at least one credit manager has a clear and consistent view of the rapidly growing and changing situation. This would complement Financial Institutions Team’s (FI) relationship management approach. Total exposure to the group should be reviewed annually through an in-depth annual review (framework) that is independent of a given project approval. The supporting credit analysis (furnished by FI but vetted by Risk Management) should go beyond what is available in credit agency analysis to include analytical accounting analysis of key figures (including “double leverage”31), stress testing, and management analysis. In particular, the analysis should show how the insights gathered through monitoring the projects affect the overall assessment of the group. Management assessment should benefit from annual credit calling at the headquarters f the client and the network by members of EBRD senior management. Analysis of the structure of liabilities should cover the degree of the client’s reliance on official external finance, and the outlook for credit agency rating changes under stress scenarios. The annual Framework review should assess the adequacy of global consolidated regulatory supervision of the client as part of future due diligence. The “legwork” would be done mainly by FI supported by more centralised risk management oversight.

Achieving adequate global consolidated regulatory supervision in situations of strategic institutional investors. The evaluation of the relationship with the aforementioned commercial bank also showed that the EBRD should consider how it could help regulatory authorities overcome obstacles to adequate consolidated regulatory supervision of internationally active financial institutions in the Region. The Bank could take a client-level approach to complement the systemic work done by the World Bank, the IMF and the EU. For example, EBRD could monitor official progress towards execution of a consolidated supervisory review of a client. Host country regulators will

31 Double leveraging occurs when a bank holding company borrows in the debt market and transfers the proceeds to a subsidiary bank in

the form of equity. How much of the equity in network units may be debt at the consolidated level?

30

require encouragement to overcome obstacles to having regular meetings, signing pragmatic MoUs among themselves, and other impediments to global consolidated supervision. Successful implementation of such a review could enhance transition impact and have spill-over effects on building, testing and demonstrating capacity to conduct consolidated supervision across the region according to the Basle Core Principles for Effective Banking Supervision.

• Achieving head office leadership by strategic financial institution clients of

environmental collaboration with EBRD. The evaluation of the relationship with this commercial bank also showed that the EBRD should continue to cooperate with clients such as this to sign a MoU between the EBRD and the client to ensure the client’s head office leadership of environmental collaboration with the EBRD. According to the MoU, the client would assume head office responsibility for ensuring compliance with applicable EBRD environmental policies, procedures and training requirements by all of its CEE network units.

6. FINDINGS FROM THE COUNTRY STRATEGY EVALUATION EXERCISE ON

THE SLOVAK REPUBLIC 6.1 Introduction As the evaluation of country strategies has become a major activity of evaluation departments in other MDBs and in view of the fact that the EBRD’s country strategies are an essential instrument for the Board to help steer the Bank’s activities at the country level, PED has been requested by the Board of Directors to carry out country strategy evaluation, for the moment on a pilot basis. This country strategy evaluation in respect of the Slovak Republic is the first and will be followed by two more studies, one on Croatia and one on Azerbaijan to complete the pilot phase. 6.2 METHODOLOGY According to the Approach Paper approved by Excom in September 200332, the preparation of a Country Strategy Evaluation mainly consists of reviewing a series of strategies over an extended period for a selected country, including (a) analysing changes in the business environment and the transition challenges that the strategies were facing; (b) assessing the relevance of the strategies in their response to perceived challenges; (c) reviewing the implementation of the strategies; and (d) assessing their impact on the transition towards a market economy. The first three steps (a), (b) and (c) are part of what is also called the ‘top-own approach’ which starts with the stated country economic and sector objectives of each strategy and analyses how they were translated into a portfolio of projects. On the other hand, the ‘bottom up approach’, essentially the above mentioned part (d) consists of assessing the extent to which the projects selected through the strategies generated transition impact, at sector level and national levels33. The latter involves a further review of the projects, technical cooperation programmes and policy dialogue that were implemented under the strategies.

32 See Memorandum from Secretary General to Directors dated 12 September 2003, reference SGS03-250. 33 Essentially by following the methodology of the Transition Impact Retrospective Report for the sector assessments, and then

assembling the sector results to make an overall assessment at national level (See Transition Impact Retrospective – How the EBRD has advanced transition, EBRD, April 2001, Chapter 4 and the Companion Paper, Section 4, transition impact summary, pages 97-109).

31

Both approaches are complementary in providing the overall assessment of the strategy and lessons learned. Since the EBRD shares with the other members of the international community transition and development objectives, the country strategy evaluation also explores the contribution of these members (IFIs, donors, NGOs) to the fulfilment of the shared objectives. The evaluation focuses on the quality of coordination of the EBRD with its development and transition partners and the comparative advantage applied among IFIs. 6.3 SCOPE OF THE EVALUATION The Country Strategy evaluation for the Slovak Republic reviews four country strategies covering the period 1994-2003. The first strategy was issued in 1994, the second in 1996, the third in 1999 and the fourth in 2002. The analysis focuses on the dynamics of the business environment, the transition challenges emerging over time, and the response of the country strategies to the challenges. The report also takes a longer-term view by making an assessment of the impact of these strategies on the transition to a market economy. Coordination with other IFIs and donors is reviewed since the Bank shares private sector development objectives with them. The report derives from the above analysis ratings and lessons for future strategies. 6.4 OVERALL ASSESSMENT The transition challenges facing the Slovak Republic have changed considerably over the last decade, requiring continuous strategic monitoring and new medium term strategic directions by the EBRD to keep diversifying investments across sectors. As noted in the 1996 strategy statement focusing strategies on the corporate sector first and then on finance is not the usual approach taken in most countries of operations - where the core of EBRD activities lies in both finance and corporate sectors at early stages. The strategies were, however, well adapted to the way the business environment and private sector government policies have evolved in the Slovak Republic. The strategic changes were implemented in a timely fashion, as highlighted by the evolving priorities in the portfolio for the Slovak Republic from one strategy to the next. Achieving transition impact was mixed, however, during the 1990s. The corporations that the Bank supported for privatisation did not benefit from a proper protective and preventive legal framework for corporate governance, solvency and bankruptcy before mid-2000. The banking sector was also restructured with very minimal banking supervision regulations and other prudential safeguards in place. The law on secured transactions, which was supported by the Bank, had a long gestation period, was approved late in the process (2002), but was praised as the best available to date and will certainly have an impact in the coming years. Although the rating of the transition impact appears to be only satisfactory for most of the period considered, the strategies were responsive to the challenges of their time, the implementation was almost immediate and the coordination with other IFIs - implicit or explicit - did lead to a division of labour according to the relative advantage of each one. But the overall effectiveness of this coordination in keeping a regular pace of reform over the 1990s was constrained by the political conditions in the country. In this context of a difficult policy environment at key times of the transition, the transition impact of EBRD operations has been only satisfactory, although recently some improvements have been more visible in the financial sector.

32

6.5 RECOMMENDATIONS FOR FUTURE STRATEGIES A country strategy evaluation is essentially a retrospective exercise for accountability and lessons learned purposes. Very often, the lessons learned can be extended beyond the immediate context in which they are derived, thereby making recommendations for the future. In addition, the review of past strategies identifies linkages between business environment, opportunities, challenges and Bank responses that can ultimately help in the design of future strategies. The lessons learned from this evaluation range from strategy design to implementation and impact. They lead to recommendations that the strategy documents should be even more focused on immediate interventions, better designed to effectively involve policy dialogue activities. The strategies should also facilitate international coordination where the Bank can make a better use of its comparative advantage in promoting commercial operations which remain profitable in the long term. 6.5.1 Strategy design. The strategies should clearly differentiate between what belongs to the long-term transition impact associated with a wide range of potential activities from that which belongs to medium-term (two-three years) requirements for the country to reach the next step of transition, and calling for a narrower range of immediate initiatives. The operational content of a country strategy should not be limited to a list of potential projects across all sectors. It should indicate for each sector relevant for the Bank (for instance, infrastructure, power & energy, financial sectors) how future projects, related technical cooperation and policy dialogue proposals support the strategic orientations and progress along the benchmarks of the sector time lines. This would imply even more team work in the Bank to prepare country strategies. At early stages of transition when markets do not function properly, and when the regulatory framework is weak, free stand-alone (non-project related) technical assistance should be provided to help promote reforms in infrastructure and related areas. At later stages, when the transition is advanced, the free technical assistance should be only targeted to activities that have yet to reach full market performance, and then phased out. 6.5.2 Strategy implementation. A good implementation of a strategy is demonstrated by an adequate timing of changes in portfolio allocations across sectors, as well as the magnitude of shifts of allocations from one sector to the next, in order to ensure that the priority sectors are well catered for. The absence of a clear and predictable legal framework as well as a tight coordination of legal transition assistance activities among IFIs during the implementation of Bank investment strategies can lead to a more fragile and uncertain transition impact than would have been otherwise.

33

6.5.3 Coordination with other IFIs and donors. Strategic alliances of the Bank with another IFI for joint equity investments in a company being privatised can lead to a long lasting transition impact, as they help to set and diffuse better business practice and governance standards. When the EBRD decides on a special cooperation with another donor and the donor is providing grants for new operations to help prepare the ground in a challenging business environment, the efficient use of the grants must be closely monitored to ensure the long-term viability and sustainability of the operations. Whether it is based on formal agreements with letter of understanding and shared programmes or not, cooperation between the EBRD and other IFIs appears to be viable and sustainable as long as each partner fully sees its relative advantage and lets the other operate where it performs best. 7. REVIEW OF FOLLOW-UP ON RECOMMENDATIONS OF SELECTED

OPER REPORTS AND FROM PREVIOUS AEORs 7.1 RECOMMENDATION TO IMPROVE ENVIRONMENTAL MONITORING In previous AEOR’s and in the May 2001 Special Study titled “Evaluation of Environmental Performance of the EBRD”, PED has called for increased project monitoring by the Environmental Department (ED), and has suggested that ED consider the use of local staff and consultants to implement monitoring at the Resident Office level. In 2003-04, ED was reorganised to form a four-person team which will concentrate, to a large extent, on project monitoring. ED has developed and is implementing an internal risk rating for triaging its monitoring function. In addition, ED is also considering the potential for local contracting to address environmental monitoring needs. These actions relating to enhancing the monitoring function in ED are positive developments. Recommendation: To further support the environmental monitoring effort, PED recommends that Management considers allocating ED adequate budget for travel and the hiring of local consultants to assist in monitoring. The current budgeting arrangement requires that ED seeks travel funding from the respective Banking department, which is limited and therefore constrains ED’s capacity to fully monitor projects for environmental performance. 7.2 TIMS PERFORMANCE REVIEW Having suggested the establishment of a transition impact monitoring system in the AEOR of 1999, the Transition Impact Monitoring System (TIMS) was approved in December 2002 and began to be implemented in January 2003. After only 15 months of implementation, it may seem premature to evaluate the performance of the new system34. It would seem to be useful, however, to assess the actions already taken, while the TIMS keeps expanding and deepening every day.

34 This evaluation focuses on the process of TIMS. It does not cover the rationale for the ratings assigned in TIMS individual project

reports. The latter will be done when an OPER is undertaken for a project.

34

The PED evaluation paid regular attention to the way the entire process has evolved since January 2003. The team participated in about 20 credit review meetings35 where economists from OCE were also present, had many separate discussions with OCE economists following these meetings, and handled regular contacts with the OCE Director for Project Design. The PED staff reviewed the documents supporting the process as they appeared: (a) the TIMS Ex Com paper of December 2002; (b) lists of OCE attendance at the Credit/Portfolio review meetings; (c) about 60 TIMS project reviews,36 (d) stated transition goals and monitoring in appraisal reports and transition impact statements in reports from the Project Monitoring Module (PMM); and (e) a first collection of the TIMS findings classified by sector and by region. 7.3.1 TIMS objectives and the related process The purpose of the TIMS exercise was well defined in the proposal to Ex Com of December 2002. Its aim was to strengthen the monitoring of the transition impact while the projects are being implemented. The monitoring system is essentially expected to meet three objectives: (a) improve the structure of the projects by fine tuning the balance between transition target, covenants and risk mitigating factors; (b) address transition impact problems as and when they arise; (c) provide a regular assessment of progress in achieving transition impact. The process established to meet the above objectives relies mainly upon the staff of the OCE, Credit/Portfolio Review Unit and the Banking Department. OCE economists are assigned to specific sectors and countries. The economists are to regularly follow the transition impact accomplishments at project level. If projects rated ex ante for transition impact did not include monitoring benchmarks at appraisal, the very first task for the economist is to agree with the banker on these benchmarks37. The Banking department is to share with OCE recent achievements or failures of the project’s stated objectives, and discuss with OCE further actions which will help accelerate transition, in the context of the PMM reporting. The Credit/Portfolio Unit provides the focus of the discussion by allowing economists to hear from the bankers in Credit review meetings. Economists may also follow up independently and directly with the project OL as necessary. Economists convey their findings in a TIMS review report which monitors the progress against at least two predefined transition objectives set at appraisal, and translates the outcome of the analysis into ratings. The monitoring reviews are to confirm or change the previous transition impact ratings. Summaries of these monitoring reports are then collated by sector or region to provide inputs for broader sector analysis. At a later stage the process is to generate summary tables using even broader sector (corporate, finance, infrastructure), and country categories (early and intermediate transition, advanced, Russia) for further synthetic analysis and recommendations. 35 PED attended about 10 Credit Review meetings with the presence of OCE early this year and another 10 in September. 36 This assessment is based on a PED review of about 60 compact TIMS project reports collated by sector and made available to PED in

September 2003. The compact version of the TIMS reports includes the most essential elements of the TIMS Review Checklist: Transition Impact potential, Transition Impact Risk, and Latest Review comments

37 This applies mostly to project signed between April 1999 and January 2003. Before April 1999, while the transition impact objectives were spelled out in project documents, the transition impact potential and attached risk were not rated.

35

7.3.2 Process implementation The implementation of the process started slowly and then picked up a solid momentum. The TIMS activities were added to existing ones in OCE without additional resources, other than a one year contract employee since extended. OCE economists had to devote significant additional time to the new task, while not compromising their commitments to existing duties. The coverage of projects increased progressively over the months as the staff assignment system firmed up.38 It has now reached 95 per cent of the projects eligible for monitoring, which are those projects rated and scheduled for credit review meeting during 2003. Economists attending meetings in Risk Management which cover primarily portfolio credit reviews and which also can address transition impact reduces additional burden on the Banking department imposed by the new TIMS activities. In this way duplication of meeting arrangements and loss of time can be avoided. Invitations for OCE to participate and nomination of OCE participants in credit meetings are timely executed through the close cooperation of the Credit/Portfolio Review and OCE staff. The reports are framed into a ‘TIMS Review Checklist’. The list includes seven items: the project ID, main transition objectives, main risks to transition, indicators to monitor transition impact, rationale for transition impact, review comments, and a summary table with ratings. These items offer a compact and clear background against which the facts from the review must be compared. It is noticeable, however, that project covenants related to transition impact are not required to be explicitly spelled out as a separate item in the checklist. This can at times blur the monitoring assessment, as one cannot easily follow up on transition risk mitigation through conditionalities, when they exist. The organisational and structural constraints currently attached to the process may have had an impact on the content of the TIMS' product. The credit reviews focus on credit worthiness and the PMM on project implementation. Increasingly, the economist complements findings from the Credit Review meeting and the PMM reports by sending a first draft of the TIMS review checklist separately to the Banking Department and following up directly with the OL to finalise the checklist.39 When the transition monitoring indicators have not been explicitly spelled out in project documents and the economist has agreed with the Banking staff ex post on what exactly should be monitored, the Banking staff and OCE are monitoring the project under the new benchmarks. There is some ongoing learning by both parties which tends to slow down the process. 7.3.3 Assessment Already at this early stage of implementation the project coverage is 85 per cent. It is sufficient to create a solid base for a transition impact diagnosis at sector or country level.40 The process is time consuming because it deals with complex, qualitative issues difficult to track and to handle. To date, the OCE staff, which is already fully occupied with other 38 From May to early September 2003, the average OCE attendance was already 45 per cent (50 OCE participations from a total of 112

Credit/portfolio meetings). 39 For example, Albania Emergency road and Rehabilitation II (OP 28092) – meeting of 14. October 2003. 40 The number of signed projects from April 1999 to August 2003 was 670, of which 394 were rated for transition impact. From this total

of 394, OCE selected 355 for evaluation. Among the 355, 337 TIMS evaluation were completed. The coverage ratio is calculated as (337/394)*100= 85 per cent.

36

assignments, has just been able to absorb this task. Nevertheless, with the growing dimension of the monitoring activities, the need to proceed to the next phase of mass handling of data, and above all the necessity to further deepen the process in order to increase its value added, more time should be allocated to OCE staff to make the exercise even more effective and sustainable. Meetings with the credit portfolio group are as effective as can be expected, given the focus on credit review. However, more time is required outside the meetings to help the Banking department in the field to fully devote attention to transition impact issues and problem solving in this area. The Banking Department does not yet appear to follow up on transition impact-related benchmarks and conditionalities, to the same extent as it does e.g. on covenants related to creditworthiness. When there is non compliance with a financial covenant (solvency, for example), the OL immediately follows up with the Credit/portfolio unit and often quickly finds remedial actions. When it comes to transition impact (for example, installation of a new accounting system in a company), the Banking Department does not seem to believe that OCE has a definite contribution to make to take remedial action, and reserves the right to decide on the time and magnitude of that action. In addition, the Banking Department which has been focussing more on transition impact in appraisal reports does not yet seem to extend its renewed attention to implementation so far41. This appears in many PMMs which remain too general on transition impact accomplishments, and has an impact on the effectiveness of the TIMS process. 7.3.4 Recommendations Already implemented to the maximum extent within the existing staff and time constraints, the process is well on its way to supporting the first two goals of TIMS which are to: (a) help improve the structure of the projects (TI targets, covenants, risk mitigation); and (b) address the problems as and when they arise and raise the awareness of the Bank. There are strong indications that the third goal (c) to improve measurement of transition impact at project level, is likely to be reached even sooner as the TIMS review of project check list appears to put a great deal of emphasis on this aspect. Given the above mentioned accomplishments, the following recommendations are only to help further improve a process which has already made a very encouraging start:

• Additional time should be allocated by OCE staff to respond to the additional demands from TIMS. This might have budgetary consequences.

• The TIMS review worksheet should include a separate section exclusively dedicated to the monitoring of project covenants, which are directly or indirectly related to the transition impact of the project;

• The Banking Department should give more emphasis to transition impact monitoring. In the PMMs the reporting requirements should be more developed and regularly updated and include the transition benchmarks.

• More time should be devoted to transition impact in the context of the meetings of the Credit Portfolio Group. Depending on the circumstances and the nature of the project,

41 Indices of slow internalization of the TIMS process by the banking department appear in the limited attention in the PMMs devoted to

the new transition impact target and monitoring requirements. The format current section on transition impact in the PMM may need modifications to help the banker to better reflect the more complete and detailed monitoring requirements. This would in turn facilitate the work of OCE.

37

this could be realised by organising a short meeting between OCE and Banking, just after the credit review meetings.

• TIMS review summary findings will be annually transmitted for discussion to ExCom level in order to increase institutional awareness across departments.

• Integration of TIMS in the credit review process, rather than a separately operated system, has some disadvantages: some projects are not monitored by OCE because either they are monitored via memo without meetings with Credit or because they are sovereign guaranteed and all disbursements have taken place.

8. AUDIT COMMITTEE’S REVIEW OF EVALUATION REPORTS 8.1 EVALUATION REPORTS DISCUSSED In 2003 the Audit Committee initiated a systematic reviewing of evaluation reports on investment operations and evaluation special studies. It was felt that by discussing evaluation reports in the Audit Committee regularly, more attention could be given to the follow-up of recommendations by management and it would also strengthen the lessons learned uptake on the side of the Board. It would further strengthen the accountability function of evaluation and provide an important challenge for the Project Evaluation Department to secure a high quality for its evaluation products. The following documents were reviewed in the following meetings of the Audit Committee: 17 October 2003 Operation Performance Evaluation Reviews on: - A shoe factory in Russia - A steel conglomerate in Russia 30 January 2004 Operation Performance Evaluation Reviews on: - A wastewater privatisation in Central Europe - A telecommunications project in Russia - A large steel factory in southern Europe 8.2 REVIEW OF RECOMMENDATIONS AS PRESENTED IN THE MINUTES OF THE ABOVE

MENTIONED MEETINGS OF THE AUDIT COMMITTEE Based on the recommendations in the above mentioned OPER reports and taking into account the content of the discussion in the Audit Committee of theses reports, the Committee made the following recommendations. The response by management which is presented after each of the recommendations was formulated by management after a respective request from the Committee had been received. AUDIT COMMITTEE’S FIRST RECOMMENDATION: Staff should be explicitly instructed not to give in to any political pressure, and to inform superiors when they think such pressure is exercised.42

42 No similar lesson or recommendation was part of the respective OPER report.

38

Response by management: All waiver requests require the independent sign-off of the Credit Department; this procedure is designed to prevent the exertion of undue pressure leading to inappropriate approvals. Where conditionality relevant to other departments is involved Credit refer any such waiver request to, and require sign off from, other departments (e.g. Environment, Procurement, Office of the Chief Economist) as appropriate. AUDIT COMMITTEE’S SECOND RECOMMENDATION: Waivers of project conditions should only be considered upon written and reasoned request. This requirement should be inserted into the Bank’s Operations Manual.43 Response by management: Following the Audit Committee discussion, this recommendation was discussed by PED, Office of the General Counsel, Banking and the Operations Administrative Unit and it has been agreed to apply this requirement to any material items which will require Board approval. In respect of waivers that by existing procedures are decided upon by the Operations Committee, the Committee may ask for a waiver request by the client in writing if it considers this is necessary. Any wider application was deemed to be impractical. AUDIT COMMITTEE’S THIRD RECOMMENDATION: Waivers of important conditions should be reported to the Board. When conditions have been crucial in Board acceptance, waiving requires prior approval of the Board.44 Response by management: A Board approval procedure for this is set out in “Material and Non-Material Change (Projects), Approval and Reporting Procedures (BDS96-157) and in the Bank’s Operations Manual. Any waiver regarded as fundamentally changing a project or its objectives requires Board approval. AUDIT COMMITTEE’S FOURTH RECOMMENDATION: Qualifications in Audit opinion, the absence of IFRS audited accounts, and measures taken to mitigate the consequence of such absence should be highlighted properly in project documents.45 Response by management: The Bank’s policy is to require three years’ audited IFRS accounts; in practice this may be waived in the early years of projects, particularly in the public sector. Accounts are often heavily qualified, particularly in early transition countries. Where these deficiencies give cause for concern, it is the responsibility of the Operations Committee and Operation Leader to ensure that suitable commentary is included in the Board document.

43 Related recommendation from the respective OPER report: The Bank should revise the Operations Manual 6.8.1 to require that a

request to approve waivers or amendments to project documents may be processed only in response to a duly-signed letter from the client requesting the change and explaining why the change is necessary.

44 No similar lesson or recommendation was part of the respective OPER report. 45 Related lesson and recommendation from the respective OPER reports: (a) A “qualified” audit opinion signals important

disagreements between auditor and client, and auditor doubts about the quality of the financial statements. Therefore, firms in the west go to great lengths to avoid issuing financial statements that carry a qualified opinion. The Bank should avoid issuing financial statements that carry a qualified opinion. The Bank should take a dim view of qualified audit opinions, disclose the fact of a qualified audit to the Board in the financial summary, dig into the reasons for qualification and quantify the risks to relying on the qualified statements; (b) The Bank should review the adequacy of its standard conditions precedent and disbursement controls to block disbursement when the Bank is in receipt of a qualified auditor’s opinion on the borrower or the sponsor. Waiver of receipt of a clean audit opinion should be subject to specific approval.

39

•

8.3 OTHER ACTIVITIES IN RESPECT OF HIGHLIGHTING RECOMMENDATIONS FROM EVALUATION REPORTS

Apart from discussions on recommendations from evaluation exercises in the Audit Committee, it is important to highlight the following related initiatives: • Presentation in the AEOR of all recommendations from OPER reports and evaluation

special studies. • Regular discussions by the Corporate Director for Evaluation with the First Vice

President on recommendations in evaluation reports to explain background of the recommendations and the urgency of actions to be taken.

• Discussions with respective Team Leaders and/or Business Group Directors on

recommendations and lessons learned presented in evaluation reports. The aim of the discussions is to identify relevance of the recommendations and the urgency of the actions to be taken.

• During the preparation of evaluation reports extensive contacts take place between

evaluation staff and staff involved in the operation process. These contacts are essential to secure the relevance and adequacy of recommendation.

Regular meetings are organised with the secretariat of the Operations Committee to discuss findings from evaluation reports including recommendations and lessons learned.

ANNUAL EVALUATION OVERVIEW REPORT 2004

LIST OF APPENDICES

Appendix 1 Basic data sheet: Operation Performance Ratings on the 19 OPERs and 30 XMR Assessments prepared in 2003 Appendix 2 Basic data sheet: Operation Performance Ratings on the 157 OPERs and 181 XMR Assessments prepared in 1993-2002 Appendix 3 1993-2003 Technical Cooperation Operation Performance Evaluation Review (OPER) Reports Appendix 4 Selected lessons learned from projects evaluated in 2003 Appendix 5 Investment operation database Appendix 6.1 Projects rated Highly Successful Appendix 6.2 Projects rated Unsuccessful Appendix 7.1 Assessment of strength of transition potential and checklist of transition indicators for ex-ante and ex-post application Appendix 7.2 Transition impact analysis of OPER reports on investment operations Appendix 8 Outcome of performance ratings of the Bank's investment operations Appendix 9 Table of TC funds Appendix 10 Evaluation of investment operations – The Process

Basic data sheet: Operation performance ratings on the 19 OPERs prepared in 2003

Operation Year of Board

Approval

Country Portfolio Class

Type1 Transition Impact2

Environmental Performance of

Sponsor and Bank3

Extent of Environmental

Change4

Overall Rating5

Project 1 2001 UKRAINE PRIVATE L Good Good Substantial SuccessfulProject 2 2000 BULGARIA PRIVATE L Good Good Some SuccessfulProject 3 1997 KYRGYZ REPUBLIC PRIVATE L Good Good Some SuccessfulProject 4 1997 ROMANIA PRIVATE L Good Good Some SuccessfulProject 5 1997 RUSSIAN FEDERATION PRIVATE L Good Satisfactory Substantial SuccessfulProject 6 2001 ROMANIA PRIVATE L Good Satisfactory Some SuccessfulProject 7 2000 SLOVAK REPUBLIC PRIVATE E Good Satisfactory Some SuccessfulProject 8 1992 <REGIONAL> PRIVATE E/L Good Marginal None SuccessfulProject 9 2000 RUSSIAN FEDERATION PRIVATE L Satisfactory Good Some SuccessfulProject 10 2002 RUSSIAN FEDERATION PRIVATE L Satisfactory Satisfactory None SuccessfulProject 11 1997 ESTONIA PRIVATE L Satisfactory Satisfactory Some SuccessfulProject 12 1999 KAZAKHSTAN PRIVATE L Satisfactory Satisfactory None SuccessfulProject 13 2000 FYR MACEDONIA PRIVATE L Marginal Excellent Substantial Partly SuccessfulProject 14 1997 UZBEKISTAN STATE L Marginal Good Some Partly SuccessfulProject 15 1999 GEORGIA PRIVATE L Marginal Good Some Partly SuccessfulProject 16 1998 HUNGARY PRIVATE L Marginal Satisfactory None Partly SuccessfulProject 17 1997 HUNGARY PRIVATE E Satisfactory Marginal None Partly SuccessfulProject 18 1995 CZECH REPUBLIC PRIVATE L Satisfactory Marginal Some UnsuccessfulProject 19 1996 UZBEKISTAN PRIVATE L Unsatisfactory Satisfactory None Unsuccessful1 E=Equity; L=Loan2 The range is Excellent/Good/Satisfactory/Marginal/Unsatisfactory/Negative for projeccts evaluated from 2000; High/Medium/Low/None/Negative before 20003 The range is Excellent/Good/Satisfactory/Marginal/Unsatisfactory/Highly Unsatisfactory4 The range is Outstanding/Substanital/Some/None/Negative5 The range is Highly Successful/Successful/Partly Successful/Unsuccessful A

ppendix 1Page 1 of 2

Basic data sheet: Operation performance ratings on the 30 XMR assessments prepared in 2003

Operation Year of Board

Approval

Country Portfolio Class

Type1 Transition Impact2

Environmental Performance of

Sponsor and Bank3

Extent of Environmental

Change4

Overall Rating5

Project 1 2001 BULGARIA PRIVATE L Excellent Excellent Some Highly SuccessfulProject 2 2001 RUSSIAN FEDERATION PRIVATE E/L Excellent Excellent None Highly SuccessfulProject 3 2001 CROATIA PRIVATE L Good Good Some Highly SuccessfulProject 4 2001 CROATIA PRIVATE L Good Excellent Substantial SuccessfulProject 5 2000 KYRGYZ REPUBLIC PRIVATE E/L Good Good Some SuccessfulProject 6 1996 ROMANIA PRIVATE E Good Good Some SuccessfulProject 7 2001 CROATIA PRIVATE L Good Good Some SuccessfulProject 8 1996 KYRGYZ REPUBLIC PRIVATE L Good Good Some SuccessfulProject 9 1996 CROATIA PRIVATE E Good Good Some SuccessfulProject 10 2000 RUSSIAN FEDERATION PRIVATE L Good Good Some SuccessfulProject 11 2000 RUSSIAN FEDERATION PRIVATE E Good Good None SuccessfulProject 12 1998 RUSSIAN FEDERATION PRIVATE L Good Good None SuccessfulProject 13 2001 SLOVENIA PRIVATE L Good Good None SuccessfulProject 14 2000 UKRAINE PRIVATE L Good Satisfactory Some SuccessfulProject 15 1998 GEORGIA PRIVATE L Good Satisfactory Some SuccessfulProject 16 2001 LATVIA PRIVATE E Good Marginal None SuccessfulProject 17 1999 LATVIA PRIVATE L Satisfactory Good Some SuccessfulProject 18 2000 CROATIA PRIVATE L Satisfactory Good Some SuccessfulProject 19 1997 RUSSIAN FEDERATION STATE L Good Good Substantial Partly SuccessfulProject 20 1999 BOSNIA AND HERZEGOVINA PRIVATE L Good Marginal Negative Partly SuccessfulProject 21 1997 <REGIONAL> PRIVATE E Satisfactory Excellent None Partly SuccessfulProject 22 1997 AZERBAIJAN STATE L Satisfactory Good Substantial Partly SuccessfulProject 23 1999 POLAND PRIVATE L Satisfactory Good Substantial Partly SuccessfulProject 24 1997 POLAND PRIVATE E Satisfactory Good Some Partly SuccessfulProject 25 1997 ALBANIA PRIVATE L Satisfactory Marginal None Partly SuccessfulProject 26 1996 UZBEKISTAN PRIVATE E Marginal Good Some Partly SuccessfulProject 27 1997 MOLDOVA STATE L Marginal Good Some Partly SuccessfulProject 28 2001 KAZAKHSTAN PRIVATE E Marginal Satisfactory Some Partly SuccessfulProject 29 1996 <REGIONAL> PRIVATE E Satisfactory Good Some UnsuccessfulProject 30 1999 RUSSIAN FEDERATION PRIVATE E Unsatisfactory NA None Unsuccessful1 E=Equity; L=Loan2 The range is Excellent/Good/Satisfactory/Marginal/Unsatisfactory/Negative for projeccts evaluated from 2000; High/Medium/Low/None/Negative before 20003 The range is Excellent/Good/Satisfactory/Marginal/Unsatisfactory/Highly Unsatisfactory4 The range is Outstanding/Substanital/Some/None/Negative5 The range is Highly Successful/Successful/Partly Successful/Unsuccessful

Appendix 1

Page 2 of 2

Basic data sheet: Operation performance ratings on the 157 OPERs prepared in 1993-2002

Operation Year of Board

Approval

Year of evaluation

Country Portfolio Class

Type1 Transition Impact2

Environmental Performance of

Sponsor and Bank3

Extent of Environmental

Change4

Overall Rating5

Project 1 1998 2000 CROATIA PRIVATE L Excellent Excellent Some Highly SuccessfulProject 2 1993 2000 SLOVENIA STATE L Excellent Excellent Some Highly SuccessfulProject 3 1997 2000 KAZAKHSTAN PRIVATE L Excellent Good Substantial Highly SuccessfulProject 4 2000 2002 RUSSIAN FEDERATION PRIVATE L Excellent Good Substantial Highly SuccessfulProject 5 1998 2000 CZECH REPUBLIC PRIVATE E Excellent Good Some Highly SuccessfulProject 6 1993 1996 HUNGARY PRIVATE E High Excellent Substantial Highly SuccessfulProject 7 1995 1999 LITHUANIA PRIVATE L High Excellent Some Highly SuccessfulProject 8 1994 1996 POLAND PRIVATE L High Satisfactory Some Highly SuccessfulProject 9 1992 1996 <REGIONAL> PRIVATE E High Satisfactory Some Highly SuccessfulProject 10 1993 1997 SLOVENIA PRIVATE E/L High Satisfactory Some Highly SuccessfulProject 11 1991 1997 HUNGARY PRIVATE E/L High Satisfactory Some Highly SuccessfulProject 12 1994 1999 RUSSIAN FEDERATION PRIVATE E/L High Satisfactory None Highly SuccessfulProject 13 1991 1993 CZECH REPUBLIC PRIVATE E High Highly SuccessfulProject 14 1993 1994 POLAND PRIVATE E High Highly SuccessfulProject 15 1993 1994 ROMANIA PRIVATE L High Highly SuccessfulProject 16 1993 1995 ROMANIA PRIVATE E High Highly SuccessfulProject 17 1994 1996 RUSSIAN FEDERATION PRIVATE L Medium Satisfactory Some Highly SuccessfulProject 18 1995 1997 CROATIA PRIVATE E/L Medium Satisfactory Some Highly SuccessfulProject 19 1997 2000 RUSSIAN FEDERATION PRIVATE L Excellent Excellent Some SuccessfulProject 20 1995 1998 UKRAINE PRIVATE L High Excellent Substantial SuccessfulProject 21 1997 1999 ESTONIA PRIVATE L High Excellent Some SuccessfulProject 22 1992 1997 LATVIA STATE L High Satisfactory Substantial SuccessfulProject 23 1994 1998 ROMANIA STATE L High Satisfactory Substantial SuccessfulProject 24 1995 1998 HUNGARY PRIVATE L High Satisfactory Substantial SuccessfulProject 25 1992 1996 UKRAINE PRIVATE E High Satisfactory Some SuccessfulProject 26 1994 1997 SLOVENIA PRIVATE E/L High Satisfactory Some SuccessfulProject 27 1993 1998 ROMANIA STATE L High Satisfactory Some SuccessfulProject 28 1995 1998 HUNGARY PRIVATE E High Satisfactory Some SuccessfulProject 29 1997 1999 BOSNIA AND HERZEGOVINA PRIVATE L High Satisfactory Some SuccessfulProject 30 1995 2001 RUSSIAN FEDERATION PRIVATE L Good Excellent Substantial SuccessfulProject 31 1995 2002 UKRAINE PRIVATE L Good Excellent Substantial SuccessfulProject 32 1996 2002 LATVIA STATE L Good Excellent Substantial SuccessfulProject 33 1999 2002 UKRAINE PRIVATE L Good Excellent Substantial SuccessfulProject 34 1997 2002 CZECH REPUBLIC PRIVATE E Good Excellent None SuccessfulProject 35 1996 2000 RUSSIAN FEDERATION PRIVATE E/L Good Good Substantial SuccessfulProject 36 1996 2000 <REGIONAL> PRIVATE E Good Good Substantial SuccessfulProject 37 1998 2001 SLOVAK REPUBLIC PRIVATE L Good Good Substantial SuccessfulProject 38 1999 2002 CROATIA STATE L Good Good Substantial SuccessfulProject 39 1997 2000 RUSSIAN FEDERATION PRIVATE E Good Good Some SuccessfulProject 40 1997 2000 BULGARIA PRIVATE E/L Good Good Some SuccessfulProject 41 1998 2001 KAZAKHSTAN PRIVATE L Good Good Some Successful

Appendix 2Page 1 of 9

Basic data sheet: Operation performance ratings on the 157 OPERs prepared in 1993-2002

Operation Year of Board

Approval

Year of evaluation

Country Portfolio Class

Type1 Transition Impact2

Environmental Performance of

Sponsor and Bank3

Extent of Environmental

Change4

Overall Rating5

Project 42 1998 2001 AZERBAIJAN PRIVATE L Good Good Some SuccessfulProject 43 1996 2001 FYR MACEDONIA PRIVATE E/L Good Good Some SuccessfulProject 44 1997 2001 ESTONIA STATE L Good Good Some SuccessfulProject 45 1998 2001 POLAND PRIVATE E Good Good Some SuccessfulProject 46 1999 2002 <REGIONAL> PRIVATE L Good Good Some SuccessfulProject 47 1994 2001 <REGIONAL> PRIVATE E Good Good None SuccessfulProject 48 1999 2002 <REGIONAL> PRIVATE E Good Good None SuccessfulProject 49 2001 2002 POLAND PRIVATE L Good Good None SuccessfulProject 50 1997 2000 ROMANIA PRIVATE L Good Good NA SuccessfulProject 51 1995 2002 RUSSIAN FEDERATION PRIVATE E Good Marginal Some SuccessfulProject 52 1994 1999 BELARUS STATE L Medium Excellent Some SuccessfulProject 53 1997 1999 BOSNIA AND HERZEGOVINA PRIVATE E Medium Excellent None SuccessfulProject 54 1993 1996 SLOVAK REPUBLIC PRIVATE L Medium Satisfactory Outstanding SuccessfulProject 55 1992 1996 ROMANIA PRIVATE L Medium Satisfactory Substantial SuccessfulProject 56 1992 1996 HUNGARY STATE L Medium Satisfactory Substantial SuccessfulProject 57 1994 1996 POLAND PRIVATE L Medium Satisfactory Substantial SuccessfulProject 58 1995 1998 KYRGYZ REPUBLIC PRIVATE E/L Medium Satisfactory Substantial SuccessfulProject 59 1995 1999 ESTONIA STATE L Medium Satisfactory Substantial SuccessfulProject 60 1996 1999 RUSSIAN FEDERATION PRIVATE L Medium Satisfactory Substantial SuccessfulProject 61 1994 1996 ESTONIA PRIVATE E/L Medium Satisfactory Some SuccessfulProject 62 1993 1997 UZBEKISTAN PRIVATE L Medium Satisfactory Some SuccessfulProject 63 1994 1997 POLAND PRIVATE L Medium Satisfactory Some SuccessfulProject 64 1993 1998 SLOVENIA STATE L Medium Satisfactory Some SuccessfulProject 65 1994 1998 ROMANIA PRIVATE L Medium Satisfactory Some SuccessfulProject 66 1994 1999 ESTONIA PRIVATE E/L Medium Satisfactory Some SuccessfulProject 67 1993 1996 LATVIA STATE L Medium Satisfactory None SuccessfulProject 68 1991 1993 HUNGARY PRIVATE L Medium SuccessfulProject 69 1992 1994 HUNGARY PRIVATE L Medium SuccessfulProject 70 1992 1994 HUNGARY PRIVATE L Medium SuccessfulProject 71 1992 1994 HUNGARY PRIVATE L Medium SuccessfulProject 72 1992 1994 POLAND PRIVATE L Medium SuccessfulProject 73 1993 1995 POLAND PRIVATE L Medium SuccessfulProject 74 1993 1995 CZECH REPUBLIC PRIVATE E Medium SuccessfulProject 75 1992 1995 HUNGARY PRIVATE L Medium SuccessfulProject 76 1992 1995 CZECH REPUBLIC PRIVATE L Medium SuccessfulProject 77 1992 1995 SLOVAK REPUBLIC PRIVATE L Medium SuccessfulProject 78 1993 2001 SLOVAK REPUBLIC PRIVATE E/L Satisfactory Good Outstanding SuccessfulProject 79 1997 2001 UZBEKISTAN STATE L Satisfactory Good Substantial SuccessfulProject 80 1996 1999 SLOVAK REPUBLIC PRIVATE L Low Satisfactory None SuccessfulProject 81 1993 1994 ESTONIA PRIVATE L Low SuccessfulProject 82 1996 1998 RUSSIAN FEDERATION PRIVATE L High Excellent Substantial Partly Successful

Appendix 2Page 2 of 9

Basic data sheet: Operation performance ratings on the 157 OPERs prepared in 1993-2002

Operation Year of Board

Approval

Year of evaluation

Country Portfolio Class

Type1 Transition Impact2

Environmental Performance of

Sponsor and Bank3

Extent of Environmental

Change4

Overall Rating5

Project 83 1995 1997 GEORGIA STATE L High Satisfactory Some Partly SuccessfulProject 84 1993 1995 RUSSIAN FEDERATION PRIVATE L High Partly SuccessfulProject 85 1992 2000 POLAND PRIVATE L Good Excellent Substantial Partly SuccessfulProject 86 1997 2001 GEORGIA PRIVATE E/L Good Good Substantial Partly SuccessfulProject 87 1997 2000 RUSSIAN FEDERATION STATE L Good Good None Partly SuccessfulProject 88 2000 2002 LITHUANIA PRIVATE E Good Good None Partly SuccessfulProject 89 1996 2002 GEORGIA PRIVATE L Good Marginal Some Partly SuccessfulProject 90 2000 2002 RUSSIAN FEDERATION PRIVATE L Good Marginal None Partly SuccessfulProject 91 1997 1999 POLAND PRIVATE L Medium Excellent Substantial Partly SuccessfulProject 92 1992 1996 POLAND PRIVATE E/L Medium Satisfactory Substantial Partly SuccessfulProject 93 1994 1999 AZERBAIJAN STATE L Medium Satisfactory Substantial Partly SuccessfulProject 94 1993 1996 HUNGARY STATE L Medium Satisfactory Some Partly SuccessfulProject 95 1992 1998 RUSSIAN FEDERATION PRIVATE E/L Medium Satisfactory Some Partly SuccessfulProject 96 1996 1999 MOLDOVA PRIVATE L Medium Satisfactory Some Partly SuccessfulProject 97 1992 1998 BULGARIA STATE L Medium Poor None Partly SuccessfulProject 98 1992 1993 POLAND PRIVATE L Medium Partly SuccessfulProject 99 1991 1995 <REGIONAL> PRIVATE E Medium Partly SuccessfulProject 100 1992 1995 RUSSIAN FEDERATION PRIVATE L Medium Partly SuccessfulProject 101 1992 1995 HUNGARY PRIVATE L Medium Partly SuccessfulProject 102 1996 2001 LATVIA PRIVATE L Satisfactory Excellent Some Partly SuccessfulProject 103 1995 2002 FYR MACEDONIA PRIVATE E/L Satisfactory Good Some Partly SuccessfulProject 104 1999 2002 CROATIA PRIVATE L Satisfactory Good Some Partly SuccessfulProject 105 1998 2002 RUSSIAN FEDERATION PRIVATE L Satisfactory Good None Partly SuccessfulProject 106 1998 2000 BULGARIA PRIVATE L Satisfactory Marginal Some Partly SuccessfulProject 107 1993 1998 POLAND PRIVATE L Low Excellent Substantial Partly SuccessfulProject 108 1994 1997 HUNGARY PRIVATE E/L Low Satisfactory None Partly SuccessfulProject 109 1993 1996 BULGARIA PRIVATE E Low Marginal Some Partly SuccessfulProject 110 1993 1997 RUSSIAN FEDERATION STATE L Low Marginal Some Partly SuccessfulProject 111 1995 1997 UKRAINE PRIVATE L Low Marginal Some Partly SuccessfulProject 112 1994 1997 BULGARIA PRIVATE E Low Poor None Partly SuccessfulProject 113 1992 1994 HUNGARY PRIVATE E Low Partly SuccessfulProject 114 1991 1995 RUSSIAN FEDERATION PRIVATE L Low Partly SuccessfulProject 115 2000 2002 RUSSIAN FEDERATION PRIVATE L Marginal Excellent Substantial Partly SuccessfulProject 116 1995 2001 TURKMENISTAN PRIVATE E/L Marginal Good Substantial Partly SuccessfulProject 117 1996 2000 RUSSIAN FEDERATION PRIVATE L Marginal Good Some Partly SuccessfulProject 118 1999 2000 LITHUANIA PRIVATE E Marginal Good Some Partly SuccessfulProject 119 1995 2001 AZERBAIJAN STATE L Marginal Good Some Partly SuccessfulProject 120 1996 2002 KAZAKHSTAN STATE L Marginal Good Some Partly SuccessfulProject 121 1998 2001 SLOVENIA PRIVATE E/L Marginal Marginal Some Partly SuccessfulProject 122 1999 2002 HUNGARY PRIVATE E Marginal Marginal None Partly SuccessfulProject 123 1996 1999 RUSSIAN FEDERATION PRIVATE E/L None Marginal Some Partly Successful

Appendix 2Page 3 of 9

Basic data sheet: Operation performance ratings on the 157 OPERs prepared in 1993-2002

Operation Year of Board

Approval

Year of evaluation

Country Portfolio Class

Type1 Transition Impact2

Environmental Performance of

Sponsor and Bank3

Extent of Environmental

Change4

Overall Rating5

Project 124 1994 1999 MOLDOVA STATE L Medium Satisfactory None UnsuccessfulProject 125 1996 1999 POLAND PRIVATE E Low Excellent Some UnsuccessfulProject 126 1995 1999 LATVIA PRIVATE E/L Low Satisfactory Some UnsuccessfulProject 127 1994 1996 CZECH REPUBLIC PRIVATE L Low Marginal None UnsuccessfulProject 128 1993 1997 UZBEKISTAN PRIVATE L Low Marginal Some UnsuccessfulProject 129 1994 1998 RUSSIAN FEDERATION PRIVATE E Low Marginal None UnsuccessfulProject 130 1995 1998 RUSSIAN FEDERATION PRIVATE E Low Poor None UnsuccessfulProject 131 1992 1994 CZECH REPUBLIC PRIVATE E Low UnsuccessfulProject 132 1991 1994 POLAND PRIVATE L Low UnsuccessfulProject 133 1992 1995 RUSSIAN FEDERATION PRIVATE E Low UnsuccessfulProject 134 1994 2002 BULGARIA PRIVATE E Marginal Good Some UnsuccessfulProject 135 1997 2001 RUSSIAN FEDERATION PRIVATE L Marginal Marginal Substantial UnsuccessfulProject 136 1997 1999 CZECH REPUBLIC PRIVATE E None Excellent Some UnsuccessfulProject 137 1993 1997 HUNGARY PRIVATE L None Satisfactory None UnsuccessfulProject 138 1995 1998 RUSSIAN FEDERATION PRIVATE L None Satisfactory Some UnsuccessfulProject 139 1992 1997 CZECH REPUBLIC PRIVATE L None Marginal Some UnsuccessfulProject 140 1996 1998 RUSSIAN FEDERATION PRIVATE L None Marginal None UnsuccessfulProject 141 1991 1993 POLAND PRIVATE L None UnsuccessfulProject 142 1991 1993 HUNGARY PRIVATE E None UnsuccessfulProject 143 1995 2000 MOLDOVA STATE L Unsatisfactory Good Some UnsuccessfulProject 144 1998 2001 CROATIA PRIVATE E Unsatisfactory Good Some UnsuccessfulProject 145 1997 2001 POLAND PRIVATE E/L Unsatisfactory Good Some UnsuccessfulProject 146 1995 2002 UKRAINE PRIVATE L Unsatisfactory Good None UnsuccessfulProject 147 1996 2000 BULGARIA PRIVATE E/L Unsatisfactory Marginal None UnsuccessfulProject 148 1997 2001 POLAND PRIVATE E Unsatisfactory NA NA UnsuccessfulProject 149 1993 1999 HUNGARY PRIVATE E/L Negative Excellent Some UnsuccessfulProject 150 1994 2000 CZECH REPUBLIC STATE L Negative Good tbd UnsuccessfulProject 151 1994 1998 SLOVAK REPUBLIC PRIVATE E/L Negative Satisfactory Substantial UnsuccessfulProject 152 1997 2001 POLAND PRIVATE E Negative Marginal Substantial UnsuccessfulProject 153 1995 1999 KYRGYZ REPUBLIC PRIVATE L Negative Marginal Some UnsuccessfulProject 154 1993 1998 ARMENIA STATE L Negative Marginal None UnsuccessfulProject 155 1996 1999 RUSSIAN FEDERATION PRIVATE L Negative Marginal None UnsuccessfulProject 156 1995 2002 RUSSIAN FEDERATION PRIVATE L Negative Marginal None UnsuccessfulProject 157 1997 1998 ESTONIA PRIVATE E Negative Poor None Unsuccessful1 E=Equity; L=Loan2 The range is Excellent/Good/Satisfactory/Marginal/Unsatisfactory/Negative for projeccts evaluated from 2000; High/Medium/Low/None/Negative before 20003 The range is Excellent/Good/Satisfactory/Marginal/Unsatisfactory/Highly Unsatisfactory4 The range is Outstanding/Substanital/Some/None/Negative5 The range is Highly Successful/Successful/Partly Successful/Unsuccessful

Appendix 2Page 4 of 9

Basic data sheet: Operation performance ratings on the 181 XMR assessments prepared in 1996-2002

Operation Year of Board

Approval

Year of evaluation

Country Portfolio Class

Type1 Transition Impact2

Environmental Performance of

Sponsor and Bank3

Extent of Environmental

Change4

Overall Rating5

Project 1 1998 2002 RUSSIAN FEDERATION PRIVATE L Excellent Good Some Highly SuccessfulProject 2 1997 2000 BOSNIA AND HERZEGOVINA PRIVATE L Good Excellent Some Highly SuccessfulProject 3 1996 2000 HUNGARY PRIVATE E/L Good Good Some Highly SuccessfulProject 4 1997 2000 ROMANIA PRIVATE E Good Marginal Some Highly SuccessfulProject 5 1999 2002 SLOVENIA PRIVATE L Good Marginal Some Highly SuccessfulProject 6 1996 1999 RUSSIAN FEDERATION PRIVATE E/L High Excellent Outstanding Highly SuccessfulProject 7 1995 1998 HUNGARY PRIVATE E High Excellent Substantial Highly SuccessfulProject 8 1996 1998 ESTONIA PRIVATE L High Excellent Some Highly SuccessfulProject 9 1993 1999 HUNGARY PRIVATE L High Satisfactory Some Highly SuccessfulProject 10 1995 1997 HUNGARY PRIVATE E High Highly SuccessfulProject 11 1995 2000 ROMANIA PRIVATE L Excellent Excellent Some SuccessfulProject 12 1999 2002 FYR MACEDONIA PRIVATE L Excellent NA None SuccessfulProject 13 1997 1999 BULGARIA PRIVATE L High Excellent Substantial SuccessfulProject 14 1992 1998 BULGARIA STATE L High Excellent Some SuccessfulProject 15 1994 1998 UKRAINE STATE L High Excellent Some SuccessfulProject 16 1997 1999 HUNGARY PRIVATE L High Excellent Some SuccessfulProject 17 1997 1999 LATVIA PRIVATE E/L High Excellent None SuccessfulProject 18 1994 1999 KYRGYZ REPUBLIC STATE L High Excellent None SuccessfulProject 19 1995 1998 LITHUANIA PRIVATE L High Satisfactory Substantial SuccessfulProject 20 1996 1998 CROATIA PRIVATE L High Satisfactory Some SuccessfulProject 21 1995 1998 CROATIA PRIVATE L High Satisfactory Some SuccessfulProject 22 1995 1998 POLAND PRIVATE E High Satisfactory None SuccessfulProject 23 1993 1998 ALBANIA PRIVATE E/L High Satisfactory None SuccessfulProject 24 1993 1996 SLOVAK REPUBLIC PRIVATE E High SuccessfulProject 25 1994 1997 POLAND PRIVATE E High SuccessfulProject 26 1992 1997 ESTONIA STATE L High SuccessfulProject 27 1991 1997 POLAND PRIVATE L High SuccessfulProject 28 1997 2000 <REGIONAL> PRIVATE E/L Good Excellent Substantial SuccessfulProject 29 1992 2000 RUSSIAN FEDERATION PRIVATE L Good Excellent Some SuccessfulProject 30 1997 2001 RUSSIAN FEDERATION PRIVATE L Good Good Substantial SuccessfulProject 31 1995 2000 POLAND PRIVATE L Good Good Some SuccessfulProject 32 1996 2000 BOSNIA AND HERZEGOVINA PRIVATE E Good Good Some SuccessfulProject 33 1996 2000 CROATIA PRIVATE E/L Good Good Some SuccessfulProject 34 1996 2000 CROATIA PRIVATE E Good Good Some SuccessfulProject 35 1999 2001 ROMANIA PRIVATE E Good Good Some SuccessfulProject 36 1996 2001 ROMANIA STATE L Good Good Some SuccessfulProject 37 1996 2001 ROMANIA PRIVATE L Good Good Some SuccessfulProject 38 1999 2001 TAJIKISTAN STATE L Good Good Some SuccessfulProject 39 1995 2001 FYR MACEDONIA PRIVATE L Good Good Some SuccessfulProject 40 2000 2001 ARMENIA PRIVATE L Good Good Some SuccessfulProject 41 1999 2001 GEORGIA PRIVATE L Good Good Some Successful

Appendix 2Page 5 of 9

Basic data sheet: Operation performance ratings on the 181 XMR assessments prepared in 1996-2002

Operation Year of Board

Approval

Year of evaluation

Country Portfolio Class

Type1 Transition Impact2

Environmental Performance of

Sponsor and Bank3

Extent of Environmental

Change4

Overall Rating5

Project 42 1998 2002 MOLDOVA STATE L Good Good Some SuccessfulProject 43 1997 2002 UKRAINE PRIVATE L Good Good Some SuccessfulProject 44 1999 2002 RUSSIAN FEDERATION PRIVATE L Good Good Some SuccessfulProject 45 1994 2002 <REGIONAL> PRIVATE E Good Good Some SuccessfulProject 46 1999 2002 BOSNIA AND HERZEGOVINA PRIVATE L Good Good Some SuccessfulProject 47 2000 2002 POLAND PRIVATE E Good Good Some SuccessfulProject 48 1997 2001 KYRGYZ REPUBLIC STATE L Good Good None SuccessfulProject 49 1998 2001 BOSNIA AND HERZEGOVINA STATE L Good Good None SuccessfulProject 50 1999 2002 UKRAINE PRIVATE L Good Satisfactory Some SuccessfulProject 51 1997 2000 HUNGARY PRIVATE E Good Marginal Some SuccessfulProject 52 1998 2002 GEORGIA PRIVATE E/L Good Marginal Some SuccessfulProject 53 1994 1998 POLAND PRIVATE L Medium Excellent Outstanding SuccessfulProject 54 1993 1998 POLAND PRIVATE L Medium Excellent Substantial SuccessfulProject 55 1995 1998 HUNGARY PRIVATE L Medium Excellent Substantial SuccessfulProject 56 1992 1998 ALBANIA STATE L Medium Excellent Some SuccessfulProject 57 1995 1999 SLOVAK REPUBLIC PRIVATE L Medium Excellent Some SuccessfulProject 58 1997 1999 SLOVAK REPUBLIC PRIVATE L Medium Excellent Some SuccessfulProject 59 1996 1999 CROATIA PRIVATE L Medium Excellent Some SuccessfulProject 60 1993 1998 SLOVENIA PRIVATE L Medium Excellent None SuccessfulProject 61 1995 1998 SLOVENIA PRIVATE L Medium Excellent None SuccessfulProject 62 1993 1999 ROMANIA PRIVATE L Medium Excellent None SuccessfulProject 63 1995 1998 RUSSIAN FEDERATION PRIVATE L Medium Satisfactory Substantial SuccessfulProject 64 1994 1999 RUSSIAN FEDERATION PRIVATE L Medium Satisfactory Substantial SuccessfulProject 65 1994 1998 POLAND PRIVATE L Medium Satisfactory Some SuccessfulProject 66 1994 1998 LITHUANIA STATE L Medium Satisfactory Some SuccessfulProject 67 1995 1998 RUSSIAN FEDERATION PRIVATE L Medium Satisfactory Some SuccessfulProject 68 1993 1999 <REGIONAL> PRIVATE E Medium Satisfactory Some SuccessfulProject 69 1995 1998 KYRGYZ REPUBLIC STATE L Medium Satisfactory None SuccessfulProject 70 1993 1999 SLOVENIA PRIVATE E Medium Satisfactory None SuccessfulProject 71 1997 1999 SLOVENIA PRIVATE E Medium Satisfactory None SuccessfulProject 72 1996 1999 POLAND PRIVATE L Medium Satisfactory None SuccessfulProject 73 1992 1999 POLAND PRIVATE E Medium Satisfactory None SuccessfulProject 74 1993 1997 ROMANIA PRIVATE L Medium Satisfactory NA SuccessfulProject 75 1997 1999 POLAND PRIVATE L Medium Marginal Some SuccessfulProject 76 1993 1996 SLOVAK REPUBLIC PRIVATE L Medium SuccessfulProject 77 1993 1996 HUNGARY PRIVATE L Medium SuccessfulProject 78 1994 1996 RUSSIAN FEDERATION PRIVATE L Medium SuccessfulProject 79 1996 1996 ESTONIA PRIVATE L Medium SuccessfulProject 80 1994 1996 HUNGARY PRIVATE E Medium SuccessfulProject 81 1992 1996 ROMANIA STATE L Medium SuccessfulProject 82 1992 1996 POLAND PRIVATE L Medium Successful

Appendix 2Page 6 of 9

Basic data sheet: Operation performance ratings on the 181 XMR assessments prepared in 1996-2002

Operation Year of Board

Approval

Year of evaluation

Country Portfolio Class

Type1 Transition Impact2

Environmental Performance of

Sponsor and Bank3

Extent of Environmental

Change4

Overall Rating5

Project 83 1993 1996 POLAND PRIVATE L Medium SuccessfulProject 84 1994 1997 HUNGARY PRIVATE L Medium SuccessfulProject 85 1994 1997 CZECH REPUBLIC PRIVATE L Medium SuccessfulProject 86 1993 1997 UKRAINE STATE L Medium SuccessfulProject 87 1993 1997 RUSSIAN FEDERATION PRIVATE E Medium SuccessfulProject 88 1992 1997 LITHUANIA STATE L Medium SuccessfulProject 89 1999 2000 SLOVAK REPUBLIC STATE L Satisfactory Excellent None SuccessfulProject 90 1997 2000 POLAND PRIVATE E/L Satisfactory Good Substantial SuccessfulProject 91 1993 2000 RUSSIAN FEDERATION PRIVATE E/L Satisfactory Good Substantial SuccessfulProject 92 1998 2001 ESTONIA PRIVATE L Satisfactory Good Some SuccessfulProject 93 1999 2001 HUNGARY PRIVATE L Satisfactory Good Some SuccessfulProject 94 1996 2001 <REGIONAL> PRIVATE E Satisfactory Good Some SuccessfulProject 95 2000 2002 ALBANIA PRIVATE E Satisfactory Good Some SuccessfulProject 96 1995 2002 LATVIA STATE L Satisfactory Good Some SuccessfulProject 97 1999 2002 POLAND PRIVATE L Satisfactory Good Some SuccessfulProject 98 1995 1998 RUSSIAN FEDERATION PRIVATE E High NA NA Partly SuccessfulProject 99 1994 1996 POLAND PRIVATE L High Partly SuccessfulProject 100 1992 1997 <REGIONAL> STATE L High Partly SuccessfulProject 101 1994 2001 SLOVENIA PRIVATE E Good Good Some Partly SuccessfulProject 102 1995 2001 <REGIONAL> PRIVATE E Good Marginal Some Partly SuccessfulProject 103 1995 2002 ROMANIA STATE L Good Marginal Some Partly SuccessfulProject 104 1998 2002 POLAND PRIVATE L Good Marginal Some Partly SuccessfulProject 105 1996 1998 RUSSIAN FEDERATION PRIVATE L Medium Excellent Substantial Partly SuccessfulProject 106 1994 1998 ESTONIA PRIVATE L Medium Satisfactory Substantial Partly SuccessfulProject 107 1993 1998 ALBANIA PRIVATE E/L Medium Satisfactory Some Partly SuccessfulProject 108 1997 1999 TAJIKISTAN PRIVATE L Medium Satisfactory Some Partly SuccessfulProject 109 1994 1999 POLAND PRIVATE E Medium Satisfactory Some Partly SuccessfulProject 110 1996 1999 UZBEKISTAN PRIVATE E Medium Satisfactory None Partly SuccessfulProject 111 1994 1999 KYRGYZ REPUBLIC PRIVATE L Medium Satisfactory None Partly SuccessfulProject 112 1995 1999 POLAND PRIVATE E Medium Marginal Some Partly SuccessfulProject 113 1993 1998 ROMANIA PRIVATE E/L Medium NA NA Partly SuccessfulProject 114 1993 1996 ALBANIA PRIVATE L Medium Partly SuccessfulProject 115 1994 1996 LATVIA PRIVATE E Medium Partly SuccessfulProject 116 1992 1996 RUSSIAN FEDERATION PRIVATE L Medium Partly SuccessfulProject 117 1993 1996 HUNGARY PRIVATE E Medium Partly SuccessfulProject 118 1993 1996 BULGARIA PRIVATE E Medium Partly SuccessfulProject 119 1992 1996 HUNGARY PRIVATE L Medium Partly SuccessfulProject 120 1994 1997 LITHUANIA PRIVATE E/L Medium Partly SuccessfulProject 121 1994 1997 ROMANIA PRIVATE L Medium Partly SuccessfulProject 122 1996 1997 POLAND PRIVATE E/L Medium Partly SuccessfulProject 123 1995 1997 CZECH REPUBLIC PRIVATE L Medium Partly Successful

Appendix 2Page 7 of 9

Basic data sheet: Operation performance ratings on the 181 XMR assessments prepared in 1996-2002

Operation Year of Board

Approval

Year of evaluation

Country Portfolio Class

Type1 Transition Impact2

Environmental Performance of

Sponsor and Bank3

Extent of Environmental

Change4

Overall Rating5

Project 124 1993 1997 SLOVAK REPUBLIC STATE L Medium Partly SuccessfulProject 125 1994 1997 SLOVAK REPUBLIC PRIVATE L Medium Partly SuccessfulProject 126 1996 2001 POLAND STATE L Satisfactory Excellent Substantial Partly SuccessfulProject 127 1996 2002 ROMANIA STATE L Satisfactory Excellent Substantial Partly SuccessfulProject 128 1997 2000 ROMANIA PRIVATE E/L Satisfactory Excellent Some Partly SuccessfulProject 129 1998 2001 BULGARIA PRIVATE E/L Satisfactory Excellent Some Partly SuccessfulProject 130 1997 2000 BELARUS PRIVATE E Satisfactory Good Some Partly SuccessfulProject 131 1998 2001 LITHUANIA PRIVATE E/L Satisfactory Good Some Partly SuccessfulProject 132 1995 2001 RUSSIAN FEDERATION PRIVATE L Satisfactory Good Some Partly SuccessfulProject 133 1997 2001 LATVIA PRIVATE L Satisfactory Good Some Partly SuccessfulProject 134 2000 2002 POLAND PRIVATE E Satisfactory Good Some Partly SuccessfulProject 135 1997 2002 CROATIA STATE L Satisfactory Good Some Partly SuccessfulProject 136 1998 2002 ESTONIA PRIVATE E/L Satisfactory Good Some Partly SuccessfulProject 137 1997 2002 KAZAKHSTAN PRIVATE L Satisfactory Good Some Partly SuccessfulProject 138 2000 2002 CROATIA PRIVATE L Satisfactory Good Some Partly SuccessfulProject 139 1992 2000 BELARUS STATE L Satisfactory Good None Partly SuccessfulProject 140 1995 2000 UZBEKISTAN PRIVATE E Satisfactory Marginal None Partly SuccessfulProject 141 1995 2001 <REGIONAL> PRIVATE E Satisfactory Marginal None Partly SuccessfulProject 142 1994 1998 RUSSIAN FEDERATION PRIVATE L Low Satisfactory Some Partly SuccessfulProject 143 1996 1999 BULGARIA PRIVATE L Low Satisfactory Some Partly SuccessfulProject 144 1996 1999 HUNGARY PRIVATE L Low Satisfactory Some Partly SuccessfulProject 145 1994 1997 BULGARIA PRIVATE L Low Partly SuccessfulProject 146 1994 1997 ROMANIA PRIVATE L Low Partly SuccessfulProject 147 1996 1997 ROMANIA PRIVATE E Low Partly SuccessfulProject 148 1994 2001 BELARUS PRIVATE L Marginal Good Some Partly SuccessfulProject 149 1998 2002 SLOVAK REPUBLIC PRIVATE L Marginal Good Some Partly SuccessfulProject 150 1999 2001 RUSSIAN FEDERATION PRIVATE L Marginal Satisfactory Some Partly SuccessfulProject 151 1995 2000 UKRAINE PRIVATE E/L Marginal Marginal None Partly SuccessfulProject 152 1996 2002 RUSSIAN FEDERATION PRIVATE L Marginal Marginal None Partly SuccessfulProject 153 1993 1999 ROMANIA STATE L Medium Satisfactory Some UnsuccessfulProject 154 1994 2000 ARMENIA STATE L Satisfactory Good Some UnsuccessfulProject 155 1993 1998 BELARUS STATE L Low Excellent Substantial UnsuccessfulProject 156 1995 1998 RUSSIAN FEDERATION PRIVATE L Low Satisfactory Substantial UnsuccessfulProject 157 1995 1998 UZBEKISTAN PRIVATE E Low Satisfactory Some UnsuccessfulProject 158 1992 1996 POLAND PRIVATE L Low UnsuccessfulProject 159 1993 1996 ROMANIA PRIVATE E Low UnsuccessfulProject 160 1991 1996 CZECH REPUBLIC PRIVATE L Low UnsuccessfulProject 161 1991 1996 CZECH REPUBLIC PRIVATE L Low UnsuccessfulProject 162 1994 1997 CZECH REPUBLIC PRIVATE L Low UnsuccessfulProject 163 2000 2002 ROMANIA PRIVATE L Marginal Excellent Some UnsuccessfulProject 164 1999 2002 LATVIA PRIVATE L Marginal Excellent None Unsuccessful

Appendix 2Page 8 of 9

Basic data sheet: Operation performance ratings on the 181 XMR assessments prepared in 1996-2002

Operation Year of Board

Approval

Year of evaluation

Country Portfolio Class

Type1 Transition Impact2

Environmental Performance of

Sponsor and Bank3

Extent of Environmental

Change4

Overall Rating5

Project 165 1995 2001 MOLDOVA STATE L Marginal Unsatisfactory None UnsuccessfulProject 166 1993 2000 <REGIONAL> PRIVATE E Marginal NA NA UnsuccessfulProject 167 1993 1998 RUSSIAN FEDERATION PRIVATE E None Satisfactory Substantial UnsuccessfulProject 168 1995 1999 POLAND PRIVATE E None Marginal Some UnsuccessfulProject 169 1994 1999 RUSSIAN FEDERATION PRIVATE L None Marginal None UnsuccessfulProject 170 1996 1999 RUSSIAN FEDERATION PRIVATE E/L None Poor None UnsuccessfulProject 171 1997 1999 RUSSIAN FEDERATION PRIVATE E None Poor None UnsuccessfulProject 172 1995 2000 GEORGIA PRIVATE E Unsatisfactory Good Some UnsuccessfulProject 173 1998 2001 POLAND PRIVATE E Unsatisfactory Good Some UnsuccessfulProject 174 1996 2002 GEORGIA PRIVATE E/L Unsatisfactory Marginal Some UnsuccessfulProject 175 1997 2001 SLOVAK REPUBLIC PRIVATE E Unsatisfactory Marginal None UnsuccessfulProject 176 1997 2001 RUSSIAN FEDERATION PRIVATE E/L Unsatisfactory Marginal Negative UnsuccessfulProject 177 1994 2001 SLOVENIA PRIVATE E Unsatisfactory Unsatisfactory None UnsuccessfulProject 178 1997 2002 CZECH REPUBLIC PRIVATE E Unsatisfactory NA None UnsuccessfulProject 179 1997 2000 RUSSIAN FEDERATION PRIVATE E Unsatisfactory NA NA UnsuccessfulProject 180 1996 2001 LATVIA PRIVATE E/L Negative Good Some UnsuccessfulProject 181 1995 2000 RUSSIAN FEDERATION PRIVATE E Negative Marginal None Unsuccessful1 E=Equity; L=Loan2 The range is Excellent/Good/Satisfactory/Marginal/Unsatisfactory/Negative for projeccts evaluated from 2000; High/Medium/Low/None/Negative before 20003 The range is Excellent/Good/Satisfactory/Marginal/Unsatisfactory/Highly Unsatisfactory4 The range is Outstanding/Substanital/Some/None/Negative5 The range is Highly Successful/Successful/Partly Successful/Unsuccessful

Appendix 2Page 9 of 9

1993-2003 Technical Cooperation Operation Performance Evaluation Review (OPER) Reports

No. Operations Country Sector Industry TC

Funds Amt

Type 1 OpsCom Approval

Funding Approved

Project Completion

Report (PCR) Date

OPER Report Date

Overall Rating2

1993 1 Privatisation Advisory Programme in the Russian Fed. Russia State Privatisation 5,044 AS 16-Mar-92 May-92 - Dec-93 Successful2 Telecommunications Master Plan Albania State Telecoms 198 PP 09-Apr-92 May-92 - Dec-93 Partly Successful3 Banking Sector Restructuring Romania State Finance 855 SW/AS 07-Aug-92 Aug-92 - Jan-94 Successful4 Railway Sector Survey Regional State Transport 766 SW 17-Feb-92 Mar-92 22-Jun-93 Feb-94 Successful5 Roads and Road Transport Sector Survey Regional State Transport 409 SW 17-Feb-92 Apr-92 24-Sep-93 Feb-94 Successful

Subtotal 7,2721994

1 Regional Training Programme Regional State Finance 990 T 02-Dec-91 Jan-92 16-Feb-93 Aug-94 Partly Successful2 Tallinn Environment Project Estonia State Environment 158 PP 08-May-92 Oct-92 29-Nov-94 Dec-94 Partly Successful3 Tourism Development for Albania Albania State Tourism 223 AS 09-Apr-92 Apr-92 30-May-94 Jan-95 Partly Successful

Subtotal 1,371 1995

1 Wine Sector Investment Programme Moldova State Agribusiness 440 PP/PI 19-Mar-93 Jun-93 21-Dec-94 Jul-95 Successful2 SME Sector Development Project Preparation Belarus State SME 174 AS 09-Jul-93 Dec-93 06-May-95 Jan-96 Successful3 State Railways Restructuring and Rail Modernisation Bulgaria State Transport 583 PP 22-Jun-94 Jul-92 03-Apr-95 Jan-96 Partly Successful

Subtotal 1,197 1996

1 Romanian Banking Institute Romania State Finance 435 T 07-Mar-92 Apr-92 25-May-95 Aug-94 Successful 2 Bulgarian Investment Bank Bulgaria Private Finance 942 AS/PP 30-Apr-93 Jun-93 11-Sep-95 Dec-94 Successful3 Budapest Wholesale Market Hungary State Agriculture 587 PP 08-May-92 Jul-92 28-Oct-93 Jan-95 Partly Successful

Subtotal 1,964 1997

1 Unified Gas Supply System Russia State Energy 4,500 PP/PI 19-Feb-93 Apr-93 19-Jun-96 Jan-98 Successful2 INCAR JSC Enterprise Restructuring Russia State Restructuring 612 PP 15-Aug-93 Dec-93 02-Dec-96 Feb-98 Unsuccessful3 Perm Motors JSC Enterprise Restructuring Russia State Restructuring 862 PP 15-Aug-93 Dec-93 02-Dec-96 Feb-98 Partly Successful

Subtotal 5,974 1998

1 Project Preparation TC MEI Investment Programme Croatia State Environment 179 PP 27-Nov-95 Feb-96 11-May-98 Jan-99 Partly Successful 2 EC Phare/Tacis Framework Contracts for FIs Regional Private Finance 2,951 PP 18-Feb-94 Aug-94 - Jan-99 Successful3 Environmental Due Diligence of FIs Regional Private Environment 3,264 PP/AS/PI 14-Mar-94 Aug-94 06-Mar-98 May-99 Successful4 Privatisation Advisory Programme Ukraine State Privatisation 2,730 PP/AS 03-May-91 Jun-92 28-Feb-95 Sep-98 Partly Successful5 Aktau Port Rehabilitation Kazakhstan State Transport 2,364 PP/PI/SW 28-May-93 Aug-93 20-Jan-94 Aug-98 Partly Successful

Subtotal 11,489 1999 1 Mining Privatisation Kazakhstan State Mining 406 PP/AS 20-May-94 May-94 18-Mar-96 Oct-99 Partly Successful2 Municipal Utility Development and Investment Ukraine State Environment 1,042 PP 22-Mar-96 Jun-96 16-Mar-98 Jan-00 Successful/Unsuccessful3 Telecom Legislative and Regulatory Development Lithuania State Telecom 289 AS 02-Feb-96 Nov-96 05-Jan-00 Jan-00 Successful 4 Swiss American Micro Enterprise Programme Moldova Private SME 1,078 PP 03-May-96 Aug-96 16-Jul-99 Jan-00 Partly Successful

Subtotal 2,815

Appendix 3Page 1

of 4

1993-2003 Technical Cooperation Operation Performance Evaluation Review (OPER) Reports

No.

Operation Country Sector Industry TC

Funds Amt

Type 1 OpsCom Approval

Funding Approved PCR Date

OPER Report Date

Overall Rating

2000 1 Railways Modernisation Russia State Transport 844 PP/AS 01-Jun-93 Aug-93 19-Apr-96 Jul-00 Partly Successful2 Credit Worthiness of the City of Zagreb Croatia State Environment 184 PP 17-Oct-97 Jan-98 23-Mar-99 Jan-01 Successful 3 SME Credit Line I and II Kyrgyz Republic Private SME 2,233 PP/AS/PI 04-Jun-93 Nov-93 01-Jun-95 Jan-01 Successful4 Power Market Twinning Programme Ukraine State Energy 1,297 PP/AS/PI 08-Mar-96 Mar-97 22-Feb-00 Jan-01 Unsuccessful

Subtotal 4,557 2001 1 Telecommunications Emergency Reconstruction Bosnia and Herzegovina State Telecoms 1,870 AS/PI 03-Oct-97 Dec-97 22-Feb-00 July-01 Highly Successful 2 Mutnovsky Independent Power Plant Russia State Energy 1,319 PP/AS/PI 07-May- May-93 16-Sep-94 Dec-01 Partly Successful3 Road Rehabilitation and Upgrading Azerbaijan State Transport 755 PP 19-Apr-96 May-96 03-Aug-99 Dec-01 Unsuccessful 4 Creditworthiness Assessment of City of Wroclaw Poland State Energy 481 AS/PI 25-July-97 Aug-98 - Jan-02 Successful

Subtotal 4,425 2002 1 Turkmenbashi Port Development Project Turkmenistan State Transport 2,895 AS/PI 19-Sep-95 07-Jul-95 16-Jul-99 Jun-02 Successful2 Enterprise Investment Demonstration Project Kyrgyz Republic Private Finance 1,405 PP/PI 16-May- 19-Jun-97 22-Feb-02 Aug-02 Unsuccessful3 Enguri Rehabilitation Project Georgia State Energy 453 PP/PI 04-Aug-95 18-Aug-95 17-June-97 Nov-02 Partly Successful

4 Emergency Electricity Power Reconstruction Project Bosnia and Herzegovina State Energy 2,150 AS/PP/PI/T 19-Jan-96 01-Jul-96 22-Feb-00 Mar-03 Highly/Partly

Successful 5 Tajikistan Overlay Network Tajikistan State Telecoms 457 AS/PP 06-Oct-95 21-Dec-95 21-Feb-97 Mar-03 Partly Successful6 Energy Efficiency TC Studies Russia State Energy 779 PP 07-Mar-97 01-Apr-97 05-Jun-02 Feb-03 Unsuccessful

Subtotal 8,139 2003 1 Inst. Dev. & Mgt. of Baku Port Azerbaijan State Transport 991 PI 01-May- 24-Jul-98 23-Sep-02 Apr-03 Successful2 Norsi Oil Refinery Russia Private Oil & Gas 1,165 PP 25-Jul-97 07-Aug-97 05-Jun-02 Aug-03 Partly Successful3 Env. Support to Budapest Bank Credit Line Hungary Private Finance 281 PP/PI 07-Jun-96 13-Feb-97 05-Jun-02 Dec-03 Partly Successful4 Technical Assistance to Uzbekneftegas Uzbekistan State Oil & Gas 1,443 PP/AS 03-Mar-95 01-Apr-95 05-Jun-02 Oct-03 Partly Successful5 Scoping Study for Railway Restructuring Project Bosnia & Herzegovina State Transport 199 PP 04-Apr-00 30-Jun-00 05-Jun-02 Mar-04 Successful6 Azeri Multi Bank Framework Financing Facility Azerbaijan Private Finance 3,227 PP/PI/AS/T 12-Jul-96 19-Aug-96 13-Nov-02 Feb-04 Partly Successful

Subtotal 7,306 Note: The totals may not add up to the sum of the component parts due to rounding. 1 AS=Advisory Services; PP=Project Preparation; SW=Sector Work; T=Training; PI=Project Implementation

Appendix 3

Page 2 of 4

1993-2003 Special Studies and Evaluation Progress Reviews

Operation Country Sector Industry EBRD Finance

TC Funds Amt

Type 1 Board Approval

Report Date Report Type

19941 Russia Small Business Fund I Russia Private SME 2,381 2,851 Line of Credit/TC 26-Jul-93 Jul-94 Special Study/Mid-Term Review Sub-Total 2,381 2,851

1995 1 Russia Small Business Fund II Russia Private SME 16,932 5,355 Line of Credit/TC 26-Jul-93 Mar-95 Special Study/Mid-Term Review 2 Regional Bank Training Centre CAR State Finance n.a. 1,704 TC 10-Nov-92 Oct-95 Mid-Term Review 3 Agribusiness Project Preparation Units Regional State Agribusiness n.a. 4,590 TC 18-May- Sep-95 Special Study 4 Project Preparation TCs Regional State Various n.a. 8,349 TC n.a. Dec-95 Special Study5 Belarus SME Credit Line Belarus Private Finance 23,811 1,420 Loan/TC 01-Nov-94 Jan-96 Mid-Term Review Sub-Total 40,743 21,417

1996 1 SME Credit Line Project Ukraine Private Finance 96,186 - Loan 29-Nov-94 Dec-96 Special Study/Mid-Term Review 2 Wholesale Market Special Study Hungary State Agriculture 48,387 3,455 Loan/TC n.a. Jan-97 Special Study 3 Regional Bank Training Centre TC Uzbekistan State Finance n.a. 1,704 TC 10-May- Sep-96 Evaluation Progress Review Sub-Total 144,573 5,159

1997 1 Kyrgyzstan SME Credit Line Kyrgyz Rep Private Finance 8,334 1,888 Line of Credit/TC 11-Nov-94 May-97 Mid-Term Review 2 Russia Small Business Fund III Russia Private SME 172,994 32,707 Line of Credit/TC 26-Jul-93 Jul-97 Special Study/Mid-Term Review 3/ Regional Venture Funds Russia State SME 126,239 20,814 Equity/TC n.a. Aug-97 Special Study 5 Baltics Business Advisory Service Baltics State Finance n.a. 4,196 TC n.a. Sep-97 Mid-Term Review6 TAM Programme Regional State Priv/Restr n.a. 11,417 TC n.a. Feb-98 Special Study 7 Financial Institutions Development Project Russia Private Finance 79,326 - Loan 23-May- Jan-98 Mid-Term Review Subtotal 386,893 71,022

1998 1 Sample of PCR Reviews and Assessments Various Private/State Various n.a. 7,374 TC Various Jan-99 Special Study2 Regional Bank Training Centre TC Uzbekistan State Finance n.a. 1,704 TC 10-May- Sep-96 Evaluation Progress Review Subtotal 0 9,078 1999 1 Sample of PCR Reviews and Assessments Various Private/State Various n.a. 9,445 TC Various May-00 Special Study2 Thematic Study on SME Support Various Private SME n.a. n.a. n.a. n.a. Jun-00 Special Study3 Nuclear Safety Account Various n.a. Energy n.a. n.a. n.a. n.a. Nov-00 Special Study4 Technical Cooperation Funds Programme Various n.a. n.a. n.a. n.a. n.a. n.a. Jul-00 Special Study Subtotal n.a. 9,445 2000 1 Sample of PCR Reviews and Assessments Various Private/State Various n.a. 11,941 TC Various Jan-01 Special Study2 Post-Privatisation Funds Various Private n.a. 137,245 19,120 Equity /TC n.a. n.a. Special Study3 Evaluation of Environmental Performance Various n.a. Environment n.a. n.a. n.a. n.a. n.a. Special Study4 Scope Paper on Country Strategy Kazakhstan n.a. n.a. n.a. n.a. n.a. 04-Oct-00 n.a. Scope for Special Study Subtotal 137,245 31,061

Appendix 3Page 3 of 4

Appendix 3

Page 4 of 4

1993-2003 Special Studies and Evaluation Progress Reviews

2001 1 Sample of PCR Assessments Various Private/State Various n.a. 7,023 TC Various Jan-02 Special Study2 Direct Investment Facility Various SME Various 37,437 3,179 Equity 24-Feb-98 Nov-01 Mid-Term Review 3 Legal Transition Programme Various n.a. Various n.a. 12,007 n.a. n.a. Oct-01 Mid-Term Review4 Financial Institutions Development Russia Private Finance 33,756 1,140 Loan 23-May- Aug-02 Special Study 5 Energy Efficiency of the Bank's Operations Various n.a. n.a. n.a. n.a. n.a. n.a. Feb-02 Special Study Subtotal 71,193 23,349 2002 1 Sample of PCR Reviews and Assessments Various Private/State Various n.a. 15,227 TC Various Feb-03 Special Study2 Regional Trade Facilitation Programme Various Private Various 300,000 519 Guarantee/Loan/TC 13-Dec-94 Mar-03 Special Study 3 Russia Small Business Fund IV Russia Private SME 345,987 37,110 Line of Credit/TC 26-Jul-93 Apr-03 Mid-Term Review4 EBRD’s Investments in Equity Funds Various Private SME 1,500,000 n.a. Equity Funds Various Oct-02 Mid-Term Review Subtotal 2,145,987 52,856 2003 1 Sample of PCR Assessments Various Private/State Various n.a. 5,458 TC Various Mar-04 Special Study2 TurnAround Management Programme Regional Private/State Various n.a. 4,789 TC Various Apr-04 Special Study3 Extractive Industries Regional Private/State Various n.a. n.a. Equity/Loan Various May-04 Special Study4 Country Strategy Evaluation Slovak Rep. Private/State Various n.a. n.a. Equity/Loan/TC n.a. Mar-04 Special Study Subtotal 32,835 21,723 Note: The totals may not add up to the sum of the component parts due to rounding. 1 TC=Technical Cooperation

Appendix 4 Page 1 of 9

SELECTED LESSONS LEARNED FROM OPERATIONS EVALUATED IN 2003

I. Lessons on Investment Operations

A. TRANSITION IMPACT

1. Project selection

Smaller SME sub-loans and project size may perform better. Smaller sub-loans in PB-intermediated SME and other loan programmes have a better chance of reaching genuine private entrepreneurs in early transition countries than larger sub-loans. Early transition countries, because of their low income and incomplete privatisation programmes, rarely feature self-made private sector entrepreneurs that can capitalize large projects (greater than USD 500,000) without government involvement. 2. Project Design Ring fenced projects can provide positive demonstration effects. The careful definition of the fence and the type of cooperation agreements between the company inside the ring and the conglomerate outside the fence are crucial factors for a successful cooperation and ultimately a positive demonstration effect. This can be enhanced also by having regular coordination committee meetings and fair pricing of services provided by either party. Early engagement leads to long-term success in MEI projects. Judicious use of TC-funded technical assistance to work with government counter-part agencies early on during project preparation will lead to a more fully developed project concept, and a more successful privatisation for utility services. Enhance backward and forward linkages: As hotel projects procure a variety of services locally the transition impact could be enhanced through a related SME component focussing on such linkages (e.g. garden maintenance, supply of foodstuffs, etc.). These would also contain promotion of local tour operators which organise tailor-made trekking, extreme sports as well as community-based tourism. At the time of project structuring such linkages should be identified and adequately supported either through TCs, SME or other programs. Sector policy dialogue cannot be ignored even in cases of successful transactions. It is important for the Bank to continue a close sector dialogue with the Government, particularly when it is involved in both a fixed line and a mobile operator, and the Bank’s influence can be expected to be greater. The key objectives should be to provide assistance for ensuring a properly functioning telecommunications sector. The Bank should utilise bilateral funds for TC to pay for consultant assistance as required. 3. Appraisal Links between financing and regulatory reform. Tie financing to actual measurable regulatory progress: It makes sense to offer financing for certain types of very basic infrastructure independently of regulatory progress; examples are the backbone network and satellite terminals to support rural access. Even if one cannot prove that the

Appendix 4 Page 2 of 9

investments in such basic infrastructure are commercially viable, the positive secondary effects on the overall economy should dominate. But anything else that is supposed to have a commercial rationale, should receive money only as the appropriate regulatory regime is built up. Appraising project mandate compliance needs more scrutiny and effectiveness. Concerns regarding a proposed project’s compliance with Bank mandate dimensions need early flagging during the project preparatory process and dealt with more rigorously. Project proposals that do not hold adequate potential for mandate compliance, in spite of mitigating efforts, should be abandoned at the earliest possible preparatory stage. Focus on demonstration effect of project at approval. It may be preferable to focus on demonstration effect at project appraisal stage since fulfilment of such an objective is strongly correlated to the project itself while the impact on institutional changes is a function of the outcome of policy dialogue, more remotely related to the project. In this case, the disappointing scoring on legal and institutional framework cast an undeserved shadow on the successful overall performance of the project. 4. Negotiation Consolidated IAS/IFRS financial statements are essential for transparency, sound banking and transition impact. The Bank should require consolidated IAS/IFRS financial statements at the country strategy level and the project level either before proceeding with financing any member of a group of firms in a country, or as a phased and conditioned part of the project. Preparing such financial statements has a strong, positive impact on reforming a business group. Achieving compliance with IFRS compels investment in accounting skills and systems, and strategic business changes and even disposals as firms try to achieve reliable and presentable accounts. The increased corporate transparency makes for more profitable business relationships with suppliers, customers, bankers and the state. These forward and backward linkages, promoted by transparency, benefit all involved parties and further transition. Shaping operational financial covenants for utilities. Financial covenants should be operational and proportionate to the conditions under which a business is run. The targets should be set at levels that can be reached (taking account of binding and not so binding constraints) and against which realistic performance improvements can be measured. In low transition countries, the focus should be on selected indicators of financial strength in cutting edge areas (e.g. cash revenue collection and self financing capability). Financial targets geared towards high standards prevailing in competitive markets make little sense if a supportive incentive framework is missing. Moreover, the Bank should ensure that an appropriate financial accounting and reporting system is established and performance monitoring draws on transparent and reasonably accurate financial data.

5. Project Monitoring

Presidential decrees and EBRD projects. MOUs with leading ministers do not always bind the authorities to an agreed plan of action. Consequently, the Bank should consider requiring Presidential decrees that approve the basic terms of its major projects as a condition precedent for Project approval by the Board of Directors. Such a practice should reduce the risk that projects whose principal purposes have been agreed with the authorities lose momentum due to subsequent changes of policy or personnel.

Appendix 4 Page 3 of 9

Good demonstration effect can enhance policy dialogue. Policy Dialogue can be significantly enhanced through the demonstration effect of successful projects which also better justifies the cost involved in time spent on carrying it out. This is however a one-way proposition and the outcome of Policy Dialogue results from many other factors, often independent from the project itself. Assessing transition impact in SME projects. In early transition countries, assessment of transition impact of SME projects channelled through financial intermediaries requires accurate and robust reporting on sub-loan performance and sub-project ownership. Projects that fail financially and do not operate have little transition impact. Rescheduling of sub-loans in response to payment problems and insolvency may disguise the degree of distress in the loan portfolio and lead to overoptimistic assessment of transition impact. While it is not appropriate to press TC on an unwilling recipient, the Bank should be proactive in assessing a client’s training needs and in explaining the benefits to the client. In the present case TC for training was offered during negotiations but declined by the client who expressed scepticism about the benefits. In retrospect, this scepticism may have been rooted in a lack of confidence rather than a careful analysis of training needs. Subsequently the client acknowledged the benefits of training that took place at a later date as part of another operation with the client. These benefits may have accrued earlier if the Bank had presented the case more persuasively with the client when implementing the earlier project. Responding to change of ownership control of EBRD’s investments. Any material change of ownership of one of the Bank’s investments should trigger a review of project documents by the OL and OGC to determine if the change of control affects the Bank’s contractual rights. This is even more vital when the Bank is a shareholder, sits on the company’s board, when minority investors are involved, or when the stock is publicly listed. Also, the Bank should be satisfied that the change of ownership has occurred in a transparent and lawful manner that protects the rights of minority shareholders and is consistent with the Bank’s objectives in the Project. 6. Role of sponsor

The purpose of debarterisation is to enhance liquidity and stimulate probity. The Bank can play a useful role in providing adequate liquidity. The experience and skills of the Sponsor are an additional requirement to make the process successful and restore transparency and integrity. In this particular case, the Bank could rely on the sponsor’s in-depth knowledge of a prior similar experience with another Bank’s financed project. Such an additional comfort is of critical importance to mitigate the risk of a project failure in this area.

B. ADDITIONALITY The Bank must seek to optimise pricing in every situation where its additionality is demonstrated and where financial leverage would not make it punitive to the project. Successful projects typically require significant use of Bank resources upfront which must be rewarded in the early years of their life. This is usually achievable when the Bank is truly additional and the Sponsor is determined to lean on the Bank’s expertise and clout in the country of operation. It is critical as it will serve as a benchmark for a possible re-pricing, should the project eventually so deserve, while the Bank will have been adequately

Appendix 4 Page 4 of 9

compensated for its early contribution. High margins may affect the Sponsor’s return on his investment, or indeed achieve a better balance between the Bank’s return on its loan and the Sponsor’s own distributable profits. Unless financial leverage is excessive, there should be sufficient upside for an appropriately generous remuneration of the Bank not to be detrimental to the project’s financial viability. Sectoral expertise can sometimes rival Preferred Creditor Status as a driver of the Bank’s additionality. In this project, the Sponsor was less interested in the incremental legal protection that the Bank may bring about, if structured appropriately, than in the overall comfort of its reputation combined with sector expertise. Such proven additionality may be equally conducive to specific conditionality in the loan agreement.

C. ENVIRONMENT

1. Appraisal Best practice with respect to Public Consultation. The banking team must ascertain that when documents are displayed for public consultation, both at the Bank’s RO and at the Sponsor’s local office, the latter is fully informed of the requirements of public consultation and volunteer to co-operate beyond the strict requirements of displaying the document. The role of public disclosure and consultation process needs continued attention in complex projects: Civil society dimensions from initial public disclosure and consultation to on-going dialogue, may change during the life cycle of the project. NGOs are well aware of large extractive industry projects and serve a valuable QA/QC function, making sponsors and the Bank accountable not only to their respective boards, the investors, and project beneficiaries, but also to non-investor stakeholders who are directly impacted by the project. This project has and continues to directly contribute positively to the Bank’s changing public disclosure and consultation policies and procedures. SEAs should be conducted when a project’s impacts are regional and there may be cumulative impacts from multiple projects: When a Bank-supported project is part of a larger development initiative, the Bank should consider supporting the completion of a SEA, to better inform local officials and project sponsors as to regional and cumulative impacts. Pilot project EIAs need to address the potential impacts of all planned future phases: Phasing of large extractive industry projects may have advantages for financial packaging, but environmental documents for the initial phases need to also address the major expected impacts of all phases. While the details of the later phases could be addressed once the Bank has obtained a mandate for a continued role, many issues raised by civil society in a pilot phase of an extractive industry project may well be about planned activities in later phases. In this project, detailed plans about follow-on phases were not available during project preparation. Strengthening of local institutions responsible for coping with huge extractive industry projects: It is important that creative use of TC funds is made at an early stage during project structuring in order to help strengthening the local administration considering the extreme challenges they are facing, based on their demanding regular tasks combined with the management of the impact of world class oil and gas projects.

Appendix 4 Page 5 of 9

Importance of a regional development focus from the outset in line with IFIs best operating standards: Large extractive industry projects cannot be viewed as stand-alone Greenfield projects with limited environmental and transition analysis of the immediate impacts. In frontier regions where an oil and gas project will become the major employer and the chief source of revenue for the region, the regional development aspects must be considered from the outset. The Bank needs to develop internal mechanisms to co-ordinate and respond to the regional development needs in line with best operating standards by IFIs.

2. Negotiation The Importance of environmental performance during restructuring. The Bank has limited leverage over a company once the initial loan is disbursed. However, restructuring is another opportunity where the Bank may have renewed leverage to pursue objectives and ensure that our mandate is fulfilled. Efforts to achieve environmental objectives should not be abandoned. 3. Project Monitoring Importance of promoting training access and to monitor OHS risks. Occupational health and safety is a vulnerability issue for our project sponsors and for EBRD. Therefore it is essential for EBRD to consider enhancing its internal capacity to promote training access and monitor OHS risks. This could be accomplished by ED in using the services of an OHS consultant to work with staff and project sponsors to reduce OHS risks and thereby reduce the Bank’s vulnerability in this issue (conceivably a trust fund proposition). Efficiency of energy and environmental management. The financial implications of efficient use of energy and environmental management are often poorly understood and the status quo too easily accepted. Energy audit agreements should require firms to select and recommend proposed projects to their board and report the subsequent board decision to the Bank. D. SOUND BANKING

1. Project Design Consistency of proposed objectives in EBRD Board Reports and Project Agreements. Banking Teams should ensure internal consistency between EBRD documents and agreements the Project Sponsor may have made with government agencies and third parties. The Bank’s objectives should not exceed those to which the Company has committed itself, or if external Sponsor commitments are not sufficient, renegotiate with the Sponsor and the Government to reach objectives acceptable to all parties. Bankruptcy remote structures and the legal ring fence. Rarely will a legal ring fence or bankruptcy remote structure permit sound banking when working with distressed firm. Achieving a bankruptcy remote borrower through a legal ring fence is a delicate and risky exercise in the best of circumstances. It usually requires setting up a single purpose company or trust and depends on predictable legal risks as are found in established jurisdictions. It is hard to meet the strict legal criteria in a reliable way under the best of circumstances, and even more difficult when trying to recover a bad loan to an existing corporation in a legal system in transition. Ring fencing requires removing control of the assets from the bankruptcy-susceptible owner to an independently owned and controlled firm. It therefore will depend on assent of the owner and other creditors who may be left

Appendix 4 Page 6 of 9

outside the restructuring; this dependency on assent leaves the restructuring process hostage to stalling by the owners and other creditors. Finally, any ring fence solution, once implemented, may be vulnerable to legal attack in the event of bankruptcy. In some circumstances it may be better to go for a quicker cash settlement than to negotiate a ring fence that may take years to implement and risks not living up to expectations, especially in transition countries with inadequate legal regimes for insolvency. 2. Appraisal Due diligence must go into the details of who will be put in charge and the adequacy of past experience within the Sponsor’s group with the challenges of the Project. It is not sufficient to rely on a sponsor’s track record. Individual expertise and performance history must match the specific requirements of a project in terms of sector as well as international experience for each key position of the management team.

3. Project Monitoring

Cash settlements versus restructuring of troubled loans in uncertain legal environments. The legal environment in the country may not favour a long term restructuring. A weak insolvency regime that does not protect creditor rights may encourage stalling by debtors. It may be superior to demand immediate repayment and seek to enforce guarantees as a first step, while values are highest, and while performing due diligence into a restructuring as a fall back approach should the payment demands fail. 4. Role of sponsor Dedication and experience are keys to the success of turning around complex manufacturing situations. The Sponsor group had proven elsewhere in the world, among others in one of the Bank’s projects, its ability to undertake the challenge to turn around large steel conglomerates. As stakeholders through equity and loan support, the Sponsor brings to the project more dedication than would be expected from management contractors. In the selection process, this gave him a definite edge over longer established but less committed Western sponsors. E. EXIT STRATEGY

Ensuring timely exit from equity investments requires constant, independent monitoring of the exit strategy. Operation leaders may be biased against timely exit in hope of still greater gains, and especially if an exit would result in a loss. Clear exit strategies, tightly monitored and re-evaluated by a central manager with portfolio responsibility, might avoid missed opportunities and reduce the risk of loss. Avoid lock-up situations in equity deals. Avoid lock-up and other limitations on the right to exit minority equity positions, especially if such rules do not govern all investors equally. Lock-ups impede exit and remove a source of negotiating leverage in dealing with other shareholders.

Appendix 4 Page 7 of 9

II. Lessons on Technical Cooperation (TC) Operations A. GENERAL CONSIDERATIONS Project versus Programme Approach needs to be given more attention. Before embarking on a string of individual project undertakings that are inter-related or are characterised by commonalties (such as support to inter-related ports), thorough consideration needs to be given whether a programme approach is preferable over that of pursuing a series of projects individually. Pros and Cons for both approaches exist, including project delivery expedience, administrative constraints, synergies, cost savings, and impacts and their sustainability. Reaching the critical level of policy leverage. In early transition economies, the social, political and economic context should be properly assessed so that Bank resources are not tied up in slow reforming sectors, endangering the longer term benefits of technical assistance. An active policy dialogue in close co-ordination with the other IFI’s is also important to reach the critical level of policy leverage that will trigger reforms. Constraints of narrow project approach for TC versus a required wider sector approach. The TC focus on a project should not be oblivious to the sector requirements. Whilst the Bank’s project approach may facilitate continued engagement with the authorities, there is a risk that lagging sector reform hampers progress on the project level. Effective coordination with all IFIs is particularly important in cases where a major sector reform is imperative. B. TC PREPARATION For railway rehabilitation, identification of investment priorities in the absence of business plans is sub-optimal. For a successful railway restructuring, the Government’s long-term transport policy and priorities must be known and enter the business plan. TC work for identifying investment priorities need to build on these inputs. Conversely, in the absence of such guidance the investment process is jeopardised and, more likely, results in sub-optimal investment outcomes. The Bank would need to ensure that all strategic inputs are in place prior to its TC commencement, or to start with the necessary predecessor work before focusing on further down-stream processes. Market studies for major commodities and key supplies for local industries are necessary complements of traffic studies. A detailed market analysis (besides technical and engineering analysis) to capture major supplies for key local industrial complexes is imperative as a basis for investment programmes in railways, even if maintaining of the status quo constitutes the base case scenario. Relative importance of individual tasks needs to be clarified in TOR. The degree of detail of sections within consultant TORs should be in correlation with the relative importance of the respective tasks for the expected outcome. In turn, required consultant staff qualification and seniority should reflect the relative importance of the distinct tasks. Ensure that adequate staff is allocated by the client to TC supported activities. At the project design and negotiation stage, the Bank should consider the introduction of conditionalities setting out the minimum numbers of staff to be assigned by the client to the supported activities and qualifications of the staff. The client must allocate sufficient staff to ensure the successful assimilation and transfer of know-how from TC operations. Staff retention is also an important concern.

Appendix 4 Page 8 of 9 Provide TC for sector wide analysis. When a project supports an activity which heavily depends upon one form of primary input (crude oil) and a limited range of outputs (gasoline, diesel), it appears necessary to provide TC on a sector wide analysis on forward and backward linkages. Availability and costs of inputs as well as demand and price of output should be included in a thorough market analysis. This would help assess the conditions in which a transformation process can reach production levels close to full capacity. C. POST-IMPLEMENTATION SUPPORT In early transition countries extended TC input may be required at individual intermediary banks to sustain achievements in credit advisory services. In an environment where the pace of development of bank regulation and supervision is slow, individual banks may need the encouragement and support of continuous focussed TC to secure the sustainable transfer of sound lending skills. This support is necessary to foster the development of technical expertise, although the full benefits may not begin to show until the environment improves sufficiently to encourage intermediation and promote growth in business volumes. D. TC-SUPPORTED ENVIRONMENTAL WORK Use of TC for pilot tests to enhance environmental performance in companies borrowing under environmental credit lines may contribute to learning. Pilot tests are a useful way to better understand complex market forces that support or constrain investments in environmental improvements. Minimise complexity in large TC operations to the extent possible. For credit line operations, rather than involving several environmental contractors under a Bank TC for sub-loan appraisal work, the environmental due diligence function could have also been incorporated directly into the intermediary bank practices over the life of the credit line, as a side-effect of which would have been a greater capacity building for the financial intermediary involved. Providing support for environmental audits may help expand environmental investment opportunities. Either directly, through the use of TC funds, or through collaboration with other donors more active in the sector, EBRD could help facilitate environmental audits to provide project specific information to project sponsors. This could be financially structured through TC operations to provide support to local financial institutions which are promoting “green” portfolios, while also providing support and institutional developing to emerging local environmental consulting services. E. INVESTMENT FACILITATION Adhere to covenants on commercialisation and privatisation. In early transition economies where centralized decision making dominates economic management, the covenants on commercialisation and privatisation could be more successfully adhered to when associated with TC that could show the various steps to help fulfil these covenants. The TC would include an analysis of the markets, product prices and ownership structure in the sector where the project must be implemented focusing on those elements which have a direct impact on the desired benefits of the project. The TC would also indicate the intermediary steps that could be followed towards the fulfilment of the covenants. This analysis, if done at the early stages of project preparation, could also help formulating proper covenants by narrowing down the measures that can realistically be taken during the life of project implementation.

Appendix 4 Page 9 of 9

III. Lessons learned from the Special Study on PCR Assessments in respect of TC Operations The table below presents the lessons learned from the evaluation special study on Project Completion (PCR) Assessments carried out by PED in 2003 on TC operations.

Lessons Learned in TC operations

Lessons Learned Client Commitment Examples on best practices

Client does not provide clear go-ahead for TC - Reformulate objectives or approach of TCs - Monitor at an early stage the commitment of the Client

Client uncertain about how political environment will unfold

- Use sufficient time in the preparation phase - Policy dialogue at higher level to follow events

Client does not understand the benefits of the TC

- Coordinate visits to other relevant countries - Share experience of similar TCs in other countries

Client does not collaborate with effort once launched

- Shared payment of TC to avoid “free riding” - Meet with management/shareholders - Identify and manage key persons that block the process

Client does not invest enough time in managing TC

Simplify procedures and target alternative person(s) in host country

Client inexperienced in managing consultants Provide advise on how to manage consultants Client does not provide staff to assist the Consultant

Make explicit demands at the outset

Client does not pay Consultant Explore alternative payment procedures Client concerned about ambitious program - Discuss proactively concerns

- If needed, modify TOR Consultant Performance Examples on best practices

Consultant does not have relevant skills/ delivers a shallow report

- Systematic evaluation and follow-up of Bank’s experience with consulting firms, by country/sector/topic

- Conduct (phone) interviews before selecting Consultant’s in smaller TCs

- Consider comprehensive selection procedure in complicated and/or difficult (innovative) assignments under 200,000 Euro

- Assess carefully the capabilities of the team provided by consulting firms

Consultant changes staff along the way - Penalties for using staff other than committed in proposal - Only allow replacements if there are pressing needs, or if

staff is substituted for somebody with similar capabilities Consultant’s emphasis is not in line with Bank requirements

- Kick-off meeting at the outset to clarify objectives and output

- Maintain regular contact with the Consultant Consultant enters conflict with Client - Include Change Management aspects in Consultant’s ToR

- Facilitate Consultant/Client relationship at the outset - Address immediately the cause for conflict/ spot

weaknesses early Consultant enters conflict with other firms in the Consultant consortium

- Refer to clauses for termination in contract - Facilitate TC procedure to allow for timely separate TC

applications for different tasks related to the same project (e.g. technical, environmental)

Bank Performance Examples on best practices Partially unclear standards for Bank monitoring

- Clear guidelines and policies on Bank monitoring - Guidelines related to sector/topic

Poor or incomplete TC design - Develop Lessons Learned database and training for TCs, including experience from other MDBs, IFIs

- Clearly state beforehand the risks of the TC, anticipate unforeseen events and how to deal with them

Poor coordination between TC and Policy Dialogue effort

Coordinate Policy Dialogue further with TC efforts

Inadequate changing of OL staff, which leads to inexperience of staff

Clear transition management guidelines when changing OL staff

Poor project management performance - Management training - Workshops - Time management training

Appendix 5 Page 1 of 3

INVESTMENT OPERATION DATABASE

1 THE DATABASE In 1999 PED's evaluation database was incorporated into the Bank’s Data Warehouse System, allowing easier and more efficient reporting on various aspects of the evaluated database. In particular it is now possible to apply the Bank’s investment operation counting methodology to the evaluated database1.

Table 1: Investment Operation Database Summary

(as at 31 December 2003)

All investment operations No. % €million % Bank Portfolio Board-approved 1,111.4 109% 26,268 116% Signed 1,017.0 100% 22,668 100% 12 Months + Past Board Approval 978.4 96% 22,252 98% Of which subject to Evaluation Total ready for Post-Evaluation 543.1 53% 10,836 48% With OPER Reports 207.3 20% 4,970 22% With XMR Assessment 216.9 21% 3,431 15% With XMR Review 118.9 12% 2,435 11%

PRIVATE SECTOR No. % €million % Bank Portfolio Board-approved 896.1 110% 19,571 119% Signed 811.3 100% 16,383 100% 12 Months + Past Board Approval 786.2 97% 16,466 101% Of which subject to Evaluation Total ready for Post-Evaluation 447.5 55% 8,235 50% With OPER Reports 180.3 22% 4,138 25% With XMR Assessments 183.8 23% 2,609 16% With XMR Review 83.4 10% 1,488 9%

PUBLIC SECTOR No. % €million % Bank Portfolio Board-approved 215.3 105% 6,697 107% Signed 205.7 100% 6,285 100% 12 Months + Past Board Approval 192.2 93% 5,786 92% Of which subject to Evaluation Total ready for Post-Evaluation 95.6 46% 2,601 41% With OPER Reports 27.0 13% 832 13% With XMR Assessments 33.1 16% 822 13% With XMR Review 35.5 17% 947 15%

Source: Data Warehouse and Board Monthly Information Report, December 2003 Table 1 outlines PED's evaluations of investment operations against the Bank's total investment operations portfolio at the end of 2003. The investment operation count applied by PED correspond to 174 Operation Performance Evaluation Reviews (OPERs), 211 Expanded 1 Bank Operation Count (Opcount) = a measure of the number of full operations. An operation may have a fractional Opcount (i.e. less than 1) if it is a framework agreement or subproject of a framework agreement, in which case the overall operation count of a framework agreement and all its respective sub-projects will total 1 (OpCount in these cases is proportional to the value of the operations). It may also be less than 1 if the operation concerned is closely linked with another operation i.e. a loan increase, where again the total operation count of the related operations would be 1. If only part of an operation is signed then the Opcount of ‘Signed Operations’ will also be fractional as Opcount is split equally within an operation by the number of facilities in DTM.

Appendix 5 Page 2 of 3 Monitoring Report (XMR) Assessments and 123 XMR Reviews carried out during 1993-2003, in total 508 ready operations2. 1.1 PORTFOLIO PROFILE AND EVALUATION COVERAGE 1.1.1 Part of the portfolio ready for evaluation. By 2003, 543.1 investment operations (447.55 private sector and 95.6 state sector) were judged ready for post-evaluation (in accordance with prevailing Operations Manual (OM) procedures). In total 174 OPERs, 211 XMR assessments and 123 XMR Reviews cover 100 per cent of operations ready for post-evaluation up to 2003 (this equates to 704 distinct operation identity numbers being evaluated). Nineteen OPERs on investment operations were produced in the 2003 plan (comprising only 27 per cent of operations ready for post-evaluation). Therefore, a selection process for the 19 OPERs took place using criteria such as large Bank exposures, transition impact relevance, complexity, lessons learned potential, large state sector and classified operations. 1.1.2 Growing numbers of ready operations and evaluation coverage ratio. Chart 1 indicates that the number of projects ready for post-evaluation fell slightly in years 2000 and 2002 but is predicted to rise to 95 per year in 2010. However, in reality these operations may well be more evenly distributed over the adjacent years. For 2004 the total number of operations identified as ready for post-evaluation is 77.

2 In total 176 OPER reports have been produced. However, one report from 2001 revisited an operation already evaluated in 1996, and another, from 2003, evaluated the progress of a broad client relationship rather than evaluating any individual operations. Therefore these two OPERs have been discounted throughout this AEOR when considering the number of operations evaluated.

Chart 1: OPER Coverage of Investment Operations Ready for Post-Evaluation 1993-2010

PER YEAR

0

20

40

60

80

100

120

1993

1995

1997

1999

2001

2003

2005

2007

2009

No.

of E

valu

atio

ns

OPER + XMRA + XMR Review report production

OPER ReportsOPER + XMR Assessment report production

CUMULATIVE

0

200

400

600

800

1000

1200

1993

1995

1997

1999

2001

2003

2005

2007

2009

No.

of E

valu

atio

ns

OPER + XMRA + XMR Review report production

OPER reportsOPER + XMR Assessment coverage

Appendix 5 Page 3 of 3

1.2 OVERVIEW OF THE INVESTMENT EVALUATION DATABASE 1.2.1 Timing of the evaluations. The 543.1 operations were evaluated, on average, 18 months after final disbursement of the Bank's investments and 48 months after Board approval. Furthermore, 9% of operations were appraised and Board-approved during the first and second year of the Bank's existence when little institution quality control had been established and the Bank's learning curve was still at an early stage. The high attrition of Bank staff is another factor. Many staff involved in projects leave the Bank by the time evaluation takes place. This complicates evaluation. The OPER execution timing is therefore geared to providing an early post-evaluation performance feedback for lessons, quality management and accountability, and in most cases is an assessment which occurs relatively early in the operating period. It follows, however, that not all evaluation judgements can be final, particularly for assessments concerning transition impact and equity investment performance. 1.2.2. The representivity of the sample of selected operations. The 543.1 operations subjected to OPERs, XMR Assessments and XMR Reviews - and therefore the operations' performance ratings, patterns and lessons summarised in this Report - comprise a growing portion of the Bank's Board-approved portfolio. By the end of 2003 it already has reached a cumulative evaluation coverage level of 53% of the total number of signed operations and 48% in value terms. As evaluation coverage levels increase the relevance of the evaluation findings for the portfolio as a whole also increases (see also the Addendum of Appendix 8). 1.2.3 The composition of the evaluated portfolio. Among the 384.7 post-evaluated loan investments, 94.0 facilities are fully pre-paid and 123.7 facilities are fully repaid, while 10.5 operations have been written off. Of the equity investments, 14.2 operations have been written off, 62.0 operations have fully divested facilities and 22.8 operations have partially divested facilities. 1.2.4 Diversification of the selected operations for evaluation. Of the evaluated investment portfolio, there is a reasonably diversified balance of: − Countries. The following 26 countries were covered: Albania (1.5%), Armenia (0.6%),

Azerbaijan (1.7%), Belarus (1.1%), Bosnia & Herzegovina (1.6%), Bulgaria (4.4%), Croatia (4.6%), Czech Republic (3.7%), Estonia (5.4%), FYR Macedonia (1.8%), Georgia (1.7%), Hungary (7.7%), Kazakhstan (1.3%), Kyrgyz Republic (1.7%), Latvia (3.4%), Lithuania (3.2%), Moldova (1.7%), Poland (12.4%), Romania (8.4%), Russian Federation (14.4%), the Slovak Republic (4.3%), Slovenia (3.8%), Tajikistan (0.7%), Turkmenistan (0.4%), Ukraine (4.2%) and Uzbekistan (1.8%), besides Regional operations (2.6%);

− Operation Teams. Thirteen sector teams have been covered, and all three regional business groups, besides Corporate Recovery;

− Industries (according to Bank standard industry name). In total 56 industries were covered; − Investment types. 57% are straight senior debt operations, 27% are ordinary share

investments, eight per cent are a combination of ordinary shares and senior debt, three per cent are straight subordinated debt, two per cent are a combination of senior debt and subordinated debt, the remaining percentage are a combination of the investment products above and guarantees, preference shares, other on balance sheet debt-style facilities and other participating interests.

It can be concluded the evaluated operations cover a great variety of countries, sectors, operation teams and types of deals, which form an excellent basis to generate lessons learned.

Projects rated "Highly Successful" (1 = "Highly Successful" occurrence)

Financial Factor commercial Factor Institutional Factor External factor

Project name Board date Sector

Financial analysis

cost performance

Sales performance

Market analysis competition

sponsor commitment Management Governance

Business cycle

Government behaviour

Bank handling

Commercial bank 1993 FI 1 1 1 1

Commercial bank 1993 FI 1 1 1 1 1 1

Equity fund 1992 FI 1 1 1 1 1

Commercial bank 1993 FI 1 1 1 1 1

Commercial bank 1995 FI 1 1 1 1 1

Commercial bank 1998 FI 1 1 1

Commercial bank 1998 FI 1 1 1 1

Pharmaceuticals company 1993 GEN IND 1 1 1 1 1 1 1 1

Pharmaceuticals company 1995 GEN IND 1 1 1 1

Steel plant 1997 GEN IND 1 1 1 1 1 1 1

Zinc Plant 2000 GEN IND 1 1 1 1

Food manufacturer 1991 SP IND 1 1 1 1 1

Bottling plant 1993 SP IND 1 1 1 1 1 1

Business Centre 1994 SP IND 1 1 1 1 1

Telecommunications company 1994 SP IND 1 1 1 1 1 1

Telecommunications company 1993 SP IND 1 1 1 1 1 1

Power Project 1993 ENER 1 1

Shipping company 1994 INF 1 1 1 1

Appendix 6.1

Table 1: Projects rated "Unsuccessful" (1 = "Unsuccessful" occurrence)

Financial factor commercial factor Institutional factor External factor

Project name Board date Sector

Financial analysis

cost overrrun

Sales performance

Market analysis competition

sponsor commitment Management Governance

Business cycle

Government behaviour

Bank handling

Commercial bank 1992 FI 1 1 1 1 1 1

Commercial bank 1994 FI 1 1 1

Commercial bank 1994 FI 1 1 1 1

Commercial bank 1995 FI 1 1 1

Equity Fund 1995 FI 1 1 1 1

Agribusiness company 1996 FI 1 1 1

Commercial bank 1996 FI 1 1 1 1

Commercial bank 1997 FI 1 1 1

Computer and electronics manufacturer

1991 GEN IND 1 1 1 1 1 1 1 1

Motorcycle manufacturer 1992 GEN IND 1 1 1 1

Non-metallic mineral plant 1994 GEN IND 1 1 1 1 1

Petrol refinery 1994 GEN IND 1 1

Truck manufacturer 1995 GEN IND 1 1 1 1 1 1

Shoe manufacturer 1995 GEN IND 1 1 1 1 1

Truck manufacturer 1995 GEN IND 1 1 1 1 1 1

Radiator manufacturer 1997 GEN IND 1 1 1 1

Steel plant 1997 GEN IND 1 1 1 1 1

Printing house 1997 GEN IND 1 1 1 1

Steel plant 1997 GEN IND 1 1 1

Appendix 6.2Page 1 of2

Appendix 6.2

Page 2 of 2

Table 1 - Continued: Projects rated "Unsuccessful" (1 = "Unsuccessful" occurrence)

Financial factor commercial factor Institutional factor External factor Project name Board

date Sector Financial analysis

cost overrrun

Sales performance

Market analysis competition

sponsor commitment Management Governance

Business cycle

Government behaviour

Bank handling

Cold storage facility 1991 SP IND 1 1 1 1 1 1 1

Cold storage facility 1991 SP IND 1 1 1 1 1 1 1

Frozen fruit and vegetable producer

1993 SP IND 1 1 1 1

Wine producer 1994 SP IND 1 1 1 1 1

Agribusiness Company 1995 SP IND 1 1 1 1 1

Fruit and vegetable processor 1996 SP IND 1 1 1 1

Fruit and vegetable processor 1997 SP IND 1 1 1

Food manufacturer 1998 SP IND 1 1 1 1 1

Power plant 1993 ENER 1 1 1

Mining company 1993 ENER 1 1 1 1 1

Airline company 1992 INF 1 1 1 1 1

Road rehabilitation 1993 INF 1 1 1 1 1

Railiway financing 1994 INF 1 1 1

Road rehabilitation 1995 INF 1 1 1

Shipping company 1996 INF 1 1 1 1 1 1

Appendix 7.1 Page 1 of 2

ASSESSMENT OF STRENGTH OF TRANSITION POTENTIAL & CHECKLIST OF TRANSITION CRITERIA/OBJECTIVES

FOR EX ANTE AND EX POST APPLICATION

ASSESSMENT OF STRENGTH OF TRANSITION POTENTIAL

1. COUNTRY SECTOR AND REGIONAL CONTEXT

a. Current stage of transition (advance transition country or otherwise) b. State of sector reform and development (largely unreformed or otherwise) c. Conditions for market entry and competition (few players versus strong competitive

pressures)

2. THE TRANSITION CHALLENGES FACING SECTOR, COUNTRY AND REGION

a. Market reform objectives in the Bank's country or sector strategy b. Economic priorities facing the country c. Application of the transition indicators (TI Checklist) - Structure and extent of markets - Market organisations, institutions and policies that support markets - Business behaviour and practices

3. THE WAY CHALLENGES ARE ADDRESSED IN THE SELECTION AND DESIGN OF THE PROJECT

a. Consistency with Bank country/sector strategy; b. Key project covenants and undertakings (strong set of transition-related covenants

is likely to be a sufficient sign of transition potential; it is not a necessary condition); c. TC components (TC-funded programmes that can help achieve some of the transition

objectives); d. Policy dialogue

CHECKLIST OF SEVEN TRANSITION CRITERIA/OBJECTIVES

PROJECT CONTRIBUTIONS TO THE STRUCTURE AND EXTENT OF MARKETS

1. GREATER COMPETITIVE PRESSURES Project contributes to greater competition in the project sector: efficiency, innovation

and customer orientation of other suppliers through competitive pressure. To what extent does the project directly improve the competitive environment and/or

extend the use of market-type mechanisms in the economy? (e.g. more rational pricing, significant new entry into the market, setting new quality or technical standards that other firms must follow, trade facilitation, etc.)

2. MARKET EXPANSION VIA LINKAGES TO SUPPLIERS AND CUSTOMERS Stimulation of competitive behaviour through the project entity's interactions with

suppliers (backward/upstream linkages) and clients (forward/downstream linkages); project contributions to the integration of economic activities into the national, regional or international economy, in particular by lowering the cost of transactions.

(a) To what extent does the project change the market behaviour of local suppliers of inputs? (backward linkages);

(b) To what extent does the project change the market behaviour of downstream marketing and/or processing activities of customers? (forward linkages)

APPENDIX 2 Page 2 of 2

CHECKLIST OF TRANSITION CRITERIA/OBJECTIVES (CONT.)

PROJECT CONTRIBUTIONS TO MARKET ORGANISATIONS, INSTITUTIONS

AND POLICIES THAT SUPPORT MARKETS

3. INCREASED PRIVATE SECTOR PARTICIPATION Significant increase or consolidation of private provision of goods and services,

including provision of public goods and services and support for entrepreneurial initiative (e.g. unbundling in infrastructure projects).

To what extent does the project contribute directly to increased private ownership?

4. INSTITUTIONS, LAWS, REGULATIONS AND POLICIES THAT PROMOTE MARKET FUNCTIONING AND EFFICIENCY

Creation/strengthening of public and private institutions that support the efficiency of markets; improvements to the functioning of regulatory entities and practices; contributions to government policy formation and commitment, promoting competition, predictability and transparency; contributions to laws that strengthen the private sector and the open economy. Improved legislation, regulation and legal and regulatory implementation.

To what extent is the project associated with institutional spin-offs effects giving rise to improvements in the functioning of existing institutions or in the establishment of new institutions and practices important for a market-type economy?

PROJECT CONTRIBUTIONS TO BUSINESS BEHAVIOUR AND PRACTICES

5. TRANSFER AND DISPERSION OF SKILLS Project contributes to significant upgrading of technical and managerial skills in the

economy beyond the project entity. To what extent does the project create, upgrade or transfer new skills relevant to a market

economy? (e.g. management, marketing, financial and banking skills, specialised technical skills, etc.)

6. DEMONSTRATION EFFECTS FROM INNOVATION Demonstration of (replicable) products and processes which are new to the economy;

demonstration of ways of successfully restructuring companies and institutions; demonstration to both domestic and foreign financiers of ways and instruments to finance activities. New ways of financing restructuring instruments.

To what extent does the project create a new and easily replicable line of activity? (demonstration effects, e.g. in manufacturing or finance, incl. new modes of financing industrial projects, new products, enterprise restructuring)

7. HIGHER STANDARDS OF CORPORATE GOVERNANCE AND BUSINESS CONDUCT Improved governance standards that are highly visible and invite replication in non-

project entities. To what extent does the project give rise to improvements in corporate governance and/or

the business culture? (incl. fostering entrepreneurship, improving decision-making processes, encouraging innovation and strategic thinking in business)

APPENDIX 7.2 Page 1 of 19

ENERGY DISTRIBUTION TRANSITION IMPACT ANALYSIS

COMMENTS

SHORT- TERM

VERIFIED IMPACT

LONGER- TERM

IMPACT POTENTIAL

RISK TO POTENTIAL TRANSITION

IMPACT

STEP I: CHANGE BY THE PROJECT AT CORPORATE LEVEL Rating1 Rating2 Rating3 1 Private Ownership: The privatisation of its largest electricity

distribution utility produced the first major foreign investment in the energy sector in the host country. The Sponsor won a tender, which failed to attract more than two bidders, on the basis of a low anticipated IRR offer. Although this may have been partly due to a misevaluation of the size of investments required to turn the company around, It could only have been justified by the Sponsor’s strategy to use such an investment as a stepping stone for ultimate regional expansion. Significant demonstration effect was expected from this investment to foster foreign direct investment in the sector and further privatisation in the upstream generation industry. As a result of the impact of corrupt practices on the project, in spite of commendable efforts to eradicate them from within, ownership was eventually transferred to the single party interested in taking over the Company. This resulted in a considerable financial prejudice for the foreign investor and created the worst possible demonstration effect on any potential new investors. The losses incurred by the Sponsor have nevertheless benefited the new owner who has bought the Company for a cash price less than that paid at the time of privatisation despite debt assumption and considerable new investments realised by the latter. The new owner is regarded as non public sector corporation, notwithstanding the majority control still exercised by the government of its country of incorporation. None of the anticipated privatisation in the sector has taken place, but for the acquisition/management contract by the Company of two generation units to secure its independence from unreliable government owned suppliers. This was not the kind of privatisation that was envisaged earlier on to stimulate competition in the energy industry. Because it could be hoped that the new owner will continue the Sponsor’s policy of tariff enforcement, collection and reduction in commercial and technical losses, which would be all benefit to it while the investment cost was borne by its predecessor, the medium term outlook of transition impact potential is better than the short term verified impact. The risks of falling short of earlier expectations remain high.

Unsatisfactory Marginal High

1 Ratings Scale: Excellent, Good, Satisfactory, Marginal, Unsatisfactory, Negative, N/A. 2 Ratings Scale: Excellent, Good, Satisfactory, Marginal, Unsatisfactory, Negative, N/A. 3 Ratings Scale: Low, Medium, High, Excessive, N/A

APPENDIX 7.2 Page 2 of 19

2 s to the Company. Most salient features included:

Improving and updating the interoffice telecommunications system.

Installing of a Management Information System and introducing corporate emailing.

Development of billing and collection process where non-paying customers were disconnected.

Introduction of international audit of financial statements. Technical assistance was provided in accordance with the provisions of a contract with an affiliate company of the Sponsor. The Company was able to improve its methodology for demand forecast, billing and collection process, accounting and health and safety standards. It can be expected that such know how will remain in the new company in the long run, with a medium risk of skills erosion through the loss of

Good Good Medium Know How: The project introduced new financial and technical skill

technical assistance from a major international expert in the field. 3 ct

in an

eporting and auditing practices were successfully

int duc

Good Good High New standards for business conduct at enterprise level: The projeimplemented with success new standards for consumption metering, billing and cash collection. It actively fought technical and commerciallosses. This determined action against corruption could hardly extend beyond the Company’s corporate boundaries. It provides an indicationof the limited extent to which business conduct may be improved environment hostile to integrity and transparency. Internationalaccounting, r

ro ed.

STEP II: TRANSITIO EVEL AND ON THE N IMPACT AT INDUSTRY LECONOMY AS A WHOLE

1

y

that

gh , given the current state of the

Satisfactory Satisfactory High Competition: The Company was operating a monopoly activity whichlimits opportunity for competition. The Energy legislation provides however for third party access, with a view to developing eventuallsuppliers’ choice. The Company’s financials indicate that a small portion of revenues come from wheeling charges which confirmsome use, albeit limited, of third party access did occur. More disappointing though was the absence of privatisation upstream, to create greater competition between suppliers. Indeed the upstream integration resulting from the acquisition of / control over additional generation capacity could be seen as inhibiting competition althousuch impact is a longer term threatenergy sector in the country.

2

lt to attribute to

N/A N/A N/A Market expansion via competitive interaction in the sector and industry: Because of the limited competitive impact of the project and the absence of any follow-on privatisations, it is difficuthe project any particular impact in this respect.

3

t

e

ves

but the negative demonstration

Marginal Marginal High Frameworks for markets; institutions, laws and policies that promote market function and efficiency: The framework for the sector is well advanced and the Sponsor’s bid was the first significanstep in implementing such a framework through privatisation of the largest distribution company in the country. This was meant to be followed by the rationalisation of other distribution companies, thimplementation of a fluid Wholesale Electricity Market which is created but lacks sufficient depth to operate on a fully competitive basis and the privatisation of some of the generation. No privatisation took place and the authority of the independent regulating body leaa lot to be desired. In this disappointing context, the project had a positive, though short lived, impactimpact augurs badly of the future.

4

ial tion

Good Good High Skills transfer and dispersion to the industry and economy as awhole: The company dedicated considerable efforts in effectively training its staff in the areas of billing, collection process and financreporting. This should have a positive impact on other distribu

APPENDIX 7.2 Page 3 of 19

companies which ought to draw lessons from the Company’s achievements in improving cash collection standards. Furthermorthe Company implemented a strict policy of enhanced health and safety procedures, a badly needed improvement in this area which was primarily based on training, incentives and penalties. It remaiseen whether a

e,

ns to be s high standards will be upheld under the new

management. 5

ction by

ew wner may adversely affect the newly enforced payment discipline.

Good Good High Demonstration effects; transfer of new behaviour and patterns: The most important demonstration effect relates to the implementation of electronic billing system and the ability to enforce cash colledisconnecting non-paying customers. The improvement in the Company’s health and safety record is also a result of new behaviourbeing encouraged which should have a lasting demonstration effect. Customers’ resentment about the take over of the Company by the no

6 gh

e

l

ite of

clear future standards of business

Marginal Marginal Medium New standards for business conduct: Corporate governance wasgood at the Company where the sponsor struggled to enforce hiethical standards and eradicate endemic corruption. While thCompany should be commended for the efforts made and itsachievements in terms of employees’ morale as well as self consciousness in terms of enhanced safety procedures, the serious events that led to its decision to withdraw from the host country cannot be ignored. It must be pointed out that unmitigated intimidation as welas physical assault of Company’s employees has sealed the fate of an initially successful campaign to improve business standards. In spthe Sponsor’s dedication, the project’s ultimate record is one of successful intimidation leading to unconduct under the new ownership.

SUMMARY OF VERIFIED, POTENTIAL AND RISK RATINGS Marginal Marginal High

VERALL TRANSITION IMPACT RATING:4 MARGINAL

O

4 Ratings Scale: Excellent, Good, Satisfactory, Marginal, Unsatisfactory, Negative. The Overall Rating is not an

arithmetical mean of individual ratings, since verified impact weighs more than potential impact.

APPENDIX 7.2 Page 4 of 19

AGRICULTURAL COMMODITY PROGRAMME

TRANSITION IMPACT ANALYSIS

COMMENTS

SHORT- TERM

VERIFIED IMPACT

LONGER- TERM

IMPACT POTENTIAL

RISK TO POTENTIAL TRANSITION

IMPACT

STEP I: CHANGE BY THE PROJECT AT CORPORATE LEVEL Rating Rating Rating 1 Private Ownership: Both the Client and the Borrowers are private

companies. All warehouses are also privately owned and the warehousing process is managed and insured through privately owned organisations. The licensing of the warehouses, in turn, is not yet a government regulated process which would sanction an efficiently privately run system. The Project has not therefore contributed meaningfully to the promotion of private ownership but has some potential in this respect through the expected legislative involvement in the licensing of privately owned and managed warehouses.

Marginal Satisfactory Medium

2 Know How: The Company has implemented an effective, albeit costly, mechanism to provide fully secured financing to agricultural commodity purchasers, traders and processors. Its success can be measured by the absence of any delinquency so far and the comfort level built within the Bank which increased its exposure from an initial USD 50 million to the present USD 140 million. This is run by a 22 strong team of professionals. It has involved among others the enforcement of strict standards for the internal licensing of warehouses and the supervision through qualified surveyors of the management of such warehouses as well as the provision of adequate insurance coverage. While it could be regarded globally as second best to proper WHR trade financing, in the absence of a legislative framework making it possible, such know-how allowed for a good substitute, pending implementation of adequate legislation. Long-term impact should become marginal in the hypothetical situation of marked progresses on WHR legislation issues.

Good Marginal Medium

3 New standards for business conduct at enterprise level: The Company is implementing the highest standard in the conduct of the business that can be practically met. This is evidenced by the sophisticated rating system used in the selection of warehouses, the care used in the choice of surveyors carrying out daily inspections on assigned warehouses and by the discipline that the nature of the business imposes on the Borrowers. The latter are increasingly behaving as long term business clients of the Company rather than traders keen on looking for excuses to renege on transactions. Most Borrowers though have not yet reached the size and level of business ethics development that would qualify them for dealing on a non guaranteed basis, hence the importance of conditional off-takes often provided by third parties.

Good Good Low

STEP II: TRANSITION IMPACT AT INDUSTRY LEVEL AND ON THE ECONOMY AS A WHOLE

1 Competition: Given the profitability of the programme and the immaculate default record, it should have generated a lot more appetite from potential competitors. A number of foreign banks have investigated ways to set up similar programmes but have eventually shown reluctance to launch a similar full fledged investment. Domestic institutions have gone as far as hiring an executive from the Client to replicate the programme but so far to no avail. Real competition is to be expected from trade financing based on WHR, if and when granted proper status. This will seal the fate of the presently unchallenged but expensive and labour intensive programme run by the Company.

Marginal Good Medium

APPENDIX 7.2 Page 5 of 19

2 Market expansion via competitive interaction in the sector and industry: The Company’s portfolio has increased significantly over its four years of existence. This is a distinct sign that it is fulfilling a need to financially support agricultural commodity processors and, indirectly through the provision of liquidity, the primary agricultural sector. It should also be noted that the portfolio is increasingly diversified, from a primarily sugar commodity portfolio to one that balances out grain and sugar with increasing presence of sunflower and soybean. In turn, such expansion is likely to level off from this point since the Company will have soon reached the maximum level of exposure it deems prudent to accept on individual markets and competition has yet to materialise in the absence of a WHR based trade financing substitution. Due to pending changes in the quotas legislation likely to be implemented soon, this sub-division of the programme has a very limited future.

Satisfactory Satisfactory Low

3 Frameworks for markets; institutions, laws and policies that promote market function and efficiency: The facility was expected to help promote new legislation and consequently a more flexible and cost efficient medium for agricultural commodity trade financing. While the Bank’s efforts to lobby in this respect must be commended, there is no close correlation between the programme itself and such efforts which are yet to produce concrete results. The programme however pioneers the concept of warehouse licensing and should in this respect be regarded as setting a precedent that will eventually contribute to greater market efficiency.

Satisfactory Good Medium

4 Skills transfer and dispersion to the industry and economy as a whole: Transfer of skills materialises at the warehouse handling level through implementation of strict selection and monitoring encouraging best practices. It could also have resulted from the replication of this programme by other institutions that have shown a keen interest in doing so and have gone as far as poaching key personnel from the client. It has however not taken place yet.

Good Good Low

5 Demonstration effects; transfer of new behaviour and patterns: While the non replication of the programme by other institutions is somewhat of a disappointment, compensated by its on-going strong profitability, its very existence is a demonstration that secured commodity financing can be developed. The inherent high cost is a further incentive for all parties involved to design a more effective system with the introduction of appropriate legislation. The licensing of warehouses and the introduction of a systematic rating system has already proven its demonstration effect through the increase in warehouses certified by the Company.

Good Good Low

6 New standards for business conduct: The size of the Borrowers is typically too small to expect significant in roads in term of corporate governance. An increasing number of them however are improving the quality of their financial reporting. They also are able to secure larger amount of financing through this programme than they would obtain as working capital facilities from other financial institutions. This constitutes a strong incentive to improve their image through higher business standards. The introduction of corporate governance standard in the warehousing industry in Russia is at a pioneering stage. Success in this area is dependent on the introduction of appropriate legislation which has so far been stalled.

Good Good Medium

SUMMARY OF VERIFIED, POTENTIAL AND RISK RATINGS Satisfactory Good Medium

OVERALL TRANSITION IMPACT RATING:

SATISFACTORY

APPENDIX 7.2 Page 6 of 19

WASTEWATER SERVICES TRANSITION IMPACT ANALYSIS

COMMENTS

SHORT- TERM

VERIFIED IMPACT

LONGER- TERM

IMPACT POTENTIAL

RISK TO POTENTIAL TRANSITION

IMPACT

STEP I: CHANGE BY THE PROJECT AT CORPORATE LEVEL Rating Rating Rating 1 Private Ownership: N/A N/A N/A 2 Know How: Some transfer from the Western consortium has

occurred. Satisfactory Satisfactory Low

3 ards for business conduct at enterprise level:. Some new Satisfactory Satisfactory Low New standmanagement practices have been introduced the Consortium.

THE STEP II: TRANSITION IMPACT AT INDUSTRY LEVEL AND ONECONOMY AS A WHOLE

1 Competition: N/A N/A N/A 2 Market expansion via competitive interaction in the sector and

industry: N/A N/A N/A

3 r markets; institutions, laws and policies that Marginal Marginal Low Frameworks fopromote market function and efficiency:

4 ry and economy as a N/A N/A N/A Skills transfer and dispersion to the industwhole:

5 ration effects; transfer of new behaviour and patterns:

ng

N/A N/A N/A DemonstAlthough management and maintenance has been privatised, asset ownership has not, thus allow the Municipality to continue of securisupport from WB and EU-Phare.

6 uct: It is assumed that some of ther

Marginal Marginal Low New standards for business condpractices noted under Step I, No. 3 have permeated to peer facilities in omunicipalities of the country and to public authorities.

ATINGS Marginal Marginal Low

SUMMARY OF VERIFIED, POTENTIAL AND RISK R

VERALL TRANSITION IMPACT RATING: MARGINAL O

APPENDIX 7.2 Page 7 of 19

TELECOMMUNICATIONS COMPANY IN CIS TRANSITION IMPACT ANALYSIS

Comments

The Project includes the loan, the cancelled equity deal and some direct impact of the Technical Co-operation program

Short-Term Verified Impact

Longer-Term Impact

Potential

Risk to Potential Transition

Impact

STEP I: CHANGE BY THE PROJECT AT CORPORATE LEVEL

Rating Rating Rating

1 Private Ownership: The company was already partly privately owned. So far, the Project has not achieved privatisation but the investments may help to attract future private investment.

Marginal Marginal High

2 Know How: The Project contributed to improved understanding of financial reporting and to better technical quality of service.

Good Good Low

3 New standards for business conduct at enterprise level: Testimonies, ranging from complaints of overcharging and bullying of customers to full satisfaction, indicate a questionable impact. Observers disagree on whether there is improvement.

Marginal Marginal High

STEP II: TRANSITION IMPACT AT INDUSTRY LEVEL AND ON THE ECONOMY AS A WHOLE

1 Competition: While the firm has become more aware of the competitive threat from the alternative operators, its recent purchase of a competitor has led to decreased competition.

Marginal Unsatisfactory High

2 Market expansion via competitive interaction in the sector and industry: While the market has expanded, the main reason for this has been the overall economy. High tariffs and difficult interconnection slow expansion.

Satisfactory Satisfactory Medium

3 Framework for markets; institutions, laws and policies that promote market function and efficiency: Based on the parallel Technical Co-operation program, liberalisation is progressing, albeit slowly.

Good Excellent High

4 Skills transfer and dispersion to the industry and economy as a whole: Improved telecommunications generate positive secondary effect in the overall economy.

Good Good Low

5 Demonstration effects; transfer of new behaviour and patterns: The firm is reportedly the most transparent of the big national companies.

Good Good Medium

6 New standards for business conduct: Financial transparency has clearly improved. The alternative operators are critical of the firm and complain of its monopolistic behaviour.

Marginal Satisfactory Medium

SUMMARY OF VERIFIED, POTENTIAL AND RISK RATINGS

Satisfactory Good Medium

OVERALL TRANSITION IMPACT RATING:

Satisfactory

APPENDIX 7.2 Page 8 of 19

HEAVY INDUSTRY FIRM IN CEE TRANSITION IMPACT ANALYSIS

COMMENTS

SHORT- TERM VERIFIED

IMPACT

LONGER- TERM

IMPACT POTENTIAL

RISK TO POTENTIAL TRANSITION

IMPACT

STEP I: CHANGE BY THE PROJECT AT CORPORATE LEVEL Rating Rating Rating 1 Private Ownership: The privatisation was imperfect, leaving the firm

in the hands of a board split between Incumbent Management and local shareholders and financial owners. But it did transfer the firm to the private sector.

Satisfactory Satisfactory Medium

2 Know How: Piecemeal and weakly implemented investment and restructuring programs had limited impact on the firm. But the firm did stay profitable 1999-2001 from meeting the needs of tough US firms.

Satisfactory Satisfactory Medium

3 andards for business conduct at enterprise level: The Board

.

Satisfactory Satisfactory Medium New stforced better investment justifications and slowly improved external reporting, but doubts persist about the quality of accounting and MIS

STEP II: TRANSITION IMPACT AT INDUSTRY LEVEL AND ON THE ECONOMY AS A WHOLE

1 Competition: The Ban dest contribution to Satisfactory Satisfactory Medium k’s investment made a moenhancing competition in the sector by helping the firm to remain, for a few years, a competitive supplier to its customers.

2 e sector and t

.

Marginal Marginal Medium Market expansion via competitive interaction in thindustry: The OPER Team could not identify specific and significansupplier development initiatives. The firm has experienced a long standing staff drain, particularly of younger staff, to a nearby firm

3 pact

NA NA NA Frameworks for markets; institutions, laws and policies that promote market function and efficiency: The Project had no imin these areas.

4 and dispersion to the industry and economy as a Marginal Marginal Medium Skills transfer whole: The firm’s underperformance as a business and its slow modernisation mean that it has made scant impact on the sector.

5 s:

n

Marginal Marginal Medium Demonstration effects; transfer of new behaviour and patternThe project demonstrated how not to privatise a firm that required major restructuring, both in terms of shareholder structure, retentioof management, and untimely recourse to public equity markets.

6 ed Satisfactory Satisfactory Medium New standards for business conduct: The firm may have performmarginally better on corporate governance and shareholder communication than other locally listed firms.

RATINGS Marginal Marginal Medium

SUMMARY OF VERIFIED, POTENTIAL AND RISK

VERALL TRANSITION IMPACT RATING: MARGINAL O

APPENDIX 7.2 Page 9 of 19

SUBORDINATED AND SENIOR DEBT TO A MAJOR BANK TRANSITION IMPACT ANALYSIS

COMMENTS

SHORT- TERM

VERIFIED IMPACT

LONGER- TERM

IMPACT POTENTIAL

RISK TO POTENTIAL TRANSITION

IMPACT

STEP I: CHANGE BY THE PROJECT AT CORPORATE LEVEL Rating Rating Rating 1 Private Ownership: The client was created as a private company. N/A N/A N/A 2 Know How: In view of the perceived competence of the client’s

management, no technical assistance was proposed or required when the Bank extended a debenture and an SME loan. Nevertheless the client financed some consultancy works itself when needed from a leading firm and then from a major western bank. Later, the client (and another major local bank) received a TC grant from another source to provide training to accompany another loan. This training was very successful and extended for a supplementary period.

Satisfactory Satisfactory Low

3 New standards for business conduct at enterprise level: The client’s management always possessed a high level of confidence in its ability to guide the bank without outside assistance. Procedures and policies were introduced without the assistance of lenders or TC supervised by IFIs. The only training received by the client accompanied the facility mentioned above. Since acquisition by the strategic investor, the new owner of the bank has been reviewing all procedures systematically and has introduced its own policies and procedures as appropriate.

Satisfactory Satisfactory Low

STEP II: TRANSITION IMPACT AT INDUSTRY LEVEL AND ON THE ECONOMY AS A WHOLE

1 Competition: The main and constant objective of EBRD’s operations in the region has been to increase competition in the banking sector by financing a certain number of banks. The facilities under review contributed to this process.

Good Good Low

2 Market expansion via competitive interaction in the sector and industry: EBRD’s strategy has been to focus not only on traditionally structured SME lines but also on more sophisticated and long term products such as housing loans, preferred share issues, and subordinated loans. Loan syndication was carried out for the SME loan at a time where financial institutions had only limited appetite for business originating in this region.

Good Good Low

3 Frameworks for markets; institutions, laws and policies that promote market function and efficiency: The operations under review did not necessitate any new rules or regulations.

N/A N/A N/A

4 Skills transfer and dispersion to the industry and economy as a whole: There was no technical assistance component from EBRD attached to any operations under review. TC accompanied a later loan. There is some evidence of transfer of skills from the bank towards its clients which had to improve their financial management expertise to be able to apply for longer term sub-loans.

Satisfactory Satisfactory Low

APPENDIX 7.2 Page 10 of 19

5 Demonstration effects; transfer of new behaviour and patterns:

The strategy of EBRD to lend funds to several competing banks has forced these banks to stay alert and be aware of the needs of their customers. New financial instruments were proposed to clients.

Good Good Low

6 New standards for business conduct: Competition required bank management to improve skills and cost effectiveness to maintain market share and grow business.

Satisfactory Satisfactory

Low

SUMMARY OF VERIFIED, POTENTIAL AND RISK RATINGS Satisfactory Satisfactory Low

OVERALL TRANSITION IMPACT RATING: THE SUCCESS OF EBRD IN THIS COUNTRY HAS BEEN ITS CONTRIBUTION IN DEVELOPING A COMPETITIVE BANKING SECTOR THROUGH THE FINANCING OF SEVERAL SUSTAINABLE BANKS AT THE SAME TIME AND THROUGH THE INTRODUCTION OF VARIOUS FINANCIAL INSTRUMENTS. THE OPERATIONS UNDER REVIEW, COMING SEVERAL YEARS AFTER THE BANK’S INITIAL INVOLVEMENT IN THE SECTOR, ARE CONSIDERED TO HAVE MADE SOME INCREMENTAL CONTRIBUTION TO THE PROCESS.

Satisfactory

APPENDIX 7.2 Page 11 of 19

AIR CARGO TERMINAL

TRANSITION IMPACT ANALYSIS

COMMENTS

SHORT- TERM

VERIFIED IMPACT

LONGER- TERM

IMPACT POTENTIAL

RISK TO POTENTIAL TRANSITION

IMPACT

STEP I: CHANGE BY THE PROJECT AT CORPORATE LEVEL Rating Rating Rating 1 Private Ownership: [Privatisation occurred before Bank’s

involvement.] N/A N/A N/A

2 Know How: Project Company’s management is led by a representative from the strategic investor, one of the biggest cargo handlers in the world. Skills and know how transferred corresponds to international standards.

Excellent Excellent Low

3 New standards for business conduct at enterprise level: Project Company sets a higher standard for corporate governance and business conduct, for example, using generally accepted accounting principles, including IAS reporting.

Good Good Low

STEP II: TRANSITION IMPACT AT INDUSTRY LEVEL AND ON THE ECONOMY AS A WHOLE

1 Competition: [No related TI beyond the Company intended.] N/A N/A N/A 2 Market expansion via competitive interaction in the sector and

industry: A new, larger and better cargo handling service added to the development of the transport infrastructure per se and facilitates trade. However, the effect is limited because it is a replacement for an old and less efficient terminal.

Satisfactory Satisfactory Low

3 Frameworks for markets; institutions, laws and policies that promote market function and efficiency: Marginal short term impact. However, the project’s success could potentially be taken into account when new laws and policies are framed.

Marginal Good Medium

4 Skills transfer and dispersion to the industry and economy as a whole: [No related TI beyond the Company intended.]

N/A N/A N/A

5 Demonstration effects; transfer of new behaviour and patterns: Adopted international business practices in operations, financial management, financial accounting and business marketing are believed having had an influence on other airports in the region.

Satisfactory Satisfactory Low

6 New standards for business conduct: [No related TI beyond the Company intended.]

N/A N/A N/A

SUMMARY OF VERIFIED, POTENTIAL AND RISK RATINGS Satisfactory Satisfactory Low

OVERALL TRANSITION IMPACT RATING:

SATISFACTORY

APPENDIX 7.2 Page 12 of 19

WATER PROJECT TRANSITION IMPACT ANALYSIS

COMMENTS

SHORT- TERM

VERIFIED IMPACT

LONGER- TERM

IMPACT POTENTIAL

RISK TO POTENTIAL TRANSITION

IMPACT

STEP I: CHANGE BY THE PROJECT AT CORPORATE LEVEL Rating Rating Rating 1 Private Ownership: The previous state-owned company first went

through commercialisation, and then was privatized via this concession arrangement. The municipality retains ownership of fixed assets.

Excellent Good Medium

2 Know How: The international sponsors have brought in international management practice, GIS capacity, new accounting systems etc.

Good Good Low

3 New standards for business conduct at enterprise level: Introduced external audits, fixed formula for tariffs, and contract supervision.

Good Good Medium

STEP II: TRANSITION IMPACT AT INDUSTRY LEVEL AND ON THE ECONOMY AS A WHOLE

1 Competition: This is a natural monopoly. Selection was through open competition. Also competitive out sourcing.

Excellent Good Low

2 Market expansion via competitive interaction in the sector & industry: Other national water and wastewater companies remain in the public sector. Initial efforts to privatise another system stalled due to negative demonstration impacts from this project, but others are now under consideration.

Good Good Low

3 Frameworks for markets; institutions, laws and policies that promote market function and efficiency: The project is based on contract law and the government supervises the project through a contract management unit..

Good Good Medium

4 Skills transfer and dispersion to the industry and economy as a whole: If other Concessions are developed this project will have had a significant impact. Within the company, knowledge and skills transfer is occurring, but the market place for these skills is currently limited.

Satisfactory Good Medium

5 Demonstration effects; transfer of new behaviour and patterns: Initially negative but now perceived to be good in the long term. Project has positive demonstration impacts on concessions in other sectors (e.g. transportation).

Good Good Medium

6 New standards for business conduct: Initially limited to this project and related concessions.

Satisfactory Good Medium

SUMMARY OF VERIFIED, POTENTIAL AND RISK RATINGS Good Good Medium

OVERALL TRANSITION IMPACT RATING:

GOOD

APPENDIX 7.2 Page 13 of 19

PRE-PRIVATISATION BANK EQUITY TRANSITION IMPACT ANALYSIS

COMMENTS

SHORT- TERM

VERIFIED IMPACT

LONGER- TERM

IMPACT POTENTIAL

RISK TO POTENTIAL TRANSITION

IMPACT

STEP I: CHANGE BY THE PROJECT AT CORPORATE LEVEL Rating Rating Rating 1 Private Ownership: As a result of the project the client bank was

transferred from full state control to full private ownership. Excellent Excellent Low

2 Know How: There was some transfer of skills during the EBRD’s short tenure as a shareholder. The strategic investor, however, found it necessary to make substantial changes to staff.

Marginal Marginal Low

3 New standards for business conduct at enterprise level: The EBRD, through its representation on the client bank’s Board, contributed to the development of a commercial approach to strategic thinking and the proper consideration of risk management and lending policy.

Good Good Low

STEP II: TRANSITION IMPACT AT INDUSTRY LEVEL AND ON THE ECONOMY AS A WHOLE

1 Competition: The project helped position the client bank to develop more efficient operations focused on customer orientation thus improving the competitive environment in the banking sector.

Good Good Low

2 Market expansion via competitive interaction in the sector and industry: The strategic investor has launched a branch network restructuring and strengthened risk management processes with the aim of improving service to customers and enhancing the bank’s competitive position.

Good Good Low

3 Frameworks for markets; institutions, laws and policies that promote market function and efficiency: The legal agreements introduced the client bank and other parties to the transaction to standards of documentation governing complex relationships.

Good Good Low

4 Skills transfer and dispersion to the industry and economy as a whole: Building on the groundwork laid by the project, the strategic investor has undertaken a transformation process to improve operations and the quality of the branch network with an emphasis on the modernisation of infrastructure and quality of service.

Good Good Low

5 Demonstration effects; transfer of new behaviour and patterns: Post-privatisation, the client bank has been improving existing banking products and developing new ones, thus setting sound standards to be emulated by other banks.

Good Good Low

6 New standards for business conduct: The acquisition of control by the strategic investor took the client bank into a major international banking group and introduced a corporate culture emphasising customer care and shareholder value.

Good Good Low

SUMMARY OF VERIFIED, POTENTIAL AND RISK RATINGS Good Good Low

OVERALL TRANSITION IMPACT RATING: The ultimate goal of privatising the client bank by introducing a strategic investor was achieved swiftly. The Bank made a strong start on the prior goal of restructuring the client through its presence on the Supervisory Board in particular.

GOOD

APPENDIX 7.2 Page 14 of 19

HOTEL COMPANY IN CIS TRANSITION IMPACT ANALYSIS

COMMENTS

SHORT- TERM

VERIFIED IMPACT

LONGER- TERM

IMPACT POTENTIAL

RISK TO POTENTIAL TRANSITION

IMPACT

STEP I: CHANGE BY THE PROJECT AT CORPORATE LEVEL Rating Rating Rating 1 Private Ownership: In 1994 the initial Sponsor of the Project had

won a tender organised by the Government to redevelop and operate the dilapidated hotel which had stopped operations in 1991. The Company took over a lead role as Sponsor and Guarantor in 1997with the initial sponsor becoming a contractor. The Bank joined the Project at this stage.

Good Good Medium

2 Know How: The Company brought in significant expertise for the design and quality control of the hotel redevelopment. For the operation of the hotel the Company brought in experience ex pat staff for the key positions and trained successfully a large number of local staff.

Excellent Excellent Low

3 New standards for business conduct at enterprise level: The Company has set up an efficient Board of Directors and is highly regarded by the local authorities with respect to providing world class services which enabled the country to host international conferences which could not have taken place at any otherwise.

Excellent Excellent Low

STEP II: TRANSITION IMPACT AT INDUSTRY LEVEL AND ON THE ECONOMY AS A WHOLE

1 Competition: This is the only 4 star hotel in the country and has clearly raised the standards and expectations in the industry. The Project is a category by itself and in order to have competition in this luxury segment there would need to be another hotel in a similar category which is right now not easy due to the limited amount of demand for this quality of hotels in the country.

Good Good Low

2 Market expansion via competitive interaction in the sector and industry: The Company is clearly putting pressure on its competitors to achieve higher standards. However, there is a market for cheaper hotels given the present spending patterns of domestic and international tourists.

Excellent Excellent Low

3 Frameworks for markets; institutions, laws and policies that promote market function and efficiency: The Company actively participates through its General Manager in a variety of civic activities. It is recognised as one of the better employers in the country. And buys the majority of its food products from the local market.

Good Good Low

4 Skills transfer and dispersion to the industry and economy as a whole: Also due to cost reasons the expat component in management has been substantially reduced. The training schemes have allowed the training of an increasing number of local staff for the various functions. At this stage it is unlikely that skill transfer will take place through rotation to other hotels of staff due to the fact that the “Company” is the only international 4 star hotel operating in the “country”. A training component: training of the trainers of other hotels would be highly desirable to further improve the chances of skills transfers to the entire industry at an early stage.

Satisfactory Good Low

5 Demonstration effects; transfer of new behaviour and patterns: The successful operation of the hotel has a strong demonstration effect.

6 New standards for business conduct: Strong business ethics imposed

by the majority shareholder stand out in this sector. Excellent Excellent Low

APPENDIX 7.2 Page 15 of 19

SUMMARY OF VERIFIED, POTENTIAL AND RISK RATINGS Good Excellent Low

OVERALL TRANSITION IMPACT RATING:

GOOD

APPENDIX 7.2 Page 16 of 19

HEAVY INDUSTRY FIRM IN CEE TRANSITION IMPACT ANALYSIS

COMMENTS

SHORT- TERM

VERIFIED IMPACT

LONGER- TERM

IMPACT POTENTIAL

RISK TO POTENTIAL TRANSITION

IMPACT

STEP I: CHANGE BY THE PROJECT AT CORPORATE LEVEL Rating5 Rating6 Rating7

1 Private Ownership: The firm had been privatised via the voucher scheme in 1993, before our loan. Reckless directed lending by local banks to the firm’s group during our loan, and the transfer of these bad loans to the state, de-privatized the firm. The state and the trustee of bankrupt holding sold the remnants to a private group in 2003. Collecting the loan accelerated the private spin-off of one unit..

Marginal Good Medium

2 Know How: The Project gave local management its first experience of planning and financing a large capex project under market conditions. The loan covenants and monitoring led the firm to invest in better management and financial accounting systems.

Marginal Satisfactory Low

3 New standards for business conduct at enterprise level: The group’s poor governance due to corrupt and dispersed ownership limited impact. But disciplined disbursement and collection was the group’s first experience with hard lending constraints.

Marginal Satisfactory Medium

STEP II: TRANSITION IMPACT AT INDUSTRY LEVEL AND ON THE ECONOMY AS A WHOLE

1 Competition: The project increased global capacity in the Project’s main market by one third, contributing to a sharp fall in prices.

Satisfactory Good Medium

2 Market expansion via competitive interaction in the sector and industry: The main external capex suppliers failed with no impact on them. The firm mainly relied on affiliated suppliers. The Project supported export of the new products to the West; impact in the country was minimal.

Marginal Marginal Medium

3 Frameworks for markets; institutions, laws and policies that promote market function and efficiency: The voucher privatisation damaged the Project and led the state to reacquire and re-privatise it. This was a late but necessary government response to the failed privatisation by vouchers of the group..

Marginal Good Medium

4 Skills transfer and dispersion to the industry and economy as a whole: The recovery effort helped train some local lawyers and bankers in out-of-court restructuring.

Marginal Satisfactory Low

5 Demonstration effects; transfer of new behaviour and patterns: Accepting the group’s lack of transparency had negative impact.

Negative Satisfactory Low

6 New standards for business conduct: The EBRD was disciplined in disbursement and worked to restructure the loans out of court. The Bank’s focus on loan collection and a fair restructuring supported a key standard of business conduct: hard budget constraints on firms imposed by banks and accelerated the state’s work on the group .

Good Good Low

SUMMARY OF VERIFIED, POTENTIAL AND RISK RATINGS Marginal Good Medium

OVERALL TRANSITION IMPACT RATING:8

SATISFACTORY

5 Ratings Scale: Excellent, Good, Satisfactory, Marginal, Unsatisfactory, Negative, N/A. 6 Ratings Scale: Excellent, Good, Satisfactory, Marginal, Unsatisfactory, Negative, N/A. 7 Ratings Scale: Low, Medium, High, Excessive, N/A 8 Ratings Scale: Excellent, Good, Satisfactory, Marginal, Unsatisfactory, Negative. The Overall Rating is not an

arithmetical mean of individual ratings, since verified impact weighs more than potential impact.

APPENDIX 7.2 Page 17 of 19

MOBILE TELEPHONE COMPANY TRANSITION IMPACT ANALYSIS

COMMENTS

SHORT- TERM

VERIFIED IMPACT

LONGER- TERM

IMPACT POTENTIAL

RISK TO POTENTIAL TRANSITION

IMPACT

STEP I: CHANGE BY THE PROJECT AT CORPORATE LEVEL Rating Rating Rating 1 Private Ownership:

Private operation from the outset. n.a. n.a. n.a.

2 Know How: The international Sponsors transferred latest international technology, management skills.

Excellent Good Low

3 New standards for business conduct at enterprise level:.

STEP II: TRANSITION IMPACT AT INDUSTRY LEVEL AND ON THE ECONOMY AS A WHOLE

1 Competition: Two new mobile providers competed vigorously and offered consumers choices against a weak incumbent.

Excellent Good Medium

2 Market expansion via competitive interaction in the sector and industry: Growth new product offerings to differentiate from competitors. Nature and volatility of the market will also be influenced by regulatory developments, challenges for new market entrants etc.

Excellent Good Low

3 Frameworks for markets; institutions, laws and policies that promote market function and efficiency: Bank did not provide separately further TC for strengthening regulatory side. Company technical expertise contributes to dialogue with regulatory authorities.

Marginal Marginal Medium

4 Skills transfer and dispersion to the industry and economy as a whole: Introduction of new business practices e.g. :

- electronic billing and collection (bank payment SMS) - customer relation management (industry seminar, rotations

within the Company) - human resources management

Excellent Excellent Low

5 Demonstration effects; transfer of new behaviour and patterns: - western style corporate governance - creating a brand for the country

Excellent Good Medium

6 New standards for business conduct: As detailed under 4 & 5.

Excellent Good Low

SUMMARY OF VERIFIED, POTENTIAL AND RISK RATINGS Excellent Good Medium

OVERALL TRANSITION IMPACT RATING:

GOOD

APPENDIX 7.2 Page 18 of 19

MINI STEEL MILL WORKING CAPITAL FACILITY TRANSITION IMPACT ANALYSIS

COMMENTS

SHORT- TERM VERIFIED

IMPACT

LONGER- TERM

IMPACT POTENTIAL

RISK TO POTENTIAL TRANSITION

IMPACT

STEP I: CHANGE BY THE PROJECT AT CORPORATE LEVEL Rating Rating Rating 1 Private Ownership: The Company was from the outset privately

owned. Good Good Low

2 Know How: Management have installed state of the art accounting equipment and demonstrated excellent achievements in marketing, market adaptation and quality control of products, sales and MIS within the Company.

Excellent Excellent Low

3 New standards for business conduct at enterprise level: The management information systems installed and used are of the highest standards and clearly ahead of traditional plants in the country. IAS introduced in accounting and audits.

Excellent Excellent Low

STEP II: TRANSITION IMPACT AT INDUSTRY LEVEL AND ON THE ECONOMY AS A WHOLE

1 Competition: The Company established the only mini mill concept in the entire country and has had an impact on the entire market Due to their approach also able to increase production levels and product range.

Good Good Low

2 Market expansion via competitive interaction in the sector and industry: Competitive procurement and contracting introduced.

Satisfactory Good Low

3 Frameworks for markets; institutions, laws and policies that promote market function and efficiency: Tax authority’s lack of or delay in reimbursement of VAT on export products is hampering the industry. Also the export tax on scrap does not seem to deter excessive exports which restrict domestic production capabilities. These country-specific problems (systemic risks) have not yet been resolved.

Unsatisfactory Unsatisfactory High

4 Skills transfer and dispersion to the industry and economy as a whole: Latest management systems and state of the art production systems in place. With natural rotation will in due course spread to other parts of industry and economy. However, IU size compared to the sector is very small and the effect on the whole industry accordingly.

Satisfactory Good Low

5 Demonstration effects; transfer of new behaviour and patterns: Ring fencing concept for upgrades is now also adopted by a neighbouring company along with environmental and operational patterns. A stronger impact could possibly be obtained with a larger project in this sector.

Good Good Low

6 New standards for business conduct: Professional management with consolidated financial statements for the entire group.

Good Excellent Low

SUMMARY OF VERIFIED, POTENTIAL AND RISK RATINGS Good Excellent Low

OVERALL TRANSITION IMPACT RATING: MUCH WAS UNDERWAY BEFORE THE PROJECT. HOWEVER, THE BANK’S INVOLVEMENT GAVE A STAMP OF APPROVAL AND CONTRIBUTED TO A SATISFACTORY STRUCTURE.

GOOD

APPENDIX 7.2 Page 19 of 19

POWER PLANT IN CENTRAL ASIA TRANSITION IMPACT ANALYSIS

COMMENTS

SHORT- TERM VERIFIED

IMPACT

LONGER- TERM

IMPACT POTENTIAL

RISK TO POTENTIAL TRANSITION

IMPACT

STEP I: CHANGE BY THE PROJECT AT CORPORATE LEVEL Rating Rating Rating 1 Private Ownership: Marginal Marginal High 2 Know How:

The project’s TC components in accounting and tariff studies were unfortunately delayed and as a result there were negative impacts on the project.

Satisfactory Satisfactory Medium

3 New standards for business conduct at enterprise level:. Marginal Satisfactory Low

STEP II: TRANSITION IMPACT AT INDUSTRY LEVEL AND ON THE ECONOMY AS A WHOLE

1 Competition: There is no open power market. The state-owned power company still has strong monopolistic power. Dispatch does not follow marginal costs, no benchmark competition. Political decisions re purchases from the countries.

Unsatisfactory Satisfactory High

2 Market expansion via competitive interaction in the sector and industry: Sector development is driven by central planning and policy of command and control.

Unsatisfactory Marginal Medium

3 Frameworks for markets; institutions, laws and policies that promote market function and efficiency: Ill-conceived corporatisation efforts in the absence of sensible legal/regulatory framework.

Marginal Satisfactory High

4 Skills transfer and dispersion to the industry and economy as a whole: Skills transfer limited to the Company only and delays due to TC change of scope.

Unsatisfactory Satisfactory High

5 Demonstration effects; transfer of new behaviour and patterns: The Company is a single company town, limited staff rotation.

Marginal Satisfactory High

6 New standards for business conduct: Satisfactory Satisfactory Medium

SUMMARY OF VERIFIED, POTENTIAL AND RISK RATINGS Marginal Satisfactory High

OVERALL TRANSITION IMPACT RATING:

MARGINAL

Appendix 8 Page 1 of 23

OUTCOME OF PERFORMANCE RATINGS OF THE BANK'S

INVESTMENT OPERATIONS1

1. POST- EVALUATION OUTCOME 1.1 GENERAL This Appendix analyses performance ratings of evaluated investment operations. It seeks to draw conclusions and serves as a basis for some recommendations in the main text. Projects for evaluation are selected from all fully disbursed projects in the portfolio considered ready for evaluation.2 Performance evaluations of individual projects are generally only conducted once in their lifetime, normally with no subsequent re-validation. 1.2 EVALUATION COVERAGE IN 2003 The Bank's evaluation policy sets a minimum annual coverage ratio of 60 per cent (increased from 50 per cent in 2004) to ensure a good representation of projects, accountability and timely identification of relevant lessons. Chart 1.1 presents the actual and projected coverage ratio.

Chart 1.1: Evaluation coverage for investment operations (actual to 2004 and projected)

0%10%20%30%40%50%60%70%80%90%

100%

1993 1995 1997 1999 2001 2003 2005 2007 2009% o

f inv

estm

ent o

pera

tions

rea

dy fo

r ev

alua

tion

Coverage gap - XMR Reviewed Operations% of investment operations covered by XMR Assessments% of investment operations covered by OPER reports

2004 onwards to 2003

In 2003 the Operation Performance Evaluation Reviews (OPERs) covered 26 per cent of all new projects ready for evaluation. An additional 43 per cent was covered with independent assessment reports by PED on bankers’ expanded monitoring reports (XMR assessments).3 This brought the year's coverage to 48 projects4 or 69 per cent of ready operations, which is well above the increased minimum coverage ratio.5

1 Appendix 5 presents an overview of the investment operation and evaluation databases. 2 Investment projects are considered ready for evaluation one year after the last disbursement of loans and two years thereafter in cases of

equity. At least one year of commercial operations, with at least one year of audited accounts, should normally have passed for all investment projects. From 2004 onward the one year mentioned above in respect of loan will be extended to 18 months.

3 An XMR assessment takes about four days work of PED staff. It does not involve a field mission and is based on a desk-study. It includes: a) study of the XMR (a joint monitoring and self-evaluation report by bankers); b) review of project documents and various industry reports; c) interviews with operation teams, other EBRD staff and sometimes external parties; and d) independent validation of performance ratings and lessons. The performance ratings assigned to projects that are XMR assessed are aggregated in the overall performance rating of all evaluated projects as presented in this report. Lessons from XMR Assessments are included in PED's Lessons Learned Database (LLD).

4 In total, 49 reports were produced. However, one of these was not an evaluation of a ready project but of the relationship between EBRD and one of its major clients. Therefore it does not count towards the coverage ration of projects ready for evaluation.

5 The evaluation coverage gap is compensated, in part, by PED's review of all XMRs. In contrast to OPERs and XMR assessments, XMR reviews do not seek to validate self-evaluation ratings and no editing is made of the lessons. In contrast, the reviews seek to ascertain

Appendix 8 Page 2 of 23

1.3 SIZE AND REPRESENTATION OF THE SAMPLE OF EVALUATED PROJECTS The total sample comprises OPER and XMR assessments by PED on 385 projects evaluated in 1993-2003.9 The projects were selected to represent a cross-section of all EBRD operations, while also looking at the prospect of generating useful lessons. The annual evaluation coverage was 100 per cent at the end of 1996 and well over the 60 per cent target thereafter, as shown in Chart 1.1. A total of 508 investment projects had reached the ready-for-evaluation stage by the end of 2003. The evaluated sample of 385 projects is fairly large (76 per cent) and represents a reasonably balanced cross-section of all EBRD projects. The evaluated share of all 1017 signed projects was lower at 42 per cent, as many of the more recently approved operations had not yet reached the evaluation stage.10 Section 10 of this Appendix presents an analysis of the country, sector representation in the sample as well as and the risk rating distribution of the evaluated sample. The sample of projects selected from the groups of operations ready for evaluation continues to be a good representation of the Bank’s portfolio as a whole.

2. PERFORMANCE RATING OF EVALUATED PROJECTS 2.1 THE COMPOSITE OVERALL PERFORMANCE RATING OF A PROJECT The overall performance rating of an evaluated operation builds on several underlying performance ratings, derived from the Bank's mandate. Transition impact is the overriding individual rating for all operations. Environmental performance and change are significant indicators for projects with high environmental risks. The following broad performance dimensions are addressed: a. Transition impact

- transition impact is defined as the effects of a Bank project on businesses, markets or institutions that contribute to the transformation from central planning to a well functioning market economy

b. The environment

- environmental performance measures how well the environmental objectives of the project (institutional, emissions control, regulatory compliance, social issues and public participation) were identified and have been met

- environmental change is measure as the difference between the environmental performance before the project started and its performance at the time of evaluation

completeness and clarity in consultation with the teams and report the quality ratings given with PED's sign-off. The independent OPER reports, XMR assessments and quality-control by XMR reviews, together cover 100 per cent of all operations ready for evaluation.

9 See annotations of the respective tables and charts. Those concerning the broader sector and country distribution analysis cover 1993-2003, while others, including those on transition impact, cover 2000-03 when the current ratings system for transition impact was applied.

10 See Appendix 5 for more detailed data; records on signed operations may also at times divide one operation into separate commitments.

Appendix 8 Page 3 of 23

c. Additionality

- the Bank's additionality in terms of whether the Bank provides financing that could not be mobilised on the same terms by markets and/or whether the Bank can influence the design and functioning of a project to secure transition impact

d. Sound banking principles

- project and company (financial) performance provide the sustainability element to allow transition impact to enfold beyond the project/company, and

- fulfilment of project objectives concerns the extent of verified and expected risk weighted fulfilment potential of the operation’s “process” and “project” objectives (“efficacy”) upon validation of their relevance

e. The Bank's investment performance

- the Bank’s investment performance measures the extent to which the gross contribution of a project is expected to be sufficient to cover its full average transaction cost and contribute during its life to the Bank’s net profit. Unlike the other dimensions, however, it does not represent any impact of the project “on the ground” in the country.

f. Bank handling

- Bank handling assesses the due diligence, structuring and monitoring of the project, as undertaken by all departments and units involved in the operation process, and the Bank as a whole. A judgement is made on the quality of the work and on how effectively the Bank carried out its work during the life of the project. Positive and negative lessons are generated. In case operations are evaluated that are handled by the Corporate Recovery Unit, Bank Handling will also take into account problem recognition, remedial action and recovery efforts.

In the past, multilateral development banks (MDBs) have had different ways of measuring overall performance and performance with respect to their mandates. Recently, however, the MDBs have been asked, by their shareholders, to harmonise their evaluation procedures and processes, to ensure their results are more comparable with the outcomes of other MDBs. Therefore, the evaluation departments of the MDBs, through the Evaluation Co-operation Group (ECG), have attempted to harmonise their rating systems so that some comparisons can be made. For the EBRD, this means that the Bank will measure transition outcome ratings by grouping all the ratings that measure “results on the ground” in the respective countries. The composite rating categories for the transition outcome rating are: transition impact; environmental performance and change; project and company financial performance; and fulfilment of project objectives. The overall performance and the transition outcome ratings, when compared, are highly similar. 2.2 OVERALL PERFORMANCE RATINGS 1996-2003 Table 2.1 and Chart 2.1 present the assigned overall performance ratings given to evaluated EBRD investment projects. The outcome for all projects assessed since 1996, when PED introduced refined methods of evaluation, is shown. By the end of 2003 evaluated operations represented 76 per cent of all EBRD investment projects which have completed the investment phase and are now ready for evaluation. Of these projects, there is a good level of representation in terms of country, sector and risk classification compared to all disbursed operations as shown in Section 10 of this Appendix.

Appendix 8 Page 4 of 23

Table 2.1 and Chart 2.1: Overall performance ratings and percentage distribution of assigned ratings

Chart 2.1

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1996 1998 2000 2002

Year of evaluation

% o

f eva

luat

ed p

roje

cts

Unsuccess-ful

PartlySuccessful

Successful

HighlySuccessful

Table 2.1

Year of evaluation

Un-success-

ful

Partly Success-

ful

Sub-total

Success-ful

Highly Success-

ful

Sub-total

No. of eval-

uations

1996 15% 29% 44% 44% 12% 56% 34

1997 11% 42% 53% 36% 11% 47% 36

1998 22% 21% 43% 53% 4% 57% 49

1999 24% 22% 46% 46% 8% 54% 50

2000 19% 24% 43% 40% 17% 57% 42

2001 22% 31% 53% 47% 0% 47% 49

2002 14% 38% 52% 42% 6% 48% 50

2003 9% 31% 40% 54% 6% 60% 49

1996-97 13% 36% 49% 40% 11% 51% 70

1996-98 17% 29% 46% 46% 8% 54% 119 1996-99 19% 27% 46% 46% 8% 54% 169

1996-2000 19% 27% 46% 44% 10% 54% 211

1996-2001 20% 27% 47% 45% 8% 53 % 260

1996-2002 19% 29% 48% 44% 8% 52% 310

1996-2003 17% 29% 47% 46% 8% 54% 358

Just over half (191 of 358 evaluated projects, or 53 per cent) of the operations evaluated obtained ratings in the Successful - Highly Successful bracket.11 This share has remained relatively stable over the years. Seven per cent of the projects scored Highly Successful overall, while more than twice as many were rated Unsuccessful (17 per cent). Projects with the highest overall rating scored well on transition impact and the other performance indicators. Generally, the lowest-scoring projects had low or poor project and company financial performance (see Section 6 of this Appendix). This resulted in low sustainability and lost positive external factors in the sector and economy as a whole. Only sustainable projects can yield notable transition impact, via linkages or positive demonstration of successful market-driven reform of enterprises. The overall performance outcome may seem modest, with almost half of the projects rated only Partly Successful or Unsuccessful. However, it must be remembered that the region remains a risky investment environment, and this continues to be true even for the more recent EBRD projects developed in advanced transition economies, as EBRD is supporting further deepening of the transition process in more challenging areas where unemployment is high and financial intermediation limited. Moreover, the need to secure the Bank’s additionality in projects means that the EBRD must accept at times relatively high-risk projects as sponsors for lower-risk projects in these countries can readily attract financing from the market. This constraint continues to put high demands on the Bank to select and structure projects that mitigate both the industry and sector-specific risks in the particular investment environment and the risks to transition impact.

11 Weighting by volume of investment yields better results, with 64 per cent Successful or higher, 23 per cent Partly Successful, while the

Unsuccessful ratings share is 13 per cent.

Appendix 8 Page 5 of 23

2.3 PERFORMANCE RATINGS BY COUNTRY GROUPS 2.3.1 Regional comparisons. Chart 2.2 below shows that approximately 57 per cent of evaluated projects in the south-eastern Europe (SEE) group of countries now fall in the Successful or Highly Successful bracket. This is about the same as for the “CEB” group, comprising central and eastern Europe and the Baltic states. However, the SEE group has a lower share of Unsuccessful projects, compared to CEB. This difference coincides with a higher share of projects in the SEE group with strong strategic sponsors, or for which the reform challenges were rather less complex. By contrast, the Bank was not always additional for such lower-risk projects in CEB, where the proportion of financed projects with strategic external sponsors fell in recent years, including in the generally riskier general industry sector. The pace of reform and achieved economic transition in the investment environment seems to significantly influence the project success scorings: only about 38 per cent of evaluated projects in ETCs obtained the two highest performance ratings, and the rate is 42 per cent for Russia, compared with 59 per cent in the CEB group and 57 per cent in SEE.

Chart 2.2: Overall performance ratings by country groups (369 evaluated investment operations 1993-2003)

10% 8% 9% 4%

49% 49%33%

38%

52%

24% 34%

31% 36%

36%

17%8%

27% 26%

8%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

CE Europe& Baltics

(177)

SE Europe(61)

Russia (64) ETCs (42) Other CIS(25)

Region

% o

f eva

luat

ed p

roje

cts

Unsuccess-ful

PartlySuccessful

Successful

HighlySuccessful

Note: 16 regional projects omitted

ETCs: Armenia, Azerbaijan, Georgia, Kyrgyz Republic, Moldova, Tajikistan, Uzbekistan Other CIS: Belarus, Kazakhstan, Turkmenistan, Ukraine

The average success ratings in the various country groups suggest that investment projects in countries where the characteristics of the planned economy prevailed longest may face the highest financial and transition risks. Projects in Russia and the rest of CIS, and in the Caucasus and Central Asia, still face more difficult challenges than did projects in the early 1990s in central Eastern Europe and the Baltic states, and more recently those in the more reform-oriented countries in South-Eastern Europe. It should be noted, however, that Ukraine and Kazakhstan are emerging in the ‘other CIS’ group as better performers, which raises the overall performance of that group above Russia and ETCs, and brings it closer to the reference groups of CEB and SEE (Chart 2.2).

The pattern in Chart 2.3 shows the change in Overall performance ratings over time for the main geographical areas.

Appendix 8 Page 6 of 23

Chart 2.3: Overall performance ratings: Changes by year in the major country groups

2.3.2 Central and Eastern Europe. It is noticeable that the overall rating Central and Eastern Europe and the Baltic states has been falling on average from the late 1990s. The trend can be associated with the decrease of additionality in the projects in the region (see below section 4.1).

Chart 2.4: Sectoral distribution of evaluated projects in Central Europe

Central Europe

21%34%

46%9%

16%11%

63%43%

39%

7% 8% 4%

0%10%20%30%40%50%60%70%80%90%

100%

1993-96 (43) 1997-2000 (80) 2001-2003 (54)

Year of evaluation

% o

f eva

luat

ed p

roje

cts Telecoms

Industry &Commerce

Infrastructure

FinancialInstitutions

(a) Central Europe

9%

51%54%

41%

21%18%

35%

19% 14% 20%

4%15%0%

10%20%30%40%50%60%70%80%90%

100%

1993-96 (43) 1997-2000 (80) 2001-2003 (54)Year of evaluation

Unsuccess -ful

PartlySuccess ful

Success ful

HighlySuccess ful

(b) SE Europe

25%4%

25%50% 57%

38%37% 30%

13% 7% 9%

7%0%10%20%30%40%50%60%70%80%90%

100%

1993-96 (8) 1997-2000 (30) 2001-2003 (23)Year of evaluation

Unsuccess -ful

PartlySuccess ful

Success ful

HighlySuccess ful

(c) Russia

14% 13%

14% 30%42%

57%27%

29%

14%36%

17%

6%0%10%20%30%40%50%60%70%80%90%

100%

1993-96 (7) 1997-2000 (33) 2001-2003 (24)

Year of evaluation

Unsuccess -ful

PartlySuccess ful

Success ful

HighlySuccess ful

(d) All CIS excl. Russia

28%54%

38%

35%31%

11%

3%0%10%20%30%40%50%60%70%80%90%

100%

1993-1996 (1) 1997-2000 (29) 2001-2003 (37)

Year of evaluation

Unsuccess -ful

PartlySuccess ful

Success ful

HighlySuccess ful

(e) ETCs

21%

52%

35%

13%

37%

42%

0%10%20%30%40%50%60%70%80%90%

100%

1993-1996 (0) 1997-2000 (19) 2001-2003 (23)

Year of evaluation

Unsuccess -ful

PartlySuccess ful

Success ful

HighlySuccess ful

(f) Other CIS (excl. ETCs & Russia)

57%

40%36%

7%

10%

40%

10%

0%10%20%30%40%50%60%70%80%90%

100%

1993-1996 (1) 1997-2000 (10) 2001-2003 (14)

Year of evaluation

Unsuccess -ful

PartlySuccess ful

Success ful

HighlySuccess ful

Appendix 8 Page 7 of 23

Chart 2.5 Overall Ratings of Financial Institutions Projects in Central Europe

Financial Institution

11%22%

8%

67% 52%

40%

11% 15%

36%

11% 11% 16%

0%10%20%30%40%50%60%70%80%90%

100%

1993-96 (9) 1997-2000 (27) 2001-2003 (25)

Year of evaluation

% o

f eva

luat

ed p

roje

cts

Unsuccess-ful

PartlySuccessful

Successful

HighlySuccessful

2.3.3 South Eastern Europe. Performance in South Eastern Europe appears to be rising steadily, with a much more striking improvement in Russia and the remainder of the CIS. In these last two groups, the number of projects evaluated in 1993-96 is too small to give any meaningful results, but the outcomes from 1997-2000 and 2000-2003 show the marked improvement over time. 2.3.4 The early transition countries (ETCs). As part of the EBRD’s 2003 medium term strategy update, the Bank highlighted the strategic priority to focus on the poorest member countries of operations. They are part of the CIS Group excluding Russia and comprise the ETC countries: Armenia, Azerbaijan, Georgia, Kyrgyz Republic, Moldova, Tajikistan and Uzbekistan. The seven countries have suffered from political instability and face difficult transition challenges. Domestic markets are small and access to world export markets often limited to a few products. The financial system and the local infrastructure are underdeveloped. There are widespread deficiencies in Governance. This very poor investment climate makes for low level of foreign investment and creates major challenges for EBRD investments. Overall performance of the projects in this region has been quite low. The earlier chart 2.2 shows that in the ETCs group only 38% of the projects are rated successful or highly successful, which is among the lowest in the Region. Breaking the projects down by year, however, shows that this is largely due to the poor overall performance of projects in the region evaluated in 1997-2000 (Chart 2.6). In the last three years, despite the absence of any Highly Successful projects, 52 per cent of projects have been rated Successful, the same as the percentage of projects rated Successful or Highly Successful in the whole population.

Appendix 8 Page 8 of 23

Chart 2.6: Overall ratings in the ETCs from 1993 to 2003

21%

52%

35%

13%

37%

42%

0%10%20%30%40%50%60%70%80%90%

100%

1993-1996 (0) 1997-2000 (19) 2001-2003 (23)

Year of evaluation

% o

f eva

luat

ed p

roje

cts

Unsuccess-ful

PartlySuccessful

Successful

HighlySuccessful

The poor overall performance in 1997-2000 was not due to the low transition impact of projects in the ETCs countries (see section 3.3 and Chart 3.2). It can be seen from Chart 2.7 and Chart 2.8 that the failure in 1997-2000 was in:

(a) Fulfilment of Objectives, in which category only 32 per cent was rated Good or Excellent, which compares with 56 per cent of all evaluated projects during the period. (b) Project Financial Performance in which category only 32 per cent rated Good or Excellent, which compares with 68 per cent in the whole population.

Chart 2.7 Fulfilment of Objectives

ETCs

17%21%

17%

47%43%

32% 22%

0%10%20%30%40%50%60%70%80%90%

100%

1993-1996 (0) 1997-2000 (19) 2001-2003 (23)

% o

f eva

luat

ed p

roje

cts

Year of evaluation

NA

Unsatis-factoryMarginal

Good

Excellent

Chart 2.8 Project Financial Performance

ETCs

17%21%

52%47%

22%21%

9%

11%0%

10%20%30%40%50%60%70%80%90%

100%

1993-1996 (0) 1997-2000 (19) 2001-2003 (23)

Year of evaluation

% o

f eva

luat

ed p

roje

cts NA

Unsatis-factoryMarginal

Good

Excellent

Appendix 8 Page 9 of 23

The improvement in the overall ratings in 2001-2003 (Chart 2.3) appears to be related to the same rating components:

(a) there was drastic improvement in the fulfilment of objectives in the ETCs (Chart 2.7); with 69 per cent rated Good or Excellent, this is now slightly higher than in the overall population; and (b) Project financial performance (chart 2.8) has slightly improved but is still low at 34 per cent Good or Excellent (compared to 54 per cent in the entire population). As is further explained in section 3.3 Transition Impact in the ETCs in the |Satisfactory to Excellent range has fallen slightly from 70 to 65 per cent (Chart 3.2).

The decomposition of the improvements in the overall ratings in 2001-2003 in ETCs by sector leads to an analysis of samples that are still too small to draw definite conclusions. The preliminary results indicate that the largest improvements were in industry, followed by infrastructure. Results in the financial sector are not so clear, since the number of Successful and Unsuccessful project have been growing relatively fast, at the expense of the Partly Successful category. 2.4 OVERALL PERFORMANCE RATINGS BY SECTORS The large difference in performance between different industry sectors, seen in previous AEORs, appears to be decreasing. Infrastructure operations are still assigned better overall success ratings, 56 per cent of the evaluated projects rating Successful or Highly Successful, in contrast with projects in Financial Institutions (54 per cent) and general industry (52 per cent, or 48 per cent if the 29 Telecoms projects are excluded).13 However, these differences are not as marked as in previous years.

Chart 2.9: Overall performance ratings by sector groups (385 evaluated investment operations, 1993-2003)

10% 8%

44% 53% 40%

66%

28% 28%31%

24%18% 15% 21%

3% 10%0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

FinancialInstitutions (130)

Infrastructure (60) Industry excl.Telecoms (166)

Telecoms (29)

Sector

% o

f eva

luat

ed p

roje

cts

Unsuccess-ful

PartlySuccessful

Successful

HighlySuccessful

Infrastructure = municipal/environment, power, energy efficiency and transport;

Industry = agribusiness, general industry, natural resources, commercial services, property/tourism, and telecommunications

Chart 2.10 below shows that performance in Infrastructure and Financial Institutions has tended to fall over the years, while Industry and Commerce has been maintaining and slightly increasing its share of projects rated Successful and Highly Successful.

13 General industry comprises a broad range of sub-sectors in industry, manufacturing and commercial services. The group has a high

dominance of private sector operations.

Appendix 8 Page 10 of 23

Chart 2.10: Overall performance ratings of different industry sectors, 1993-2003

Industry and Commerce excl. Telecom

12%

32% 42% 44%

38% 27% 31%

18% 23% 20%

5%8%0%10%20%30%40%50%60%70%80%90%

100%

1993-96 (34) 1997-2000 (77) 2001-2003 (55)Year of evaluation

Telecoms

10%

75%60% 70%

25% 27% 20%

13%0%

10%20%30%40%50%60%70%80%90%

100%

1993-96 (4) 1997-2000 (15) 2001-2003 (10)Year of evaluation

Financial Institutions

18%

47%41%

46%

18%25% 35%

18% 20% 16%

4%14%0%

10%20%30%40%50%60%70%80%90%

100%

1993-96 (17) 1997-2000 (56) 2001-2003 (57)

Year of evaluation

Unsuccess-ful

PartlySuccess ful

Success ful

HighlySuccess ful

Infrastructure

17%

67%

52% 52%

24% 40%

17% 21%8%

3%0%10%20%30%40%50%60%70%80%90%

100%

1993-96 (6) 1997-2000 (29) 2001-2003 (25)

Year of evaluation

Unsuccess-ful

PartlySuccess ful

Success ful

HighlySuccess ful

s

Unsuccess-ful

PartlySuccess ful

Success ful

HighlySuccess ful

Unsuccess-ful

PartlySuccess ful

Success ful

HighlySuccess ful

3. THE TRANSITION IMPACT (TI) RATING 3.1 METHODOLOGY The case presentations in Appendix 7 illustrate the evaluation methodology used after project signing (ex-post). This uses the same framework and indicators as the ex-ante (before project signing) methodology, applied by the Bank during the approval stage of new projects. It should be noted that this methodology includes short-term verified impact, the assessed potential for further transition impact, as well as an assigned risk for the realisation of this potential. From 2000 a six-grade scale was applied for all post-evaluated operations, similar to the scale adopted for ex-ante assessment of TI-potential and attendant risks. 3.2 THE DISTRIBUTION OF TI RATINGS TI ratings carry a high weight in the overall performance ratings of projects. Table 3.1 shows the distribution of the ex-post TI ratings by PED over the past four years. No trends can yet be derived. The share of projects with Good to Excellent ratings is half (50 per cent in the four years). Exactly 75 per cent of the projects rated Satisfactory or higher, but it is notable that 11 per cent of the projects during the same period rated Unsatisfactory or Negative.14

14 Weighting by volume of investment yields better results, with 61 per cent Good or Excellent and 86 per cent rated Satisfactory or

higher.

Appendix 8 Page 11 of 23

Table 3.1: TI ratings of 189 investment operations evaluated 2000 - 03

Period Negative Unsatis-factory

Marginal Subtotal Negative - Marginal

Satisfactory Good Excellent Subtotal Satisfactory -

Excellent

Total projects

evaluated

2000 5% 10% 10% 25% 21% 40% 14% 75% 42

2001 4% 14% 14% 32% 25% 43% 0% 68% 49

2002 2% 6% 16% 24% 24% 46% 6% 76% 50

2003 0% 4% 15% 19% 29% 48% 4% 81% 48

2000-2001 4% 12% 12% 28% 23% 42% 7% 72% 91

2000-2002 4% 10% 13% 27% 24% 43% 6% 73% 141

2000-2003 3% 8% 14% 25% 25% 44% 6% 75% 189 3.3 TRANSITION IMPACT RATINGS BY COUNTRY GROUPS Chart 3.1 presents the TI rating distribution by country groups of 177 projects15 evaluated in 2000-03, applying the six grade TI scale. The ratings follow a pattern similar to the findings for the higher level of overall performance, with SE Europe performing relatively better than other regions, including those more advanced in the transition.

Chart 3.1: TI ratings of 177 post-evaluated investment operations in 2000-03 by country groups

4% 9% 12%

37%

56%42%

48% 39%

34%

22%

15%15% 22%

9%

9%

15% 22%28%

10% 9% 15%

6%

6%3%4% 6%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

CE Europe& Baltics

(67)

SE Europe(32)

Russia (33) ETCs (27) Other CIS(18)

Region

% o

f eva

luat

ed p

roje

cts Negative

Unsatis-factoryMarginal

Satis-factory

Good

Excellent

Note: 16 regional projects omitted

ETCs: Armenia, Azerbaijan, Georgia, Kyrgyz Republic, Moldova, Tajikistan, Uzbekistan Other CIS: Belarus, Kazakhstan, Turkmenistan, Ukraine

Further time series analysis indicates that 87 per cent of projects in south-eastern Europe had a Satisfactory to Excellent TI rating in 2000-03, which is close to the 80 per cent performance in central eastern Europe and the Baltic states during 1996-99 (see Chart 3.2 below). This provides further evidence that south-eastern Europe is entering into a transition phase similar to central eastern Europe and the Baltic states during the second half of the 1990s. The high transition impact ratings in SE Europe are the result of well-focused projects supported by an accommodating business environment. At the same time, however, Marginal and Negative transition impact ratings have increased on average in 2000-03 over the previous period in central eastern Europe and the Baltic states where transition is well advanced and the impact from new EBRD investments tends to decrease at the margin. 15 Twelve regional projects were omitted from the total 189 evaluated projects for transition impact

Appendix 8 Page 12 of 23

Chart 3.2: Development of TI ratings over time for projects evaluated 1993-2003:

presented by region

(a

77%

23%

0%10%20%30%40%50%60%70%80%90%

100%

1993-96(43) (80) (54)

Year of evaluation

) Central Europe

80% 74%

20% 26%

1997-2000 2001-2003

Marginal-Negative

Satis factory-Excellent

(b) SE Europe

75% 80% 87%

25% 20% 13%

0%10%20%30%40%50%60%70%80%90%

100%

1993-96 1997-2000 2001(8) (30)

-2003(23)

Year of evaluation

Marginal-Negative

Satis factory-Excellent

(c) Russia

71%52%

71%

29%48%

29%

0%10%20%30%40%50%60%70%80%90%

100%

1993-96(7)

1997-2000(33)

2001-2003(24)

Year of evaluation

Marginal-Negative

Satis factory-Excellent

(d) All CIS excl. Russia

69% 65%

31% 35%

0%10%20%30%40%50%60%70%80%90%

100%

1993-1996(1)

1997-2000(29)

2001-2003(37)

Year of evaluation

Marginal-Negative

Satis factory-Excellent

(e) ETCs

68% 65%

35%32%

0%10%20%30%40%50%60%70%80%90%

100%

1993-1996(0)

1997-2000(19)

2001-2003(23)

Year of evaluation

Marginal-Negative

Satis factory-Excellent

(f) Other CIS (excl. ETCs & Russia)

100%

70% 64%

36%30%

0%10%20%30%40%50%60%70%80%90%

100%

1993-1996(1)

1997-2000(10)

2001-2003(14)

Year of evaluation

Marginal-Negative

Satis factory-Excellent

Meanwhile, the TI performance of Russia, continues to improve, as the business environment has been more favourable to private sector development, at least in the last few years. By contrast, Transition Impact in ETCs 68 per cent rated Satisfactory to Excellent, which compares with 73 per cent in the whole population) performed worse than in other regions, but not by such a large margin as overall achievements (see previous chart 3.2). This suggests that the projects introduced transition reforms, but their sustainability and hence overall performance was affected by the economic and political climate.

Appendix 8 Page 13 of 23

3.4 TI RATINGS BY SECTORS Though the 2000-03 sample is limited, the infrastructure sector had the highest score for Good and Excellent-rated projects, at 51 per cent. The sector also, however, had a large number of Marginal-rated projects (see Chart 3.3). A total of 47 per cent of projects in the industry and commerce sector, excluding telecommunications projects, were rated Good or Excellent, while 49 per cent of financial institution projects were rated Good or Excellent. These results are largely consistent with evaluation findings from prior years, up to 1999, applying the five-grade TI-rating scale, as shown in previous AEORs. The sector pattern appears, therefore, fairly stable over time.

Chart 3.3: TI ratings 2000-03 by sector groups (189 investment operations)

8%

46% 48% 39%

58%

28% 16% 26%

25%11% 26% 14%

10% 11%

3%3%17%

3%3%3%3%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

% o

f eva

luat

ed p

roje

cts

0%Financial

Institutions (72)Infrastructure

(31)Industry excl.Telecoms (74)

Telecoms (12)

Sector

Negative

Unsatis-factoryMarginal

Satis-factory

Good

Excellent

Infrastructure = municipal/environment, power, energy efficiency and transport excluding shipping;

Industry and commerce = agribusiness, general industry, natural resources, commercial services, property/tourism, and telecommunications

The sector group TI ratings for 2000-03, shown in Chart 3.3, are similar to the overall sector performance results for the same period. PED had expected a higher transition impact in industry and finance. The private sector projects were also expected to have benefited more from freer market incentives and, therefore, generated higher transition impact. However, this does not always occur as planned. While the Bank is moving further east, enhancing foreign direct investment policies16 in ETCs will be beneficial to increasing transition impact in industry projects.

16 Priority would be given to repatriation of profits, streamlining licensing procedures, import liberalisation in material and equipment

and removal of foreign exchange controls. 18 Only nine evaluated operations have received the lowest additionality rating (between 1993-2003). The rating was assigned due to

the projects’ low financial additionality, combined with a design and structure that was non-conducive to improved transition impact or sound banking.

Appendix 8 Page 14 of 23

4. ADDITIONALITY 4.1 EVALUATED ADDITIONALITY OF 385 OPERATIONS CONFIRMS ADHERENCE TO THE BANK'S MANDATE Of 385 operations evaluated in 1993-2003, 62 per cent had additionality Verified in All Respects, while 28 per cent had additionality Verified at Large. This left 10 per cent of projects with additionality Verified in Part or Not Verified. Very few projects were rated in the latter group.18 This outcome supports the EBRD’s additionality requirement under the Bank’s mandate and confirms that there was no crowding out of market financing.

Chart 4.1 and Table 4.1: Ratings of the Bank’s additionality

Chart 4.1 Table 4.1

Ratings 1993-96

1993-97

1993-98

1993-99

1993-2000

1993-2001

1993-2002

1993-2003

Not Verified 0% 0% 1% 2% 2% 2% 3% 2%

Verified in Part

2% 4% 4% 6% 5% 6% 8% 8%

Subtotal 2% 4% 5% 8% 7% 8% 11% 10%

Verified at Large

34% 28% 22% 25% 24% 26% 27% 28%

Verified in All Respects

64% 68% 73% 67% 69% 66% 62% 62%

Subtotal 98% 96% 95% 92% 93% 92% 89% 90%

Total (No. of projects)

61 97 146 196 238 287 337 385

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1993 1995 1997 1999 2001 2003

Year of evaluation

% o

f eva

luat

ed p

roje

cts Not

Verified

Verified inPart

Verified atLarge

Verified inAllRespects

4.2 ADDITIONALITY RATINGS BY REGION AND OVER TIME Charts 4.2 to 4.5 below show additionality ratings by region and over time. The overall decrease in ratings for additionality is apparent in every region of the Bank's operations. It can also be seen that Central Europe and Russia are the two regions where the ratings on additionality are the lowest.

Appendix 8 Page 15 of 23

Chart 4.2-4.5: Additionality ratings: changes in year in major country groups

(a) Central Europe

60%39%

37% 26%

39%

8%19%

64%

2% 4%3%

0%10%20%30%40%50%60%70%80%90%

100%

1993-96 (43) 1997-2000 (80) 2001-2003 (54)Year of evaluation

Not Verified

Verified inPart

Verified atLarge

Verified inAllRespects

(b) SE Europe

88%

57%

13%17%

30%

13%

80%

3%

0%10%20%30%40%50%60%70%80%90%

100%

1993-96 (8) 1997-2000 (30) 2001-2003 (23)Year of evaluation

Not Verified

Verified inPart

Verified atLarge

Verified inAllRespects

(c) Russia

71%

33%

29%

21%

38%

12% 25%

58%

9% 4%

0%10%20%30%40%50%60%70%80%90%

100%

1993-96 (7) 1997-2000 (33) 2001-2003 (24)Year of evaluation

Not Verified

Verified inPart

Verified atLarge

Verified inAllRespects

(d) All CIS excl. Russia

70%

7%

24%

100% 93%

0%10%20%30%40%50%60%70%80%90%

100%

1993-1996 (1) 1997-2000 (29) 2001-2003 (37)Year of evaluation

Not Verified

Verified inPart

Verified atLarge

Verified inAllRespects

5. COMPANY AND PROJECT FINANCIAL PERFORMANCE 5.1 COMPANY AND PROJECT FINANCIAL PERFORMANCE RATINGS 1993-2003 The company and project financial performance ratings reflect whether the Bank financed “sound and economically viable operations”. Sustainable and financially viable projects are a key condition for sustained transition impact. The company performance ratings are based on the profitability, debt-service performance, financial status and prospects of borrowers and investee entities. Project performance is also assessed using indicators, such as financial internal rates of return (FIRR), that reflect a company’s success and financial strength.20 The financial performance ratings of 385 evaluated operations by end-2003 are presented in Charts 5.1 and 5.2:

20 The key financial and economic performance indicators are addressed in each of the respective OPER reports; the macro-economic

viability is captured in some types of projects in the economic internal rate of return, EIRR.

Appendix 8 Page 16 of 23

Charts 5.1 and 5.2: Company and project financial performance (385 investment operations)

Chart 5.1: Company performance Chart 5.2: Project performance

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1993 1995 1997 1999

Year of evaluat%

of e

valu

ated

pro

ject

s

2001 2003

ion

NA

Unsatis-factoryMarginal

Good

Excellent

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1993 1995 1997 1999 2001 2003

Year of evaluation

% o

f eva

luat

ed p

roje

cts

NA

Unsatis-factoryMarginal

Good

Excellent

1993-97

1993-98

1993-99

1993-

2000

1993-

2001

1993-

2002

1993-

2002

1993-97

1993-98

1993-99

1993-

2000

1993-

2001

1993-

2002

1993-

2003 Unsatis-factory

19% 21% 22% 21% 21% 19% 18% Unsatis-factory

19% 20% 21% 21% 21% 20% 18%

Marginal 24% 23% 22% 23% 24% 24% 23% Marginal 23% 22% 22% 22% 24% 24% 24% Good 32% 36% 37% 38% 37% 38% 38% Good 30% 35% 38% 38% 36% 36% 38% Excellent 27% 20% 19% 18% 17% 18% 19% Excellent 28% 23% 19% 19% 18% 19% 18% NA 0% 0% 0% 0% 1% 1% 2% NA 0% 0% 0% 0% 0% 1% 2% Total (No. of projects)

97 146 196 238 287 337 385 Total (No. of projects)

97 146 196 238 287 337 385

The close correlation between company and project performance reflects that these ratings may be identical (as in greenfield investments) or closely interrelated. The above tables show that a large number of projects were assigned the highest performance rating just before the mid-1990s, but this level dropped thereafter, although an improvement is shown in 2002 and 2003. This data reflects the EBRD’s strategy to move further east, increasing its share in Russia and other CIS countries, which on average have riskier investment environments than central eastern Europe and the Baltic states. Most of the early projects (either in public infrastructure or privatised/private companies) were in central eastern Europe and the Baltic states. At the time of investment, these countries had strong government and/or strategic investor support from leading Western firms. The Bank's role in the private sector was often to provide political comfort to strong strategic investors. During the mid-1990s, over one-third of projects were assigned Excellent company and project performance ratings. Since then, however, there has been a falling share of projects in this region with the highest project and company performance ratings. The additionality requirement led the Bank taking higher risk projects in the CIS and countries further East. The Bank’s ETC initiative will try to enhance project financial performance in this region.

Appendix 8 Page 17 of 23

5.2 COMPANY AND PROJECT FINANCIAL PERFORMANCE RATINGS BY COUNTRY GROUPS It can be seen from Charts 6.3 and 6.4 that a geographical breakdown of evaluated projects produces results that mirror those for other ratings: Russia and the ETCs have fewer projects rated Good or Excellent than other regions. In addition, Russia has the greatest number of projects with Unsatisfactory or Marginal ratings for company and project performance.

Charts 5.3 and 5.4: Company and project financial performance ratings by country groups (322 investment operations)

Chart 5.3 Company performance

25%15% 14%

7% 8%

38%49%

30%

24%

60%

18% 23%

27%45%

20%

17% 11%28% 21%

8%4%2%2%2%2%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

CEEurope &

Baltics(177)

SEEurope

(61)

Russia(64)

ETCs(42)

OtherCIS (25)

Region

% o

f eva

luat

ed p

roje

cts NA

Unsatis-factoryMarginal

Good

Excellent

Chart 5.4 Project performance

23% 18% 13% 10% 12%

37% 48%

34%

19%

64%

20%25%

27%

45%

8%

18%10%

27% 26%12%

4%2%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

CEEurope &

Baltics(177)

SEEurope

(61)

Russia(64)

ETCs(42)

OtherCIS (25)

Region

% o

f eva

luat

ed p

roje

cts Unsatis-

factoryNA

Marginal

Good

Excellent

N.B. Sixteen Regional operations are omitted

Appendix 8 Page 18 of 23

6. FULFILMENT OF PROJECT OBJECTIVES Fulfilment and relevance of project objectives is assessed against the objectives submitted at project approval. Chart 6.1 presents post-signing evaluation ratings on objective fulfilment. The pattern and trend resemble the finding for company and project performance (see Section 5 above). The cumulative ratings for all evaluated projects have dropped slightly over the years and the underlying reasons are similar to those affecting project and company performance: more projects in riskier countries, and additionality requirements in project selection.

Chart 6.1 and Table 6.1: Fulfilment of objectives, percentage distribution of assigned ratings (385 investment operations)

Chart 6.1 Table 6.1

1993-95

1993-96

1993-97

1993-98

1993-99

1993-2000

1993-2001

1993-2002

1993-2003

Unsatis-factory

7% 5% 5% 7% 10% 10% 12% 12% 10%

Marginal 7% 7% 6% 13% 16% 16% 17% 18% 19%Good 22% 25% 28% 34% 33% 34% 36% 38% 40%Excellent 63% 63% 61% 46% 41% 39% 35% 32% 31%

Total (No. of projects)

27 61 97 146 196 238 287 337 385

As could be expected, there is a high correlation between project and company financial performance and fulfilment of objectives. The apparent falling trend mostly reflects the increasing share of evaluated projects in Russia and early transition stage countries further east. This trend may not continue as the last two years tend to indicate some turnaround. 7. THE ENVIRONMENT 7.1 The environmental requirement Projects are designed and conditioned to fulfil all aspects of the Bank's mandate, including the environmental policy of the Bank at the time of appraisal. Environmental ratings form part of the overall performance rating. Environmental evaluation concerns the physical environment, as well as occupational health and safety, and issues such as public consultation. The analysis in this Appendix refers to 318 evaluated projects during 1996-2003. OPERs were rated for these criteria from 1996 and XMRAs from 1998. 7.2 Environment rating system The series from 1996-2003 covers two environmental dimensions. The first dimension concerns environmental performance of the sponsor and of the Bank, e.g. environmental due diligence and audits; the preparation and implementation of environmental action plans; compliance with contractual environmental conditions and statutory regulations etc. The second dimension is the extent of environmental change (positive or negative) brought about by the evaluated operation.

0%

10%

20%

1993 1995 1997 1999 2001 2003

Year of evaluation

30%

40%

50%

60%

70%

80%

90%

100%

% o

f eva

luat

ed p

roje

cts

NA

Unsatis-factoryMarginal

Good

Excellent

Appendix 8 Page 19 of 23

The rating scale for environmental performance changed in 2003 from a four-point to a six-point scale: Excellent, Good, Satisfactory, Marginal, Unsatisfactory and Highly Unsatisfactory. Previously this was a four-point scale: Excellent, Good, Marginal and Unsatisfactory. As most of the available evaluation data is based on the four point scale, here PED reports in the four point scale, with the new Satisfactory category added to the old Good category. While PED has also added a Highly Unsatisfactory category, to date, no projects have received this rating. As the population expands in the future, PED will start reporting on the six-point scale. The range of ratings of environmental change is: Outstanding, Substantial, Some, None and Negative. Also in 2004, the bank handling aspects of environmental performance were dropped from this indicator and are assessed as part of overall Bank Handling. 7.3 Evolution of environmental ratings The Charts and Tables 7.1 and 7.2 present ratings of environmental performance and of the extent of environmental change as assigned to 318 evaluated projects in 1996-2003.

Charts 7.1 and 7.2: Environmental ratings distribution, 1996-2003 (318 investment operations)

Chart 7.1: Environmental performance Chart 7.2: Extent of environmental change

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1996

Ye

% o

f eva

luat

ed p

roje

cts

1998 2000 2002

ar of evaluation

NA

Unsatis-factoryMarginal

Good

Excellent

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1996 1998 2000 2002

Year of evaluation

% o

f eva

luat

ed p

roje

cts NA

Negative

None

Some

Substantial

Outstanding

Table 7.1 Table 7.2 1996-

98 1996-99

1996-2000

1996-2001

1996-2002

1996 2003

1996-98

1996-99

1996-2000

1996-2001

1996-2002

1996 2003

Unsatisfactory 5% 5% 4% 4% 3% 3% Negative 0% 0% 0% 0% 0% 1% Marginal 11% 12% 13% 14% 15% 14% None 20% 26% 23% 21% 22% 22% Subtotal 16% 17% 17% 18% 18% 17% Some 46% 48% 50% 53% 54% 54%

Good 61% 56% 56% 59% 59% 61% Subtotal 66% 74% 73% 74% 76% 77% Excellent 20% 26% 25% 21% 20% 19% Substantial 29% 22% 21% 20% 19% 19% Subtotal 81% 82% 81% 80% 79% 80% Outstanding 3% 2% 2% 2% 1% 1%

Not applicable 3% 2% 2% 2% 3% 3% Subtotal 32% 24% 23% 22% 20% 20% No of projects 79 129 171 220 270 318 NA 3% 2% 4% 3% 3% 3% No. of

projects 79 129 171 220 270 318

80 per cent of evaluated operations obtained a Satisfactory/Good or better rating of environmental performance of the sponsor and the Bank. A total of 14 per cent were rated

Appendix 8 Page 20 of 23

Marginal in this respect and only 3 per cent were evaluated as having Unsatisfactory performance. None were rated Highly Unsatisfactory (3 per cent of projects were given Not Applicable ratings). The ratings confirm that the Bank has generally been successful in improving the environmental performance of projects, with very few exceptions. The extent of environmental change of evaluated projects was rated as Substantial or Outstanding in 20 per cent of the cases, Some for 54 per cent and None for 22 per cent (3 per cent were rated Not Applicable and 1% negative). Financial institutions and SME financing projects involve lower environmental risk than operations in natural resources, power and energy, heavy industry, chemical and process industries. Investments in heavy industry, chemical processing etc. may have higher potential for environmental change in “brown-field” projects, but they also present higher or excessive turnaround and financial sustainability risks. Two projects resulted in Negative environmental change. Both involved project expansions without corresponding investments in agreed to pollution control infrastructure, thus resulting in greater pollution than the without project scenario. 8. BANK HANDLING The quality of the Bank's handling of operations has been assessed in XMR Assessments since 1999, but has only been formally rated in OPER reports from 2002. This measure was included in the AEOR for the first time last year. The figures in Table 8.1 and Chart 8.1 take account of XMR assessments since 1999 and OPER reports since 2002, a total of 180 reports.

Chart 8.1 and Table 8.1: Ratings of Bank handling

Chart 8.1 Table 8.1

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1999 2000 2001 2002 2003

Year of evaluation

% o

f eva

luat

ed p

roje

cts Unsatis-

factory

Marginal

Good

Excellent

The results show that 76 per cent of the operations rated for bank handling have achieved a rating of Good or Excellent. However, about a quarter of evaluated projects obtained an Unsatisfactory or Marginal rating. This group of projects, in particular, generated important lessons learned.

Ratings 1999-2000 1999-2001 1999-2002 1999-2003

Unsatisfactory 15% 13% 9% 7%

Marginal 13% 12% 17% 17%

Subtotal 28% 25% 26% 24% Good 40% 49% 51% 52%

Excellent 32% 26% 23% 24%

Subtotal 72% 75% 74% 76%

Total (No. of projects)

53 82 132 180

Appendix 8 Page 21 of 23

9. THE BANK’S INVESTMENT PERFORMANCE In 2000 the Bank introduced improved models for ex-ante investment return projections. These models are to be used by the banking department on projects at the approval stage. PED, in consultation with the Strategic and Corporate Planning and Budgeting Department (SCPB), adapted the new formulae for analysing the Bank’s investment performance. The formulae will provide more actual cost elements and reflection of risk ratings. Chart 9.1: The Bank’s investment performance, ratings distribution in OPER reports 2000-03

Outstanding Excellent Satisfactory Marginal

Inadequate Unsatisfactory No. of reports

2000 11% 6% 44% 17% 0% 22% 18 2001 5% 5% 48% 16% 5% 21% 19 2002 0% 18% 27% 32% 9% 14% 22 2003 0% 33% 44% 11% 6% 6% 18

2000-01 8% 5% 46% 16% 3% 22% 37 2000-02 5% 10% 39% 22% 5% 19% 59

2000-2003 4% 16% 40% 19% 5% 16% 77 Chart 9.1 shows the outcome of this analysis, which is based on evaluations conducted since 2000. The sample number is still relatively small as only OPER reports are rated for this indicator. Over the four years, 60 per cent of fully evaluated operations achieved a Satisfactory or better rating for investment performance. 10. SIZE AND REPRESENTATION OF THE SAMPLE OF EVALUATED

PROJECTS 10.1 SAMPLE SIZE The full evaluated sample comprises OPER and XMR assessment reports by PED on 385 projects evaluated in 1993-2003. The majority, or 358 projects, date from 1996 onwards.21 The projects were selected with structured sampling to combine the level of representation with potential for useful lessons. The annual coverage was 100 per cent by the end of 1996 and well over 60 per cent thereafter (see Appendix 8, Chart 1.1). The sample represents 76 per cent of all 508 investment projects that had reached the ready-for-evaluation stage. The evaluated share of all signed projects by end 2003 was lower, at 41 per cent, as many more recently approved projects were still under implementation.22 Charts 10.1-10.9 below illustrate the level of representation in respect of countries, sectors and facility risk ratings.

21 See annotations of the respective tables and charts. Those which concern broader sector and country distribution analysis cover

evaluations 1993-2003. 22 See Appendix 5 for more detailed data; records on signed operations may also at times split one operation into separate facilities.

Appendix 8 Page 22 of 23

10.2 COUNTRY REPRESENTATION The charts below confirm a good level of representation of the evaluated sample in respect of country coverage compared with the projects ready for evaluation.

Country distribution23

Chart 10.1: Cumulative signed Chart 10.2: All projects ready Chart 10.3: PED’s EBRD operations for evaluation evaluated sample

19%

20%39%

6%16%

Regional SE EuropeCIS excl. Russia CE EuropeRussia

18%

17%

48%

3%14%

Regional SE EuropeCIS excl. Russia CE EuropeRussia

16%

17%

48%

3%16%

Regional SE EuropeCIS excl. Russia CE EuropeRussia

The evaluated sample in respect of country coverage, however, differs somewhat from the population of signed projects. Most of the projects ready for evaluation in the first few years of the bank's existence were in Central Europe, reflecting the Bank's portfolio at the time. More recent commitments have a higher share in countries of the CIS and in south-eastern Europe, many of which are not yet ready for evaluation. Regional projects are under-represented because they are often frameworks with sub-operations, and take longer to reach maturity than a simple stand-alone operation. 10.3 SECTOR REPRESENTATION Charts 10.4 to 10.6 present the comparative sector distribution. There is an over-representation in the evaluated sample in respect of general industry at the expense of infrastructure. This reflects the fact that infrastructure projects have long gestation periods, tending to delay their evaluation.

Sector distribution of projects by numbers of projects:

Chart 10.4: Cumulative signed Chart 9.5: All projects ready Chart 9.6: PED’s EBRD operations for evaluation evaluated sample

37%

22%

41%

Industry & Commerce Financial Institutions

Infrastructure

36%

18%

46%

Industry & Commerce Financial Institutions

Infrastructure

35%

15%

50%

Industry & Commerce Financial Institutions

Infrastructure

23 Weighting by volume did not change the below picture significantly.

Appendix 8 Page 23 of 23

10.4 FACILITY RISK RATING LEVEL OF REPRESENTATION The following charts present overall portfolio facility risk ratings (as at 31 December 2003), comparing the “ready for evaluation” group with the evaluated sample and the signed portfolio as a whole. It can be concluded that the 22 per cent of projects with a facility risk rating of 7-10 in the sample of evaluated projects, compares well with the portfolio of signed projects where 19 per cent of the projects are rated between 7-10 for their facility risk.

Facility risk distribution by numbers of projects:

Chart 10.7: Cumulative signed Chart 10.8: All projects ready Chart 10.9: PED’s EBRD operations for evaluation evaluated sample

17%

64%

19%

1-4 5, 6 & 6W 7-10

21%

56%

23%

1-4 5, 6 & 6W 7-10

22%

56%

22%

1-4 5, 6 & 6W 7-10

10.5 A GOOD LEVEL OF REPRESENTATION IS CONFIRMED FOR EVALUATED PROJECTS The sample of 385 evaluated investment projects by the end of 2003 represents well all the projects ready for evaluation, with a 76 per cent sample coverage. There is also an adequate representation of the signed portfolio, but its gradually changing country and sector patterns will not be captured fully until the more recent projects become ready for evaluation.

Table of TC Funds

Appendix 9

DonorFund Country No. % Amount % Amount %ALBANIA 0 0.00% 0 0.00% 40 0.00%AUSTRALIA 0 0.00% 0 0.00% 80 0.01%AUSTRIA 0 0.00% 0 0.00% 4,001 0.45%BELARUS 0 0.00% 0 0.00% 431 0.05%BELGIUM 0 0.00% 0 0.00% 1,428 0.16%CANADA 4 2.20% 1,081 1.91% 16,676 1.89%DENMARK 3 1.65% 229 0.40% 4,875 0.55%FINLAND 3 1.65% 241 0.43% 8,056 0.91%FRANCE 4 2.20% 544 0.96% 16,114 1.83%GERMANY 4 2.20% 1,110 1.96% 12,963 1.47%GREECE 0 0.00% 0 0.00% 1,000 0.11%ICELAND 1 0.55% 50 0.09% 206 0.02%IRELAND 3 1.65% 88 0.16% 2,054 0.23%ISRAEL 1 0.55% 38 0.07% 306 0.03%ITALY 3 1.65% 115 0.20% 37,856 4.29%JAPAN 44 24.18% 12,267 21.71% 96,963 10.99%LUXEMBOURG 0 0.00% 0 0.00% 1,792 0.20%NETHERLANDS 11 6.04% 2,705 4.79% 35,663 4.04%NEW ZEALAND 1 0.55% 68 0.12% 175 0.02%NORWAY 1 0.55% 908 1.61% 4,078 0.46%PORTUGAL 1 0.55% 19 0.03% 468 0.05%REPUBLIC OF KOREA 0 0.00% 0 0.00% 304 0.03%SPAIN 0 0.00% 0 0.00% 2,586 0.29%SWEDEN 2 1.10% 321 0.57% 9,250 1.05%SWITZERLAND 6 3.30% 1,490 2.64% 12,828 1.45%TAIPEI CHINA 7 3.85% 876 1.55% 15,767 1.79%TURKEY 1 0.55% 105 0.19% 300 0.03%UNITED KINGDOM 6 3.30% 1,521 2.69% 34,518 3.91%UNITED STATES 4 2.20% 2,251 3.98% 52,271 5.92%Other donorsEU 71 39.01% 30,387 53.77% 383,919 43.51%European Agency for Reconstruction 0 0.00% 0 0.00% 1,200 0.14%Gobal Environment Facility 0 0.00% 0 0.00% 1,003 0.11%UNDP (TAM) 0 0.00% 0 0.00% 1,237 0.14%RPC (TAM) 0 0.00% 0 0.00% 306 0.03%Chevron Muaigas Inc1 0 0.00% 0 0.00% 376 0.04%Multiple-donor funds2 1 0.55% 95 0.17% 120,045 13.60%Financial Sector3 0 0.00% 0 0.00% 1,253 0.14%TOTAL 182 100.00% 56,509 100.00% 882,384 100.00%

1 First private donor to TC funds

Portfolio data from Funds Reporting and Datawarehouse December 2003OPER report data from Datawarehouse December 2003

2 Funds include TAM Nordic Council, Baltic Fund, RSBF, EBRD TC Special Fund, Balkan Region Fund, Mongolia TC Fund, RVF for North West Russia, EBRD Annual General Meeting 2000. Donors include the G-7, Nordic countries, Japan, Taipei China, Luxembourg, Netherlands, EC, Russia, Switzerland3 Contributions to these funds consist of technical assistance fees payable by the borrowers under the terms of loan agreements between EBRD and certain financial intermediaries.

Covered by OPER reports Portfolio-wide

Appendix 10 Page 1 of 2

EVALUATION OF INVESTMENT OPERATIONS

THE PROCESS

XMR Preparation by Operation

Leader in Banking Department on all projects ready for

evaluation

(thaX

X

r

Review of XMRs by Team Leader, and all

other departments involved in the operation

process (signing-off process)

XMRs are filed in operation file and integrated in

XMR Assessments and

OPER reports

100% coverage

XMR

Review ose not covered by n OPER report or MR Assessment)

Investment Operations Self Evaluation

(Banking Department) *******************

Direct Evaluation of Investment Operations

by PED

XMR Assessment

(about 30 from crop of ready operations)

Selection: Random

sampling

Special Studies &

Mid-Term Reviews (5 reports)

Sample coverage

FeedbaMechan

• Distribution• Workshops • Direct conta

bankers • XMR traini

LESSONS LEARNED D

In-depth OPER and OPER

reports (22 reports from

crop of ready operations)

Selection: lessons

Review of all MRs by PED on

quality. Upgrade equired in case of

low quality

ck ism reports at HQ & ROs cts with

ng

ATABASE

Appendix 10 Page 2 of 2

EVALUATION OF TC OPERATIONS

THE PROCESS

S

****

Inde

As(about 2

from

Sample

PCR Preparation Operation

Leader BD

PCR Clearance Team Leader

(BD)

PCR

Review (OCU)

PCR Acceptance

(OCU)

PCR Central File

(OCU) LES

TC Coverage elf Evaluation (BD/OCU)

***************TC Coverage

pendent Evaluation(PED)

100% coverage

PCR sessment 0 cases selected the reviews)

TC OPER reports (6 reports)

Special Studies &

Mid-Term Reviews (5 reports)

coverage

SONS LEARNED DATABASE

Feedback Mechanism