The World Bank and Public Procurement—An Independent ... · Table A.2. Bank Procurement —Support to Integrity, Transparency, Cost, Quality, and Timelines Question Avg Country

The World Bank and Public Procurement—An Independent Evaluation

Appendixes to Volume II: Achieving Development Effectiveness through Bank Procurement

ii

Contents ABBREVIATIONS ..................................................................................................................................................................... I APPENDIX A. BANK GUIDELINES AND PROCESSES ........................................................................................................ 3

APPENDIX B. CDD PROCUREMENT ................................................................................................................................... 31 PPP PROCUREMENT: COMPARISON OF THE BANK AND OTHER ORGANIZATIONS.................................................. 55

APPENDIX C. PROCUREMENT RISK CONCEPTS AND THRESHOLDS ........................................................................... 61

APPENDIX D. PROCUREMENT TRACKING SYSTEMS, TIME ANALYSIS, AND AFRICA REGION REVIEW ................. 89 BIBLIOGRAPHY .................................................................................................................................................................. 110

ENDNOTES .......................................................................................................................................................................... 119

Abbreviations ADB Asian Development Bank AfDB African Development Bank APEC Asian Pacific Economic Cooperation CDD Community-driven development CPAR Country procurement assessment

report CPIA Country policy and institutional

assessment DfID Department for International

Development FMIS Financial management information

system HDI United Nations Human Development

Index IAD Internal Audit Vice Presidency IAEA The International Atomic Energy

Agency ICB International competitive bidding

process ICR Implementation completion report ICRR Implementation completion and

results reports ICT Information technology IDA International Development Agency IDB Inter-American Development Bank IEG Independent Evaluation Group IFC International Finance Corporation INT Integrity Vice-Presidency ISDB Islamic Development Bank

ISR Implementation status report MAPS Methodology for Assessing

Procurement Systems MDB Multilateral development bank MEAT Most economically advantageous

tender MRRD Ministry of Rural Rehabilitation and

Development NCB National competitive bidding NEPE State Technical Unit OPRC Operational Procurement Review

Committee OPCS Operations Policy and Country

Services PAD Project appraisal document PPP Public-private partnership P-RAMS Procurement risk assessment

management system. PROCYS Procurement Cycle Tracking System—

Africa Region RPM Regional procurement managers SEPA Procurement plan execution system SME Small and medium-sized enterprises TOR Terms of reference UNCITRAL United Nations Commission on

International Trade Law WBI World Bank Institute WTO World Trade Organization

Appendix A. Bank Guidelines and Processes

Overall Perceptions

Table A.1. Bank Procurement Methods—Adequacy of Competition

Question Avg Country

mgmt

Bank proc staff

Bank TTLs

Country clients

Private sector

Civil society

Do the Bank's Procurement procedures ICB, NCB, and Consultants Selection Procedures in particular) lead to adequate competition in terms of numbers of bidders, openness and fairness, geographical reach, etc.?

2.9

2.9

3.2

2.8

3.3

2.9

2.5

Source: IEG questionnaire. Note: 4 = more than adequate; 3 = adequate; 2 = only somewhat adequate; 1 = inadequate; TTL = task team leader.

Table A.2. Bank Procurement—Support to Integrity, Transparency, Cost, Quality, and Timelines


mgmt

Bank proc staff

Bank TTLs

Country clients

Private sector

Civil society

To what extent do the Bank's procurement requirements positively support the integrity, transparency, cost, quality, and timeliness of delivering public sector projects?

3.1

2.6

3.6

3.0

3.5

3.0

2.8


Table A.3. Bank Procurement—Capturing Bidder Interest


mgmt

Bank proc staff

Bank TTLs

Country clients

Private sector

Civil society

Are bidders s more likely to show interest if the project is under Bank procurement policy and procedures, as opposed to a project under national guidelines and processes?

3.2 2.8 3.6 3.0 3.2 3.2 3.2


APPENDIX A BANK GUIDELINES AND PROCESSES

4

Table A.4. Bank Procurement—Emphasis on Price Factors Question

Avg

Bank proc staff

Bank TTLs

Country clients

Private sector

To what extent does the Bank's oversight (no-objection) of prequalification/ short-listing, bidding documents/RFP, bid/proposal evaluation, and contract award emphasize price?

2.9 2.9 2.7 2.9 3.0


Table A.5. Bank Procurement—Flexibility in Response

Question Avg

Bank proc staff

Bank TTLs

Country clients

Is the Bank able to respond quickly to special circumstances such as those requiring deviations from policies and procedures, or changes to procurement provisions of the financing agreement, or changes to previously agreed arrangements?

2.3 2.7 2.1 2.2


Table A.6. Bank Procurement—Incidence of Delay in Contract Award

Question Avg Ctry

mgmt

Bank Proc Staff

Bank TTLs

Ctry clients

Private sector

Civil society

How frequent is the incidence of delay in the award of contracts? 2.9 3.2 2.5 2.9 2.7 2.9 3.2


Methodology for Assessing Procurement Systems Indicators and Scores

Table A.7. MAPS Indicators and Passing Scores Under the Piloting Program UCS required

score Pillar I: Legislative and Regulatory Framework Indicator 1: Public procurement legislative and regulatory framework achieves the agreed standards and complies with applicable obligations.

Sub-indicator 1(a): Scope of application and coverage of the legislative and regulatory framework. 3 Sub-indicator 1(b): Procurement Methods 2+ Sub-indicator 1(c): Advertising rules and time limits 3 Sub-indicator 1(d): Rules on Participation 3 Sub-indicator 1(e): Tender documentation and technical specifications 3 Sub-indicator 1(f): Tender evaluation and award criteria 3 Sub-indicator 1(g): Submission, receipt and opening of tenders 3

5

UCS required score

Sub-indicator 1(h): Complaints 3 Indicator 2: Existence of Implementing Regulations and Documentation. Sub-indicator 2(a): Implementing regulation (sic) that provide defined processes and procedures not included in higher-level legislation

2

Sub-indicator 2(b): Model tender documents for goods, works, and services 2 Sub-indicator 2(c): Procedures for pre-qualification 2+ Sub-indicator 2(d): Procedures suitable for contracting for services or other requirements in which technical capacity is a key criterion.

2+

Sub-indicator 2(e): User's guide or manual for contracting entities 2 Sub-indicator 2(f): General Conditions of Contracts (GCC) for public sector contracts covering goods, works and services consistent with national requirements and, when applicable, international requirements

3

Pillar II: Institutional Framework and Management Capacity Indicator 3: The public procurement system is mainstreamed and well integrated into the public sector governance system.

Sub-indicator 3(a): Procurement planning and associated expenditures are part of the budget formulation process and contribute to multiyear planning.

2

Sub-indicator 3(b): Budget law and financial procedures support timely procurement, contract execution, and payment.

2

Sub-indicator 3(c): No initiation of procurement actions without existing budget appropriations. 2 Sub-indicator 3(d): Systematic completion reports are prepared for certification of budget execution and for reconciliation of delivery with budget programming.

2

Indicator 4: The country has a functional normative/regulatory body. Sub-indicator 4(a): The status and basis for the normative/regulatory body is covered in the legislative and regulatory framework.

3

Sub-indicator 4(b): The body has a defined set of responsibilities including the following… 2 Sub-indicator 4(c): The body's organization, funding, staffing, and level of independence and authority (formal power) to exercise its duties should be sufficient and consistent with the responsibilities.

2

Sub-indicator 4(d): The responsibilities should also provide for separation and clarity so as to avoid conflict of interest and direct involvement in the execution of procurement transactions.

3

Indicator 5: Existence of institutional development capacity. Sub-indicator 5(a): The country has a system for collection and disseminating procurement information, including tender invitations, requests for proposals, and contract award information.

2

Sub-indicator 5(b): The country has systems and procedures for collecting and monitoring national procurement statistics.

2

Sub-indicator 5(c): A sustainable strategy and training capacity exists to provide training advice and assistance to develop the capacity of government and private sector participation to understand the rules and regulations and how they should be implemented.

2

Sub-indicator 5(d): Quality control standards are disseminated and used to evaluate staff performance and address capacity development issues.

2


6

UCS required score

Pillar III: Procurement Operations and Market Practices Indicator 6: The country's procurement operations and practices are efficient. Sub-indicator 6(a): The level of procurement competence among government officials within the entity is consistent with their procurement responsibilities.

2

Sub-indicator 6(b): The procurement training and information programs for government officials and for private sector participants are consistent with demand.

2

Sub-indicator 6(c): There are established norms for the safekeeping of records and documents related to transactions and contract management.

2+

Sub-indicator 6(d): There are provisions for delegating authority to others who have the capacity to exercise responsibilities.

2

Indicator 7: Functionality of the public procurement market Subindicator 7(a): There are effective mechanisms for partnerships between the public and private sector. 2 Sub-indicator 7(b): Private sector institutions are well organized and able to facilitate access to the market. 2 Sub-indicator 7(c): There are no major systemic constraints (e.g. inadequate access to credit, contracting practices, etc.) inhibiting the private sector's capacity to access the procurement market.

2+

Sub-indicator 7(d) – Clarity and transparency of rules for determining whether to engage international or national markets. *Only analyzed for 3 countries.

2

Indicator 8: Existence of contract administration and dispute resolution provisions Sub-indicator 8(a): Procedures are clearly defined for undertaking contract administration responsibilities that include inspection and acceptance procedures, quality control procedures, and methods to review and issue contract amendments in a timely manner.

2

Sub-indicator 8(b): Contracts include dispute resolution procedures that provide for an efficient and fair process to resolve disputes arising during the performance of the contract.

3

Sub-indicator 8(c): Procedures exist to enforce the outcome of the dispute resolution process. 3 Pillar IV: Integrity and Transparency of the Public Procurement System Indicator 9: The country has effective control and audit systems. Sub-indicator 9(a): A legal framework, organization, policy, and procedures for internal and external control and audit of public procurement operations are in place to provide a functioning control framework.

2+

Sub-indicator 9(b): Enforcement and follow-up on findings and recommendations of the control framework provide an environment that fosters compliance.

2+

Sub-indicator 9(c): The internal control system provides timely information on compliance to enable management action.

2+

Sub-indicator 9(d): The internal control systems are sufficiently defined to allow performance audits to be conducted.

2

Sub-indicator 9(e): Auditors are sufficiently informed about procurement requirements and control systems to conduct quality audits that contribute to compliance.

2

Indicator 10: Efficiency of appeals mechanism. Sub-indicator 10(a): Decisions are deliberated on the basis of available information, and the final decision can be reviewed and rules upon by a body (or authority) with enforcement capacity under the law.

3

Sub-indicator 10(b): The complaint review system has the capacity to handle complaints efficiently and a means to enforce the remedy imposed.

3

Sub-indicator 10(c): The system operates in a fair manner, with outcomes of decisions balanced and justified on the basis of available information.

3

Sub-indicator 10(d): Decisions are published and made available to all interested parties and to the public. 2 Sub-indicator 10(e): The system ensures that the complaint review body has full authority and independence for resolution of complaints.

3

Indicator 11: Degree of access to information Sub-indicator 11(a): Information is published and distributed through available media with support from information technology when feasible.

2+

Indicator 12: The country has ethics and anticorruption measures in place. Sub-indicator 12(a): The legal and regulatory framework for procurement, including tender and contract documents, includes provisions addressing corruption, fraud, conflict of interest, and unethical behavior and sets out (either directly or by reference to other laws) the actions that can be taken with regard to such behavior.

3

Sub-indicator 12(b): The legal system defines responsibilities, accountabilities, and penalties for individuals and firms found to have engaged in fraudulent or corrupt practices.

3

Sub-indicator 12(c): Evidence of enforcement of rulings and penalties exists. 2+

7

UCS required score

Sub-indicator 12(d): Special measures exist to prevent and detect fraud and corruption in public procurement. 3 Sub-indicator 12(e): Stakeholders (private sector, civil society, and ultimate beneficiaries of procurement/end-users) support the creation of a procurement market known for its integrity and ethical behaviors.

2

Sub-indicator 12(f): The country should have in place a secure mechanism for reporting fraudulent, corrupt, or unethical behavior.

3

Sub-indicator 12(g): Existence of Codes of Conduct/Codes of Ethics for participants that are involved in aspects of the public financial management systems that also provide for disclosure for those in decision making positions.

2

National Competitive Bidding Requirements and Compliance

Table A.8. Bank NCB Requirements and Country Systems

AZ BD ET ID MA PE PH SN

Reports Reviewed

Countries Reviewed

Field Visit Field Visit Field Visit Field Visit/UCS Field Visit /UCS

Field Visit

Field Visit/UCS Field Visit/UCS

35 26 2002 CPAR

2005 LA

2008 LA

2005 LA

2010 LA

2012 CPAR Draft

2008 LA

2001 CPAR

2004 LA

2005 LA

2012 LA

2011 LA

2004 LA

2009 LA

2012 CPAR 2009 LA

Bank Requirements for NCB There may be no regional / domestic preferences regarding the sources of labor and material.1 18 15 1 1 1 1 1 1 1 1 1 1 Prospective bidders must be allowed at least 30 days for bid preparation.2 19 14 1 1 1 1 1 1 1 1 1 Appropriate standard bidding and prequalification documents must be used. 15 14 1 1 1 1 1 Eligibility cannot be restricted based on nationality of bidder and/or origin of goods.3 17 13 1 1 1 1 1 1 1 1 Award must be made to the lowest evaluated qualified and responsive bidder. 15 13 1 1 1 1 1 1 Minimum requirements must be explicitly stated in the documents. 12 12 1 1 1 1 Bidding opportunities must be advertised in the local press. 11 11 1 1 1 1 Bidders are not generally required to register with a local or federal authority except under certain conditions.4 13 11 1 1 1 1 1 1 1 The procurement process cannot be cancelled, all bids rejected, and/or rebidding conducted without approval. 13 11 1 1 1 1 1 1 1 1 Public bid opening is required. 11 9 1 1 1 1 1 1 Bid evaluation criteria other than price may be allowed only if quantified in monetary terms. 8 8 1 Price negotiations may not be conducted with “winning” bidders prior to contract signature.5 10 8 1 1 1 1 Parastatals may only be allowed to bid under certain conditions.6 9 7 1 1 1 Bids may not be rejected based only on a comparison with the procuring entity's estimate; invitations to bid shall not establish minimums and maximums.7

11 7 1 1 1 1 1 1 1 Audits and inspection of records related to bid submission and performance of the supplier are permitted. 8 7 1


9



Reports Reviewed

Countries Reviewed


Field Visit


35 26 2002 CPAR

2005 LA

2008 LA

2005 LA

2010 LA

2012 CPAR Draft

2008 LA

2001 CPAR

2004 LA

2005 LA

2012 LA

2011 LA

2004 LA

2009 LA

2012 CPAR 2009 LA

“Two envelope” bid opening procedure is permitted for procurement of goods or works under specific conditions, notably if domestic law precludes use of one envelope.8

6 6 1 1 1 Foreign firms' eligibility cannot be conditioned on joint ventures with local firms. 7 5 1 1 1 1 Qualification criteria shall be applied on a pass/fail basis. 5 5 1 Extension of the time period to prepare bids may only be allowed under exceptional circumstances. 5 5 An inflation clause is recommended for contracts over a year.9 7 5 1 1 1 Bidding documents are freely available.10 7 4 1 1 1 1 1 1 Bid security shall be in the form of a letter of credit or bank guarantee from a reputable bank.11 5 4 1 1 1 1 Joint venture partners must be jointly and severally liable. 6 3 1 1 1 1 1 There may be no restrictions on the means of delivery of bids.12 3 3 Pre-qualification should be used only for large works projects. 5 3 1 1 1 1 Award must be published. 4 3 1 1 Bidding documents and contract shall include provisions on sanctions for fraud and corruption. 3 3 No preference may be given to suppliers or contractors based on region or locality of registration, small size, ethnic ownership, etc.

4 2 1 1 1 1 For contracts subject to prior review under the Loan Agreement, scope/conditions may not be modified during implementation without Bank approval.

6 2 1 1 1 1 1 1 Bidders shall not be eliminated on the basis of minor deviations. 4 2 1 1 1 Contractors/suppliers must be prequalified for large or specialized contracts. 2 1 1 1 An invitation to prequalify must be advertised for each procurement involving large or complex potential contracts. 1 1 1 There must be no set limitations to the number of firms who can bid for a contract. 1 1 Requests for clarification of bidding documents must be made in writing, and response should be sent to all prospective bidders. 1 1


10



Reports Reviewed

Countries Reviewed


Field Visit


35 26 2002 CPAR

2005 LA

2008 LA

2005 LA

2010 LA

2012 CPAR Draft

2008 LA

2001 CPAR

2004 LA

2005 LA

2012 LA

2011 LA

2004 LA

2009 LA

2012 CPAR 2009 LA

The experience requirement for works contracts shall be (i) at least one previous contract at 80% of cost of contract being procured and (b) an average annual turnover of 100% of contract being procured.

2 1 1 1

Bidders shall be given at least twenty-eight (28) days from the receipt of notification of award to submit performance securities.

1 1 Bids shall not be disclosed excepted as needed for evaluation without bidder's written authorization until the award of the contract.

1 1 Automatic rebidding may be required if too few bids are received.13 4 1 1 1 1


11

TZ TR BT BR BF MK MU PA PL GH CO LA VN MZ SL AL BA EG Reports Reviewe

d

Countries Reviewed

Field Visit

Field Visit/ UCS

UCS UCS UCS UCS UCS UCS UCS UCS Desk Sampl

e

Desk Sample

Desk Sample

Desk Sample

Desk Sample

Desk Sample

Other Other

35 26 2009 LA

2008 LA

2012 LA

2010 LA

2011 LA

2012 LA

2003 CPAR

excerpt 2009 LA

2007 LA

2007 LA

2009 LA

2011 LA

2010 LA

2002 CPAR

2010 LA

2012 CPAR

2012 LA

2002 CPAR

2002 CPAR

Bank Requirements for NCB There may be no regional / domestic preferences regarding the sources of labor and material.1 18 15 1 1 1 1 1 1 1 1 Prospective bidders must be allowed at least 30 days for bid preparation.2 19 14 1 1 1 1 1 1 1 1 1 1 Appropriate standard bidding and prequalification documents must be used. 15 14 1 1 1 1 1 1 1 1 1 1 Eligibility cannot be restricted based on nationality of bidder and/or origin of goods.3 17 13 1 1 1 1 1 1 1 1 1 Award must be made to the lowest evaluated qualified and responsive bidder. 15 13 1 1 1 1 1 1 1 1 1 Minimum requirements must be explicitly stated in the documents. 12 12 1 1 1 1 1 1 1 1 Bidding opportunities must be advertised in the local press. 11 11 1 1 1 1 1 1 1 Bidders are not generally required to register with a local or federal authority except under certain conditions.4 13 11 1 1 1 1 1 1 The procurement process cannot be cancelled, all bids rejected, and/or rebidding conducted without approval. 13 11 1 1 1 1 1 Public bid opening is required. 11 9 1 1 1 1 1 Bid evaluation criteria other than price may be allowed only if quantified in monetary terms. 8 8 1 1 1 1 1 1 1 Price negotiations may not be conducted with “winning” bidders prior to contract signature.5 10 8 1 1 1 1 1 1 Parastatals may only be allowed to bid under certain conditions.6 9 7 1 1 1 1 1 1 Bids may not be rejected based only on a comparison with the procuring entity's estimate; invitations to bid shall not establish minimums and maximums.7

11 7 1 1 1 1 Audits and inspection of records related to bid submission and performance of the supplier are permitted. 8 7 1 1 1 1 1 1 1 “Two envelope” bid opening procedure is permitted for procurement of goods or works under specific conditions, notably if domestic law precludes use of one envelope.8

6 6 1 1 1 Foreign firms' eligibility cannot be conditioned on join ventures with local firms. 7 5 1 1 1 Qualification criteria shall be applied on a pass/fail basis. 5 5 1 1 1 1 Extension of the time period to prepare bids may only be allowed under exceptional circumstances. 5 5 1 1 1 1 1 An inflation clause is recommended for contracts over a year.9 7 5 1 1 1 1 Bidding documents are freely available.10 7 4 1


12

TZ TR BT BR BF MK MU PA PL GH CO LA VN MZ SL AL BA EG Reports Reviewe

d

Countries Reviewed

Field Visit

Field Visit/ UCS

UCS UCS UCS UCS UCS UCS UCS UCS Desk Sampl

e

Desk Sample

Desk Sample

Desk Sample

Desk Sample

Desk Sample

Other Other

35 26 2009 LA

2008 LA

2012 LA

2010 LA

2011 LA

2012 LA

2003 CPAR

excerpt 2009 LA

2007 LA

2007 LA

2009 LA

2011 LA

2010 LA

2002 CPAR

2010 LA

2012 CPAR

2012 LA

2002 CPAR

2002 CPAR

Bid security shall be in the form of a letter of credit or bank guarantee from a reputable bank.11 5 4 1 Joint venture partners must be jointly and severally liable. 6 3 1 There may be no restrictions on the means of delivery of bids.12 3 3 1 1 1 Pre-qualification should be used only for large works projects. 5 3 1 Award must be published. 4 3 1 1 Bidding documents and contract shall include provisions on sanctions for fraud and corruption. 3 3 1 1 1 No preference may be given to suppliers or contractors based on region or locality of registration, small size, ethnic ownership, etc. 4 2 For contracts subject to prior review under the Loan Agreement, scope/conditions may not be modified during implementation without Bank approval.

6 2 Bidders shall not be eliminated on the basis of minor deviations. 4 2 1 Contractors/suppliers must be prequalified for large or specialized contracts. 2 1 An invitation to prequalify must be advertised for each procurement involving large or complex potential contracts. 1 1 There must be no set limitations to the number of firms who can bid for a contract. 1 1 1 Requests for clarification of bidding documents must be made in writing, and response should be sent to all prospective bidders. 1 1 1 The experience requirement for works contracts shall be (i) at least one previous contract at 80% of cost of contract being procured and (b) an average annual turnover of 100% of contract being procured.

2 1 Bidders shall be given at least twenty-eight (28) days from the receipt of notification of award to submit performance securities.

1 1 1 Bids shall not be disclosed excepted as needed for evaluation without bidder's written authorization until the award of the contract. 1 1 1 Automatic rebidding may be required if too few bids are received.13 4 1 1 Source: World Bank; data compiled from CPARs and Loan Agreements.

1 With the exception of unskilled labor, if available locally. 2 Except for commodities/small goods contracts. 3 Primary boycotts, in which the government maintains a policy of not purchasing from a particular country, are permitted, however. 4 Registration may be required if (i) registration criteria, process and cost are reasonable/efficient and (ii) qualified foreign firms are not precluded from competing. Notwithstanding, the Bank stated that a registration requirement should be discouraged. 5 There is one exception: where the bid price is substantially above market or budget levels and then only if negotiations are carried out to try to reach a satisfactory contract through reduction in scope and/or reallocation of risk and responsibility which can be reflected in a reduction in Contract Price. (See Guidelines para 2.60)


13

6 Participation by parastatals is permitted providing they (i) are financially autonomous, (ii) operate under commercial law, and (iii) are independent from borrower and its purchasing/contracting authority. 7 Except where the bid prices exceed the available budget. The practices or rejection of bids outside a range or “bracket” of bid values is also known as "price bracketing." 8 Two envelope procedure is also permitted where there are adequate safeguards against retaining second envelope unopened are incorporated in the two envelope procedures and effective bid protest mechanisms are already in place for the due processing of bid complaints. (All technical envelopes are opened first and, after review, price envelopes of all or only qualified/responsive bids are opened in the second round.) 9 Recommended for works contracts of one year or more in duration when domestic inflation rate is high. 10 In some cases the Bank has specified there shall be no fee; in other countries fees have been permitted. 11 It may sometimes also be a certified check. In Macedonia, bid security was to "follow the generally acceptable practice used in the local market." 12 A method of delivery may be specified in the case where bidders have to submit physical sample, however. 13 Provided that (1) all responsible bidders are allowed to bid, (2) the process is efficient and (3) no serious delays result.

Stage II Equivalence Analysis

Table A.9. Stage II Equivalence Analysis—Average Scores

World Bank Guidelines Para. Policy Requirements Piloting Program Requirements Average score

(out of 3) Eligibility (paragraphs 1.6-1.8) Conditions for participation to be limited

to those that are essential to fulfil the contract requirements. Exceptions indicated in para 1.8.

Those provisions of the Guidelines would be made applicable through the Legal Agreement (LA).

1.7

Two-stage Bidding (paragraph 2.6) Turnkey contract (paragraph 2.22)

Applies to large and complex, or turnkey, contracts that potentially attract the participation of foreign bidders. Procedure may follow two stages: 1) non-priced technical proposals; 2) issuance of BDs to bidders, and submission of technical priced proposals.

Bank review of complex, high value, and non-standardized bidding processes as part of its assessment of the country systems. Possibility to use Bank SBDs or harmonized MPDs for specific bids / contracts to be identified in the Procurement Plan for the pilot pro-ject. Modifications as may be needed by the country institutional and legal framework to allow for the country policies and procedures to apply.

2.4

Notification and Advertisement (paragraphs 2.7-2.8)

General Procurement Notice and Specific Procurement Notices for ICB contracts to be published on UN Development Business online and Dg market.

The executing agency of each pilot project will be required to publish in UN Development Business, and on an electronic free access portal, all contracts identified as likely to attract foreign competition. Such contracts to be flagged in the Procurement Plan. Publication of all mandatory documents at the pilot project level on a fully functional and freely accessible website of the executing agency.

2.4

Examination, evaluation and comparison of bids; Post-qualification (paragraphs 2.48-2.54 and 2.58)

Bids to be evaluated and awarded on the basis of the criteria stated in the bidding documents and quantified using cost to the maximum extent. Post-qualification applies when there is no pre-qualification.

To be part of the bidding documents accepted for use under pilot projects when international competition is anticipated (see checklist below).

2.5

Alternative Conditions of Contract (paragraphs 2.37 and 2.38)

Contract documents must clearly define the procedure to submit alternative bids, and the scope of work, rights and obligations of the parties, and provide for fair and clear allocation of risk.

General Conditions of Contracts (GCC) for contracts covering goods, works and services to be consistent with international requirements when international competition is sought. To be part of the bidding documents accepted for use under pilot projects when international competition is anticipated (see checklist below).

2.6

Domestic preference for goods (paragraphs 2.55-2.56)

Defines the Bank's procedures for use of domestic preference.

Use of discriminatory domestic preferences (e.g. by nationality) will not be allowed in the bidding documents accepted for use under pilot projects when international

2.6


15

competition is anticipated (see checklist below).

Bid validity, Award of Contract Award to the lowest evaluated bidder which meets the requirements of the bidding documents including technical capacity and financial resources.

Assessment of legal framework to ensure that the national law covers bid validity, contract award, and rejection of bids requirements, and evaluation of compliance at the pilot project executing agency level, and monitoring during pilot project implementation.

2.6

Bidding Documents (paragraphs 2.11-2.12, and 2.16-2.18)

Bidding documents must provide information sufficient for a prospective bidder to prepare a bid. Use of Bank standard bidding documents is mandatory for ICB - this requirement will have to be modified for the pilots.

Existence of national sample bidding documents not required, but mandatory use of standard bidding documents acceptable to the Bank when international competition is sought under pilot projects. Specific provisions to be validated vis-à-vis the checklist below, and to be stated in the LA.

2.7

Publication (paragraphs 2.57, 2.59-2.60, and 2.61-2.64)

Publication of awards is required within two weeks. Rejection of all bids to be justified.

The executing agency for each pilot project would be required to have (a) a functioning electronic system for retaining information on its procurement processes and managing invoices and certificates payments; and (b) a fully functional website that is freely accessible to all stakeholders, including bidders and civil society, for posting information regarding its organization, procurement regulations, SBDs, procurement opportunities, information related to the award of contracts, and procurement plans.

2.7

Pricing (paragraphs 2.21 and 2.23)

Bid must be invited on the basis of specified INCOTERMS

Mandatory use of INCOTERMS for comparison purposes, or fully equivalent provisions.

2.7

Prequalification(paragraphs 2.9-2.10)

Use of prequalification for large complex requirements limited to capability and resources to perform the particular contract with regard to experience, past performance, personnel, facilities and financial position.

The Procurement Plan will be used to identify bidding processes that would benefit from prequalification. Advertisement as per requirements above. Prequalification requirements to be assessed by comparison with the harmonized prequalification MPD.

2.8

Joint ventures(paragraph 1.10) Mandatory association between firms is not acceptable.

Those provisions of the Guidelines would be made applicable through the LA.

2.8

Validity of bids and bid security (paragraphs 2.13-2.14)

Bidding documents must state bid validity period. Bid security not mandatory

Review of clauses in the bidding documents accepted for use under pilot projects when international competition is anticipated.

2.8

Language (paragraph 2.15) English, French, or Spanish, in addition to the national language of the country.

National language can be used provided that it is used internationally and that a translation in English, French or Spanish is made available to foreign bidders for contracts that have been identified for international bidding in the Procurement Plan. Advertisement and bidding

2.8


16

documents should be issued in an international language (English, French or Spanish) in addition to the national language, and contracts with foreign entities should be signed in the international language the bidders used for their bids.

Price Adjustment (paragraphs 2.24-2.25)

Bidding documents must specify if price adjustment applies and price adjustment must be based on a prescribed formula.

Requirement for price adjustment as a contractual term to be included in the bidding documents accepted for use under pilot projects when international competition is anticipated.

2.8

Currency and Payments (paragraphs 2.28-2.36)

Bids documents must state currency provisions and the conversion mechanism. Payment provisions must also be stated in the bidding documents.

National currency acceptable provided it is freely and fully convertible and country does not apply restriction or control on foreign exchange. Contract terms to allow payment in foreign currency directly and/or as percentage of contract price at predefined exchange rates (in no more than 3 currencies) expressed in the bidding documents accepted for use under pilot projects when international competition is anticipated.

2.8

Performance Security, Liquidated Damages Force Majeure (paragraphs 2.39/2.40, 2.41 and 2.42)

Format and provisions to be included in the bidding documents.

(to be assessed as part of the assessment of the country legal and institutional framework).

2.8

Applicable law and settlement of disputes (paragraph 2.43)

The conditions of contract shall provide for settlement of disputes and state the applicable law.

National law to apply. 2.8

Bid opening procedures (paragraph 2.45)

Bids to be opened in public at the time and place designated in the bidding documents. Rejection of late bids.

Assessment of legal framework to ensure that the national law covers this requirement, and evaluation of compliance at the pilot project executing agency level, and monitoring during pilot project implementation.

2.8

Rejection of bids 2.8 Time for bid preparation (paragraph 2.44)

Time allowed for the preparation and submission of bids shall be determined on the basis of the magnitude and complexity of the requirement.

Requirement to be stipulated for each executing agency implementing pilot projects. To be closely monitored as one key compliance indicator (see Stage III).

2.9

Confidentiality (paragraph 2.47)

After bid opening, evaluation and recommendations and other information shall not be disclosed until after publication of the award.

National law to apply. 2.9

Source: IEG analysis of UCS pilot data. Note: On a scale of 0–3.

17

Applying the Concept of Value for Money

EXAMPLES FROM SIX PUBLIC PROCUREMENT JURISDICTIONS

The United Kingdom, the European Union, and Australia have taken a long-term and systematic approach to integrate value for money concepts into their public procurement practices.1 Additionally, the Asia Pacific Economic Cooperation Forum (APEC) has endorsed value for money concepts since 1999. The International Atomic Energy Agency is one United Nations agency that has adopted value for money principles.

The United Kingdom

Guidance toward the approach taken by the United Kingdom has been provided for the most part by the U.K. Office of Government Commerce and the National Audit Office.2 Their documents emphasize the following:

Focusing on cost reduction, notably both the transaction overhead costs of conducting the procurement, and the cost of the goods, works, or services acquired. Getting better prices suggests negotiating better deals (price/quality), and when possible aggregating demand to capture scale economies.

Improving management of the project, the contract, and the acquired assets.

Balancing competition with risk. Competition is encouraged to obtain value, but risks need to be taken into account and where competition is weak benchmarking and theoretical cost models are suggested to verify that the purchasing agent is obtaining favorable offers.

Taking the whole length of the contract into account (including decommissioning).

Accounting for wider societal and environmental costs of the contract.

Developing bid evaluation criteria that are relevant to the project—published by the purchaser and widely available.

Recommending the use, by purchasing agents, of electronic commerce technology and new procurement practices.

The U.K. Department for International Development (DfID) has also adopted a value for money approach (DfID 2011). It concurs with much of the above, emphasizing the importance of having a sound procurement strategy, investing up front in well-defined bid criteria and good terms of reference, and negotiating favorable contract terms. It adds that purchasing agents should develop relationships with suppliers. This last point is a feature of discussions in the United Kingdom over the need for long-term collaboration with suppliers, to save costs and

18

ensure continuity of supply and consistency of quality (albeit sometimes at the expense of small and medium-sized enterprises (SMEs) and of competition more generally).

The United States—“Best Value” Procurement

Traditionally, best value has been used in the United States to describe the trade-off process, and since 1997 the term has come to encompass a variety of techniques that can be used in conducting competitive procurement.3 Today, the Federal Acquisition Regulation describes the best value continuum as any one or a combination of selection approaches that are used to obtain best value or value for money. There are two types of procurement techniques for obtaining best value:

The tradeoff process is a competitive negotiation process where the purchaser conducts a tradeoff analysis of both price and non-price factors, and awards the contract to the bidder proposing the combination of factors that offers the best overall value. The contract may be award to other than the lowest priced, or the highest technically rated bid.

The lowest price technically acceptable process describes a distinct variation on the competitive negotiation process, where the non-price factors are stated criteria of bid responsiveness. The contract is awarded to the bidder offering the technically acceptable proposal with the lowest price, with “technically acceptable” defined as complying with the technical criteria, including any non-price factors.

The lowest price technically acceptable process is more closely aligned with standard bidding techniques, where value criteria are stated in advance, as is typical of Bank procedures. Best value is defined when planning the procurement—the non-price factors are evaluated for conformance with the technical criteria expressed in the solicitation, typically without gradations of scores for higher levels of achievement. In contrast, under the tradeoff process, best value is decided during evaluation, through the trade-off analysis of the different bids.

Overall, best value procedures differ fundamentally from traditional sealed bidding procedures in which bidders offer the same or similar service or product in response to detailed solicitations, and bids differ only on price. In best value procurement, bids differ in elements beyond price. The purchaser may issue an award to a higher priced but technically superior offer or to a lower priced but less technically qualified proposal, depending on its view of which proposal presents the best overall value. Best value procurement has its critics, whose primary concern is the perception of fairness and the role of a procurement system in encouraging inclusiveness and transparency in government contracting. Concerns center on three issues:

19

The extent and consequence of discretion in ascribing and evaluating best value criteria, and in communicating with competitors in the conduct of negotiated procurement.

Past performance is arguably the most significant noncost/nontechnical criteria in American best value procurement. Concerns are that it prejudices unjustly firms that do not have a past history of government contracting and can ultimately become little more than a tool for incumbency.

There is a general concern that non-stated evaluative criteria appear to be subjective, and may preclude open and fair competition. The Federal Acquisition Regulation note that the procuring agency can obtain best value by various means, but explicitly provide only a few alternate considerations to price. If a requirement is not well defined and there is a performance risk to consider, the agency may properly rely more heavily on technical or past performance considerations. However, there are open questions as to whether the procuring agency might adopt evaluative factors for administrative ease.

The flexibilities and innovations of best value procurement continue to attract advocates and detractors alike. Debate continues over the loosening of regulations regarding communication, the incorporation of less-traditional considerations, and the devolution of discretion entailed. The choice to place best value criteria above traditional price/responsiveness metrics requires a developed and supporting legal framework related to ethics, fraud and corruption, influence peddling, and similar issues. Where problems arise, mechanisms must be in place to address them.

The European Union

Value for money principles are incorporated in the European Union’s Most Economically Advantageous Tender (MEAT) system, which enables the contracting authority to use criteria that reflect qualitative, technical and sustainable aspects of a tender, in addition to price, when reaching an award decision.4

Application of MEAT is a choice of the procuring agency—according to the directive, public contracts in the European Union must be awarded either solely on the basis of the lowest price, or under MEAT criteria.5 But where MEAT is used, it may involve the use of various criteria such as quality, technical merit, aesthetic and functional characteristics, environmental characteristics, running costs, cost-effectiveness, after-sales service and technical assistance, delivery date and delivery period, in addition to price. Additional features of MEAT include:

The contract notice must indicate whether MEAT criteria will be used to award the contract and what the criteria are.

20

The bid assessment and award criteria should not confer unrestricted freedom of choice on the contracting authority and should be made public.

Bidding and contract award criteria should ensure effective competition.

Whereas MEAT procedures offer procuring agents considerable latitude to define requirements, the directive maintains principles of open and competitive tendering, and fairness through clear specifications and objective assessment. The directive states:

….In order to guarantee equal treatment [of bidders], the criteria for the award of the contract should enable tenders to be compared and assessed objectively. If these conditions are fulfilled, economic and qualitative criteria for the award of the contract, such as meeting environmental requirements, may enable the contracting authority to meet the needs of the public concerned, as expressed in the specifications of the contract. Under the same conditions, a contracting authority may use criteria aiming to meet social requirements, in response in particular to the needs—defined in the specifications of the contract—of particularly disadvantaged groups of people to which those receiving/using the works, supplies or services which are the object of the contract belong.

As of 2012, thresholds for application of the European Union directive for central government and subcentral contracting authorities were €5,000,000 for Works (including concession contracts); €130,000 or €200,000 for services, depending on the type of service; and €130,000 (€200,000, for certain defense contracts) for supply.

Australia

Australia has made value for money the core principle underpinning Australian government procurement.6 Its procurement guidelines elaborate value for money concepts as:

• Life-cycle analysis of all relevant (direct and indirect) costs and benefits on a fair and common basis.

• Nondiscriminatory and competitive procurement processes based on bidding requirements and evaluation criteria that are logical, clear, comprehensive, and relevant.

• Efficient, effective, and ethical use of resources.

• Accountable and transparent decision making.

The guidelines clearly state that cost is not the only determining factor in assessing value for money. In addition to life-cycle analysis, value for money assessment could include fitness for purpose, supplier performance history, and relative risk of competing proposals.

21

The Australian guidelines specifically address business development of SMEs through public procurement, saying that that government is committed to sourcing at least 10 percent of procurement to Australian and New Zealand firms with fewer than 200 employees. This is accomplished, first, by ensuring fair treatment (nondiscrimination) of SMEs, and second, by considering in the procurement process the value for money benefits of doing business with SMEs. This falls short of requiring an explicit preference for SMEs, but implies that requirements may be tailored to fit SME capabilities, and that evaluations may consider benefits of awarding contracts to SMEs.

Asia-Pacific Economic Cooperation Forum

In 1999, APEC’s Government Procurement Experts Group identified the principles of public procurement as: “transparency, value for money, open and effective competition, fair dealing, accountability and due process, and non-discrimination.”7 Value for money is integral to the APEC procurement principles, which say:

…The test of the best available value for money is a comparison of relevant benefits and costs on a whole of life basis…The lowest priced compliant offer does not necessarily represent best value for money.

Other aspects of APEC’s value for money approach include:

• The procurement function can deliver value for money, in taxpayer terms, through improvements in procurement processes and management.

• Objective and functional requirements, which focus on what is to be achieved rather than how it is to be done, can encourage innovative solutions that may improve the value for money outcome.

• Costs and benefits are to be assessed on a life-cycle basis, possibly including sensitivity analysis and calculations of discounted cash flows.

• Besides price and fitness-for-purpose, other factors that may be taken into account include performance, quality, reliability, delivery, inventory costs, running costs, warranties and after-sale support, and disposal.

• Competition is encouraged. Procurement methods should be chosen based on the best value for money outcome, especially in situations of limited competition.

• Purchasing agents should be knowledgeable about the market in which they operate, assessing risks and avoiding unnecessary costs.

• It is important to establish competence, viability, and capability of prospective suppliers.

22

• Where allowed, negotiation should be used to improve value for money.

With respect to value for money, APEC covers many of the same areas as the United Kingdom and European Union. APEC does not make specific reference to environmental and social factors, but these are inferred as justifiable considerations. Its principles address ideas of negotiation, and alternative methods to address non-competitive markets, but are vague on details and on how the principles should be operationalized.

The International Atomic Energy Agency

The International Atomic Energy Agency (IAEA), a United Nations agency, offers a different perspective from the above organizations. While the approaches described above, like those of the Bank, are meant to provide guidance to others, IAEA principles and procedures are developed to guide the Agency in its own procurement activities. Accordingly, it is able to prescribe a system that fits with its internal technical and management capacity.

Value for money is central to IAEA principles and processes, as stated in its regulations (IAEA 2011). The introduction of these concepts is the result of a concerted effort to change the Agency’s procurement from a legalistic or rules based approach, to an outcomes or strategic management approach (Tonkin 2012). IAEA emphasizes several areas where value for money can be achieved, through:

• Integration of procurement strategies and planning with program delivery.

• Identification of risks (market risks, or continuity of supply risks, for example) and development of mitigation strategies within the procurement strategy and plans.

• Exploiting innovations that improve program delivery, through strengthened agency expertise, and by tapping supplier expertise.

• Optimizing cost over the life of the acquisition, trading off quality and cost factors where appropriate.

• Managing implementation to ensure that what was contracted is delivered, and to exploit opportunities to enhance value during implementation, especially in longer-term service contracts (by taking advantage of new technologies, for example).

An interesting feature of IAEA’s practices is to separate high value, complex procurement from low-value straightforward purchasing. The former is assigned to procurement specialist teams, whereas the latter is considered low risk and is assigned to contracting officers. This lowers the administrative overhead cost of

23

managing the procurement process. The IAEA also provides more latitude to procuring agents than other organizations. Although, like others, it sees open and competitive tendering as a value for money principle, it provides a range of selection methods other than lowest price (or lowest life cycle cost). These include best technical proposal (similar to the Bank’s fixed budget selection method for consulting services), and competitive negotiation options (similar to UNCITRAL procedure).

BROADER CONCEPTS OF VALUE FOR MONEY

Non-Price Factors and Preferences

The Australian provisions to use public procurement to strategically develop its SME sector are one example. The Independent Evaluation Group (IEG) found support for preferences in its 11-country survey, and notes that it is a common feature of many national procurement systems. Apart from the Bank’s domestic preference policy, seen by IEG as ineffective, the Bank does not allow preferences as it contradicts the principles of fairness and competition. To the extent to which the Bank wishes to retain the principle and practice of giving domestic preferences, discussed earlier in the present chapter, IEG recommends that the Bank explore other mechanisms to achieve this, in view of its present limited framework.

Investing in procurement planning and developing clear and appropriate evaluation criteria

The importance of strong front-end work is emphasized by the United Kingdom and IAEA. The objectives appear to be twofold: (i) to assure transparency, and (ii) to best align the stated requirements of the procurement with the true needs of the project (fit for purpose). The Bank Guidelines mandate procurement planning, seen to be an integral part of project design. IEG notes that although the Bank has traditionally supported up-front procurement planning, such support has been found to be below optimal, especially in complex projects. Management proposes to make stronger use of procurement planning as a tool in the future.8

Reducing Cost through Competition and Negotiation

Competition is generally accepted, within value for money principles, as an effective way to lower cost of acquisition. However, value for money also envisages instances of poor competition, and suggests that good negotiation can contribute to better outcomes in terms of cost and quality. 9 Bank policy and practice is very much oriented toward competition. Negotiation to improve outcomes, post contract award, is limited to acquiring professional services, following consultant recruitment procedures. Yet it is unlikely that the Bank could recommend

24

negotiation in all but restricted circumstances, as it would run counter to other principles of transparency and fairness.

The Bank offers only limited solutions to situations of poor competition and other aspects of addressing market risk. Direct negotiation is an option, but it is an exception and little guidance is offered on how best to conduct the process or to determine if good value is achieved.10 Limited competition, such as is found in information and communications technology (ICT) procurement, or sole sourcing (to incumbent public-private partnership [PPP] concessions, for example) are also not well accommodated in Bank practice. It is suggested that the Bank explore greater use of less-competitive methods, such as competitive negotiation as provided for by UNCITRAL and European Union, and as reported here is practiced in IAEA.11

Strengthening Relationships with Suppliers

Building “appropriate” relationships with suppliers is a feature of the U.K. value for money principles, although admitting that it limits competition and could be unfair to some classes of firms. In a similar vein, IAEA principles recommend that purchasing agents understand supplier capabilities and practices, and use prequalification and staged bidding to better assess bidder capability. Bank policies and practice encourage prequalification and multistage bidding, and would not discourage procuring agents from knowing their markets. However, Bank practice suggests assurance of an “arms-length” relationship between purchaser and bidder. Many Bank procedures are written to ensure limited interaction (detailed procedures of communications during bidding, for example), partly to avoid the possibility of collusion and influence. Maintaining its fraud and corruption mitigation agenda could be difficult if the Bank were to go in this direction.

There is a concern that the traditional focus of public procurement on regulations and procedures discourages the exploitation of value-adding procurement methods thereby reducing the potential for innovation. Proposed revisions to the European Union Procurement Directive recommends introduction of the “innovative partnership” procedure (see the next session). This new method of procurement would allow procuring agents to enter into partnership arrangements with suppliers to develop, and subsequently supply new and innovative products, works and services according to specified requirements that are performance oriented. The developmental benefits of such supplier relationships are obvious, and mechanisms to support such arrangements within the Bank’s principles of fair and open competition, are recommended.

25

Reducing Transaction Costs

The United Kingdom and IAEA both note that reducing the administrative overhead cost of procurement offers value for money benefits. IAEA has established lower cost processes for low-value, low-risk procurement. The Bank’s shopping procedure, might be seen as a similar approach. Besides, as noted in previous chapters, Bank practices in terms of more “proportionality” of resource use, relative to the objective of procurement, could be further improved through a series of measures; improved guidelines and processes for complex procurement (Chapter 2); more tolerance of risk, and the use of risk-based rather than value thresholds, frugal use of higher level clearance (Chapter 3); and more attention to elapsed time in contract processing (Chapter 4). All these measures would strengthen the application of value for money principles.

Supporting Contract Implementation

IAEA and the United Kingdom cite attention to managing contracts, as an important value for money factor—ensuring that what was contracted is delivered, is as important as the selection process. Furthermore, IAEA notes opportunities to adjust contracts during implementation, to reap gains of new technology for example, especially important for long-term service contracts. Regarding the capturing of value for money benefits through adjustment to contracts during implementation, this is to some extent covered in the Bank’s works contracts (closely modeled on the International Federation of Consulting Engineers), which give authority to the engineer to improve the implementation of works through variations. IEG has also noted the need to strengthen long-term service contracts, as these especially would almost certainly require ongoing renegotiation to obtain optimum outputs.

IEG discusses the broader issue of contract implementation earlier in this chapter, and has pointed towards client requests for support. Greater attention in Bank practices to oversee and assist contract implementation is needed, though balance will have to be achieved between the role of different players within the Bank (especially task team leaders and procurement staff), as well as outside the Bank, for example, through national audit offices, country based citizens’ groups and the like.

To summarize, the principle of value for money in procurement is being increasingly incorporated into the procurement approach of a number of public jurisdictions, with varying degrees of operational instruction and varying degrees of emphasis in terms of the aspects of value for money that they each seek to emphasize. Many aspects of these principles are already implicit in Bank procurement, and some are frequently used. Stronger direction could be given to staff and borrowers to fortify the adoption of such practices. A number of the

26

efficiency increasing (and hence transaction cost reducing) measures proposed in the present report would expand value for money, as would better management of risk, and the embracing of new procurement practices in frontier areas of Bank business. In such areas, more attention to specific procurement modalities would be required to achieve value for money. However, areas remain, where for reasons of open access, or transparency, the Bank may prefer to maintain its present systems and practices.

Bank requirements do not accommodate the inclusion of social objectives through procurement (using procurement to create employment, perhaps targeting youth or minorities, for example). The difficulty for the Bank is to balance objectives of domestic laws and policies that target social and domestic industrial strategies, with its obligations to ensure fair treatment of international bidders.

27

Proposal to Update the European Union Public Procurement Directive

A proposal to update the current European Union procurement directive (2004/18/EC) was issued on 20 December 2011; the “Proposal for a Directive of the European Parliament and of the Council on Public Procurement.”12 The extent to which the proposal will be implemented, or the timing is not known, although there is some expectation that reforms maybe introduced later this year.13

CAUTIOUS APPROACH

The proposal is cautious in its ambitions. It says that it endeavors to retain continuity with the current directive, and not to depart from concepts that have been developed over the years, have evolved through case law, and are well known to practitioners. This is in line with the Bank approach, which will retain the Guidelines and keep much of the existing system intact.

The proposal notes obligations of member states to comply with World Trade Organization (WTO) government procurement agreement requirements, although done so with respect to WTO’s proposed lighter regime for contracting authorities below the central government level. Thus it offers simplified procedures, yet invokes alignment with international best practice. This is something for the Bank to consider in light of its move towards country systems—use of country systems can simplify things for borrowers, but continued compliance with international norms is important.

TOOL BOX

A “toolbox” approach is proposed, but in fact does not change much. The options of open and restrictive tendering are retained, as are the negotiation procedures (competitive negotiation and competitive dialog—although with simplified procedures and broadened application beyond “complex” procurement). The one new procedure that is introduced is the “innovative partnership” procedure (e-procurement procedures are also emphasized as noted below). This allows procuring agents to enter into partnership arrangements with suppliers to develop, and subsequently supply, new and innovative products, works, and services according to specified requirements that are performance oriented.

The toolbox approach is similar to the direction the Bank is suggesting – to transform guidelines and standard bidding documents from mandatory, to optional guidance documents, for example. The Bank does not use the negotiation procedures of the European Union (and others), although IEG recommends this. If adopted, the new “innovative partnership” is perhaps something worth exploring

28

by the Bank, especially for borrowers who would follow European Union directives. The objective is to use public procurement strategically to improve innovation (and competitiveness). If properly managed, for example by using open and competitive bidding, it offers developmental benefits and could be done without violating Bank principles.

EMPHASIS ON E-PROCUREMENT

Several measures are proposed to promote and mandate e-procurement:

• Mandatory electronic notices

• Mandatory electronic documents

• Switch to electronic communications, including electronic bid submissions, within two years

• Acceptance of dynamic purchasing systems, electronic auctions, electronic catalogues.

This goes beyond the Bank’s current position on e-procurement, or what the Bank proposes in terms of promoting greater use. The Bank allows all of these features, but does not mandate them--given the Bank’s diverse client base, it is unlikely that it could (with some exceptions, such as the current mandatory publication of notices on the United Nations Development Business website, for example).

MODERNIZATION/SIMPLIFICATION

In addition to e-procurement, further suggestions to modernize, streamline, simplify, include:

• Shortened time limits for bidding

• Exclusion of firms from bidding for nonperformance

• Provisions to modify contracts during implementation.

These are relatively minor items, but the Bank might consider similar measures. The Bank’s mandatory bidding periods were criticized by clients in IEG’s country case studies, especially for international competitive bidding (ICB). With modern communications and more of the bidding process going on-line, shortened periods could be considered where it can be demonstrated not to compromise competition. Modification of contracts, especially long-term service contracts, has been noted earlier in this chapter.

29

VALUE FOR MONEY

As reported earlier in this appendix, value for money is a feature of the existing European Union directive. It is proposed to expand this in several respects:

• More explicit instruction to apply lif- cycle costing, noting that external costs are to be considered (such as environmental costs) where they can be monetized and verified. A common European Union methodology for life-cycle costing is to be used, when developed.

• Factors directly linked to production processes may be included in technical specifications and technical award criteria.

• Labels may be used in technical specification related to environmental factors (eco label) and other social factors (child labor, for example).

• Firms may be sanctioned for violating social, labor and environmental laws.

Value for money is a key element of the Bank’s proposed procurement reform, and as commented, the Bank’s system already incorporates many of the measures noted above (life-cycle costing, environmental specifications, and so forth). But the Bank should also consider developing methodologies around these approaches.

SUPPORT TO SMES

To address its objective of supporting local industry, the Bank could also consider the European Union suggestions to support SMEs. Several suggestions are made to promote SMEs: removing barriers to SMEs (reducing turn-over requirements, for example), simplifying bidding requirements (for example, allowing self-declarations as prima-facie evidence), “packaging” procurement to make packages more attractive to SMEs, and direct payment to subcontractors.

KNOWLEDGE CENTERS

The European Union proposal observes that there is a lack of capacity in many contracting authorities, and that this limits effective procurement. The proposal obliges member states to provide support structures offering legal and economic advice, guidance, training and assistance in preparing and conducting procurement procedures. The Bank has similarly identified borrower capacity building as important. It might similarly consider the establishment of a knowledge center that borrowers can tap into on an as needs basis.

31

Appendix B. CDD Procurement Summary of Previous Reports

Commensurate with the increasing importance of community-driven development (CDD) projects in the Bank’s portfolio, procurement under CDD projects has been the subject of many reviews by the Networks and by IEG, and they yield mixed evidence. An early review, with positive findings, was undertaken by the central Social Protection Team unit. 14 It found that contracting by community groups through local shopping was “more efficient and transparent” and got lower prices than central government bidding. The review also noted several “good practice” procedures, including project designs that reflect skills and needs of communities, use of very simple contracting procedures, building-in incentives for efficient procurement, provision of active support to communities on prices and quality to help them in negotiating with suppliers, and ensuring transparency of procurement through proactive information sharing and disclosures.15

A later field-based review by the South Asia Region of 84 CDD subprojects in India also concluded, positively, that better value for money was achieved in works implemented by communities without involving contractors, with savings in costs and time in most of the analyzed cases. 16 More interestingly from the procurement perspective, the review found that “compared to the conventional procurement cycle, the subprojects rated very highly in terms of procurement planning and scheduling, developing specifications, awarding decisions, and disclosure and contract management while scoring poorly in terms of market search for bidders, tendering, tender opening and evaluation.” The review concluded that success could be ascribed to the practice of “relationship-based procurement” wherein the communities established a mutually beneficial and accountable commercial relationship between the purchaser and the supplier. The report concluded that there was scope for further adaptation of the CDD procurement processes with focus on delivery of outcomes and results rather than on oversight and control of contracting for inputs.

Yet other reviews by IEG, the Internal Audit Vice Presidency (IAD), and the Integrity Vice Presidency (INT) have flagged persistent concerns about the adequacy of fiduciary oversight of CDD projects. A comprehensive review of the CDD portfolio by IEG in 2005 noted that almost a quarter of surveyed staff expressed concerns about their ability to adequately monitor compliance with agreed fiduciary

APPENDIX B CDD PROCUREMENT

32

processes in CDD projects (IEG 2005). In a similar vein, a 2008 IAD audit reported that the Bank did not conduct required procurement post-reviews for five out of 11 projects that required such reviews--impeding the Bank’s ability to detect contracts that did not comply with the agreed procurement procedures. And an INT investigation of two CDD projects in Kenya in 2011 was significantly critical in its findings: it pointed towards large scale misappropriations and leakages caused by failure to comply with agreed procedures during implementation (Al-Azar 2011; World Bank 2011a).

Responding to INT’s concerns, Operations Policy and Country Services (OPCS) undertook a major review in 2012 of governance issues related to CDD projects (World Bank 2011a). Its review concluded that a well-designed CDD project can strengthen governance at the local level, since it incorporates the three key principles of good governance: transparency, accountability and participation. However, complementary and interdependent pieces have to come together to make CDD projects work as intended. Fraud and corruption in the procurement process (during contract implementation), for example, short-changing of project quality because of collusive practices, were seen as important risks. The review recommended special attention to risk assessment, especially the risks of “political capture” during project design. It also argued for field-based, intensive, collaborative supervision by the Bank, front-loaded during the first two years. The review concluded that with adequate care in design and supervision, CDD projects can be successful even in cases where governance is poor and corruption is rampant.

33

Portfolio Analysis

CDD broadly defined is an approach that gives control over planning decisions and investment resources to community groups and local governments. CDD programs operate on the principles of local empowerment, participatory governance, demand-responsiveness, administrative autonomy, greater downward accountability, and enhanced local capacity. The CDD Anchor in the Social Development Department has been tracking the amount of Bank (IBRD/IDA) lending going toward CDD approaches since 2000 using the following CDD typology developed by a cross-regional/cross-sectoral group of CDD practitioners.

This is the universe of projects for the evaluation period. This review uses the Social Development group definition of CDD and used their database of CDD projects. A few CDDs (3 percent of total) are embedded in development policy operations. Since the review will assess the functioning of World Bank procurement systems in its operations, we will focus on Investment Lending, hence on CDDs delivered in that form. The universe of CDDs is then 692 projects. It should be noted that projects that have some CDD components are also included in the CDD database, per the Social Development group definition and classification.

The social development database includes 732 projects classified as CDDs, 711 (97 percent) are IBRD/IDA financed and the rest are financed by Global Environment Facility and other trust funds that do not consistently record implementation status report (ISR) ratings. Only IBRD/IDA financed operations are included in this analysis. As a first attempt to assess the effects of World Bank procurement systems on the development effectiveness of CDD projects, we will use two sets of data stemming from ISRs and from IEG project evaluations (Implementation Completion Report Reviews—ICRRs).

From ISRs we will extract procurement ratings and outcome ratings. For simplicity we will use ratings from the last ISR, although ratings from all ISRs are also available and could be used to track changes in projects' procurement performance. It should be noted that ISRs are only done for main projects and not for subsequent additional financing projects. For this reason, the number of CDD projects with ISR ratings is smaller, as seen in Table B.1. In tables where ISR ratings are shown, CDD total will be 549 projects because of the existence of a few unrated projects.

ICRRs do not have procurement ratings, but outcome ratings and risk to development outcome ratings can be extracted, either to confirm ISR outcome assessment and its sustainability. Only a subset of the CDDs reviews are closed (47 percent, see Table B.2) and even fewer have been reviewed by IEG as of today (21


34

percent). For this reason the tables with IEG ratings will show a CDD total of 143 projects. RDO totals are lower because this rating was introduced in the middle of our evaluation period (around FY08).

Table B.1. World Bank Project Approvals (all projects—CDD and non-CDD), FY02-11 IBRD/IDA Trust funds

and others Total

IL Dev Pol Operation

IBRD/IDA total Main

project Additional financing

Total

CDD 552 140 692 19 711 21 732 Non-CDD 1,492 313 1,805 563 2,368 1,256 3,624 Total 2,044 453 2,497 582 3,079 1,277 4,356 Source: IEG analysis and World Bank data. Note: As of November 8, 2012.

Table B.2. World Bank Project Approvals (investment lending projects), FY02-11 CDD Non-

CDD Total

Main Project Additional financing

Total CDD

Active 265 58 323 1052 1,375 Closed 287 82 369 753 1,122 Closed o/w have IEG Evaluation

153 - 153 348 501

Total 552 140 692 1,805 2,497 Source: IEG analysis and World Bank data. Note: As of November 8, 2012. Includes investment lending projects financed by IBRD/IDA and excludes those financed by Trust Funds and others.


35

Table B.3. IEG Outcome Ratings and Risk to Development Outcome Ratings for Investment Lending Projects, by Fiscal Year

Outcome ratings Risk to development outcome ratings Approval FY MS+ S+ Total % Sat

(MS+) % Sat (S+)

Moderate or Lower

Total % Moderate or lower

CDD FY02 38 21 50 76 42 26 44 59 FY03 32 7 48 67 15 20 47 43 FY04 21 14 33 64 42 12 33 36 FY05 9 2 15 60 13 11 15 73 FY06 3 2 4 75 50 3 4 75 FY07 1 1 3 33 33 2 3 67 FY08-11 Total CDD 104 47 153 68 31 74 146 51

Non-CDD FY02 72 38 97 74 39 46 79 58 FY03 65 30 83 78 36 56 81 69 FY04 49 19 69 71 28 48 67 72 FY05 36 15 55 65 27 32 55 58 FY06 19 8 26 73 31 14 26 54 FY07 8 2 12 67 17 5 12 42 FY08 2 2 4 50 50 3 4 75 FY09 1 2 50 0 1 2 50 FY10-11 Total Non-CDD

252 114 348 72 33 205 326 63

CDD and Non-CDD FY02 110 59 147 75 40 72 123 59 FY03 97 37 131 74 28 76 128 59 FY04 70 33 102 69 32 60 100 60 FY05 45 17 70 64 24 43 70 61 FY06 22 10 30 73 33 17 30 57 FY07 9 3 15 60 20 7 15 47 FY08 2 2 4 50 50 3 4 75 FY09 1 0 2 50 0 1 2 50 FY10-11 Total 356 161 501 71 32 279 472 59 Source: IEG analysis and World Bank data. Note: As of November 8, 2012. Totals refer only to those projects with available or valid ratings.


36

Table B.4. ISR Procurement Ratings for Investment Lending Projects, by Fiscal Year Approval FY HS S MS MU U HU Total % Sat

(MS+) % Sat (S+)

CDD FY02 48 12 3 3 66 91 73 FY03 44 19 5 4 72 88 61 FY04 1 45 18 4 1 69 93 67 FY05 29 27 6 1 63 89 46 FY06 33 22 4 59 93 56 FY07 1 22 22 7 52 87 44 FY08 20 27 4 51 92 39 FY09 1 21 17 9 2 50 78 44 FY10 12 20 5 37 86 32 FY11 16 12 2 30 93 53 Total 3 290 196 49 11 0 549 89 53 Non-CDD FY02 1 80 35 4 4 1 125 93 65 FY03 3 75 34 8 3 1 124 90 63 FY04 2 88 31 16 4 141 86 64 FY05 1 80 67 6 4 1 159 93 51 FY06 1 93 62 10 2 168 93 56 FY07 81 49 10 5 1 146 89 55 FY08 74 63 16 4 157 87 47 FY09 61 61 8 3 133 92 46 FY10 87 69 11 2 169 92 51 FY11 2 117 53 13 3 188 91 63 Total 10 836 524 102 34 4 1510 91 56 Source: IEG analysis and World Bank data. Note: As of November 8, 2012. Totals refer only to those projects with available or valid ratings. Latest ISR ratings were used. Additional financing projects don't have ISR ratings. HS = highly satisfactory; S = satisfactory; MS = moderately satisfactory; MU = moderately unsatisfactory; U = unsatisfactory; HU = highly unsatisfactory.


37

Table B.5. ISR Procurement, ISR Outcome Ratings, and IEG Outcome Ratings ISR Procurement Ratings

IEG outcome ratings ISR outcome ratings MS+ MU- Total MS+ MU- Total

CDD MS+ 447 36 483 100 36 136 MU- 33 27 60 4 13 17 Total 480 63 543 104 49 153 % of Total MS+ 93% 7% 100% 74% 26% 100% MU- 55% 45% 100% 24% 76% 100% Total 88% 12% 100% 68% 32% 100% Non-CDD MS+ 1,193 145 1,338 240 71 311 MU- 53 84 137 10 24 34 Total 1,246 229 1,475 250 95 345 % of Total MS+ 89% 11% 100% 77% 23% 100% MU- 39% 61% 100% 29% 71% 100% Total 84% 16% 100% 72% 28% 100% CDD & Non-CDD MS+ 1,640 181 1,821 340 107 447 MU- 86 111 197 14 37 51 Total 1,726 292 2,018 354 144 498 % of Total MS+ 90% 10% 100% 76% 24% 100% MU- 44% 56% 100% 27% 73% 100% Total 86% 14% 100% 71% 29% 100% Source: IEG analysis and World Bank data. Note: As of November 8, 2012. Latest ISR ratings were used. Data only includes those projects with available or valid ISR and ICRR ratings. Includes only investment lending projects. ISR = Implementation Status Report. Ratings: MS+ = moderately satisfactory or better; MU– = moderately unsatisfactory or worse.


38

Table B.6. ISR Procurement and IEG Risk to Development Outcome Ratings Risk to DO ratings Procurement ratings

S+ MS+ MU- Total CDD Moderate or lower 55 70 4 74 Total 98 129 17 146 % Moderate or lower 56 54 24 51 Non-CDD Moderate or lower 136 191 14 205 Total 202 294 31 325 % Moderate or lower 67 65 45 63 CDD and Non-CDD Moderate or lower 191 261 18 279 Total 300 423 48 471 % Moderate or lower 64 62 38 59 Source: IEG analysis and World Bank data. Note: As of November 8, 2012. Data only includes those projects with available or valid ISR and ICRR ratings. Includes only investment lending projects. DO = development outcome; S+ = satisfactory or better; MS+ = moderately satisfactory or better; MU- = moderately unsatisfactory or worse.

Table B.7. CDD Projects: ISR Procurement and IEG Outcome Ratings by Region Region IEG outcome ratings ISR procurement ratings

MS+ MU- Total % MS+ MS+ MU- Total % MS+ AFR 37 19 56 66 177 23 200 89 EAP 5 2 7 71 61 5 66 92 ECA 22 8 30 73 63 6 69 91 LCR 18 9 27 67 79 13 92 86 MNA 8 4 12 67 29 5 34 85 SAR 14 7 21 67 80 8 88 91 Total 104 49 153 68 489 60 549 89 Source: IEG analysis and World Bank data. Note: As of November 8, 2012. ISR = Implementation Status Report. Regions: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LCR = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Asia.

Table B.8. CDD Projects: ISR Procurement and IEG Outcome Ratings by Sector Board Sector Board IEG outcome ratings ISR procurement ratings

MS+ MU- Total % MS+ MS+ MU- Total % MS+ Agriculture and Rural Development

30 12 42 71 165 16 181 91

Education 17 8 25 68 45 8 53 85 Health, Nutrition and Population

17 12 29 59 62 7 69 90

Social Development 13 4 17 76 35 2 37 95 Social Protection 12 7 19 63 48 6 54 89 Urban Development 4 4 100 41 3 44 93 Water 7 4 11 64 33 7 40 83 Other 15 5 20 75 60 11 71 85 Total 115 52 167 69 489 60 549 89 Note: As of November 8, 2012. ISR = Implementation Status Report. S+ = satisfactory or better; MS+ = moderately satisfactory or better; MU- = moderately unsatisfactory or worse.


39

Table B.9. CDD Projects: ISR Procurement and IEG Outcome Ratings by Loan Size Loan size ($ millions)

IEG outcome ratings ISR procurement ratings MS+ MU- Total % MS+ MS+ MU- Total % MS+

< 20 33 12 45 73 99 16 115 86 20≤ L <50 39 21 60 65 183 19 202 91 50≤ L <100 23 10 33 70 101 15 116 87 ≥100 20 9 29 69 106 10 116 91 Total 115 52 167 69 489 60 549 89 Source: IEG analysis and World Bank data. Note: As of November 8, 2012. ISR = Implementation Status Report. S+ = satisfactory or better; MS+ = moderately satisfactory or better; MU- = moderately unsatisfactory or worse.

Table B.10. CDD Projects: ISR Procurement and IEG Outcome Ratings by Fragile State Status and CPIA Rating Rating Fragile state status CPIA ratings

FCS Not FCS Total ≤ 3 > 3 Total IEG outcome ratings MS+ 22 93 115 13 100 113 MU- 9 43 52 6 45 51 Total 31 136 167 19 145 164 % MS+ 71 68 69 68 69 69 ISR procurement ratings MS+ 73 416 489 46 437 483 MU- 16 44 60 14 45 59 Total 89 460 549 60 482 542 % MS+ 82 90 89 77 91 89 Source: IEG analysis and World Bank data. Note: As of November 8, 2012. The harmonized list of fragile situations for FY13 was used. Overall CPIA ratings for 2010 were used. CPIA = Country Policy and Institutional Assessment; FCS = fragile and conflict-affected situation; MS+ = moderately satisfactory or better; MU- = moderately unsatisfactory or worse. .

Table B.11. CDD Projects: ISR Procurement and IEG Outcome Ratings by CDD Concentration CDD Concentration

IEG outcome ratings ISR procurement ratings MS+ Total % MS+ MS+ Total % MS+

≤ 25% 32 44 73 157 177 89 25% < R ≤ 50% 20 33 61 85 100 85 50% < R < 90% 39 54 72 193 215 90 ≥ 90% 7 12 58 53 57 93 Total 98 143 69 488 549 89 Source: Social Development group database of CDD projects. Note: As September 24, 2012. CDD concentration is measured as a share of total IBRD/IDA lending amount. CDD = community-driven development; ISR = Implementation Status Report; MS+ = moderately satisfactory or better; MU- = moderately unsatisfactory or worse.


40


41

Table B.12. CDD Projects: ISR Procurement and IEG Outcome Ratings by CDD component CDD component IEG Outcome Ratings ISR Procurement Ratings

MS+ Total % MS+ MS+ Total % MS+ Decision making by participatory elected local governments

28 45 62% 161 178 90%

Community Control and Management of Investment Funds

52 74 70% 196 224 88%

Community Control without direct Management of Invest. Funds

53 75 71% 240 268 90%

Enabling environment 53 86 62% 224 253 89% Total CDD Projects 104 153 68% 478 540 89% Source: Social Development group database of CDD projects. Note: As of September 24, 2012. The categorization is not mutually exclusive. Each CDD category shows the number of projects that had a non-zero amount assigned to that component. Total CDD amounts counts the number of projects that had an IEG outcome rating or an ISR procurement rating, respectively. CDD = community-driven development’ ISR = Implementation Status Report; MS+ = moderately satisfactory or better.

Table B.13. CDD Projects: Procurement and Development Outcome by Main Project Objective Main project objective Procurement ratings Outcome ratings

MU- Total % MU- MU- Total % MU- Service delivery 16 138 12 11 36 31 Empowerment 4 68 6 8 29 28 Improving local governance 6 43 14 4 16 25 Livelihood security prioritization 11 111 10 7 25 28 Total CDD projects 33 281 12 24 81 30 Source: IEG analysis and World Bank data. Note: As of September 24, 2012. Includes investment lending only. To identify the main project objective, only those projects that rated each area with 1 were considered. Procurement ratings are those of the last ISR. Development outcome ratings are those of IEG's ICRRs. CDD = community-driven development; MU- = moderately unsatisfactory or worse.

43

Project Reviews

Table B.14. Sample of Projects Reviewed Approval FY

Country Project ID Project name Amount ($ millions)

2002 Central African P073525 Multisectoral HIV/AIDS Project 17 2002 Ethiopia P050383 Food Security Project 85 2002 Jordan P076961 Horticultural Exports Promotion and Technology Transfer

Project 5

2002 Morocco P073531 Support for the Social Development Agency Project 5 2002 Senegal P074059 HIV/AIDS Prevention & Control Project 30 2002 Sierra Leone P073883 HIV/AIDS Response Project 15 2002 Tanzania P047762 Rural Water Supply and Sanitation Project 26 2003 Indonesia P059931 Water Resources & Irrigation Sector Management Program 70 2003 Kenya P078058 Arid Lands Resource Management Project Phase Two 60 2003 Rwanda P071374 Multi-Sectoral HIV/AIDS Project 31 2003 Tanzania P059073 Dar es Salaam Water Supply and Sanitation Project 62 2004 Angola P081558 3rd Social Action Fund (FAS III) 55 2004 Azerbaijan P076234 Rural Investment Project (AZRIP) 15 2004 Bangladesh P086791 Bangladesh - Reaching Out of School Children Project 51 2004 Brazil P080830 Maranhao Integrated Program: Rural Poverty Reduction

Project 30

2004 Honduras P083244 Nuestras Raices Program 15 2005 Brazil P076924 Amapa Sustainable Communities 5 2005 Congo,

Democrat P086874 Democratic Republic of Congo Emergency Social Action

Project 60

2005 Guinea-Bissau P083453 Coastal and Biodiversity Management Project 3 2005 Indonesia P078070 Support for Poor and Disadvantaged Areas Project 104 2005 Peru P082588 Agricultural Research and Extension APL Phase 2 25 2006 Indonesia P085375 Third Water Supply and Sanitation for Low Income

Communities Project 138

2007 Afghanistan P0102288 Afghanistan: Emergency National Solidarity Project II 120 2007 Kenya P074106 Western Kenya CDD and Flood Mitigation Project 86 2008 Bolivia P101298 Second Participatory Rural Investment 20 2008 Ethiopia P108932 Pastoral Community Development Project II 80 2008 Haiti P106699 Haiti - Urban Community Driven Development Project /

PRODEPUR 16

2009 Brazil P110614 Sergipe State Integrated Project: Rural Poverty 21 2009 Ethiopia P106855 General Education Quality Improvement Project - APL 1

(GEQIP) 50

2010 India P101650 Andhra Pradesh Rural Water Supply and Sanitation 150 2010 Indonesia P115052 Third National Program for Community Empowerment in

Rural Areas (PNPM-Rural) 785

2011 India P102329 Rajasthan Rural Livelihoods Project (RRLP) 163 2011 Mozambique P107598 MZ PROIRRI Sustainable Irrigation Development 70 2011 Nicaragua P121779 Nicaragua Social Protection 20 EXAMPLES OF REVIEWS UNDERTAKEN FOR EACH CDD PROJECT

(Available for all 34 projects from IEG)

Sample Project 1: Afghanistan - Emergency National Solidarity Project -

44

Extracts from Project Appraisal Document

Project Objective: “Building on the current National Solidarity Project, the proposed Grant will finance expansion to new areas to lay the foundations for strengthening of community level governance, and to support community-managed subprojects comprising reconstruction and development that improve access of rural communities to social and productive infrastructure and services.”

Highlighted Components: The proposed IDA grant will finance about 23 percent of the total projected cost of the National Solidarity Program in the next three years for the following components:

• Block grants for communities ($94.3 million) to carry out subprojects involving reconstruction and development activities through a facilitated participatory planning process.

• Community Facilitation and Subproject Preparation ($9.7 million) to support local communities through: (i) facilitation exercises to establish inclusive community institutions, and identify local development needs and priorities; and (ii) assistance for preparing subproject proposals.

• Community Development Council Capacity Development ($14.4 million) to assist Community Development Councils in carrying out subprojects.

Procurement Arrangements: A large majority of the works to be procured under this project would include community subprojects. These subprojects are implemented by the communities using block grant provisions made by the project. Each community will get a block grant of $200 per family with an upper ceiling of limit of $60,000 per community. Therefore, the estimated value of most of the community subprojects would be less than $60,000 per project. The community shall carry out procurement following paragraph 3.17 of Bank Guidelines.

The overall procurement risk of the National Solidarity Project is moderate. The project includes a large number of community level subprojects that are widely spread over the country and will be procured by the communities. IDA will be unable to carry out procurement reviews of a reasonable sample of subprojects as would be the case in a normal investment project. Regarding the mitigation plan, procurement administration is overseen by an international procurement specialist who is also building in house capacity at the Ministry of Rural Rehabilitation and Development (MRRD) level. IDA’S procurement team will continue to supervise the procurement administration closely and carry out post-procurement reviews periodically. The National Solidarity Project operational manual also provides step

45

by step procedures to be followed by communities for community procurement. The procurement procedures at the community level have been further simplified. The project will continue to provide training on community procurement to FP staff and Community Development Councils.

Nature of Procurement Problems – Extracted from ISR, ICRs, ICRRs, and So Forth

ISR 7: Anomalies in the MRRD Incremental Operating Cost Expenditure. The review of MRRD Incremental Operating Cost expenditure finds that civil servants resigned from government service and joined as consultants. This is against Article 15 of the Regulation of Personal Affairs of Civil Servants.

ICR

• Lack of familiarity by MRRD staff of Bank Procurement Guidelines and procedures contributed to delays.

• There was one major contract with the oversight consultant that was amended five times. The amendments resulted in (i) time extensions from an initial period of 12 months to 42 months, and (ii) price modifications from an initial contract price of $5.3 million to $32.5 million.

• Under the District School Window, 18 contracts were awarded to construct 8 to 16 classrooms for 58 schools in seven provinces. These contracts could not be completed during the project period and were extended to the second project. There were several contractual modifications to increase the quantities due to lack of diligence in the calculation of bills of quantities by MRRD’s engineering staff working on the District School Construction and Rehabilitation Window.

• Careful engineering planning combined with more effective liaison with MoE and other donors could have contributed to avoid such incidents.

IEG Comments

- Based on the project documents, it appears that the community implemented subprojects had a smooth implementation reflecting extensive efforts at capacity building and technical assistance for the communities. No indication from the project documents of wastage or misuse of resources used for community-managed procurement. This would suggest that the Bank guidelines bearing on the CDD procurement were not an issue in this case. Transparency of the procedures at the local level as well as informal systems

46

of accountability among communities were key to assuring satisfactory use of project funds by the communities.

- Procurement problems almost exclusively related to non-CDD components (for example, hiring of consultants and contracting of classroom construction under the district window) requiring procurement by a central agency. Design of CDD projects requires adequate attention to capacity for ancillary procurement needed to complement the procurement at the community level.

- The project scope and coverage experienced major changes during implementation necessitating numerous ad hoc amendments to the procurement plans and change orders to previous awarded contracts. Up to a point this was unavoidable in a program of this nature but perhaps the task team could have done a better job in anticipating some of the changes as well as through more timely adjustments of the procurement plans.

- Given the country context, procurement risk rating of “moderate” may have been too optimistic.

Sample Project 2: Brazil—Maranhão Integrated Program: Rural Poverty Reduction Project

Extracts from Project Appraisal Document

• Project objective: “The proposed project will help Maranhão to achieve the goal of reducing poverty by increasing its [Human Development Index (HDI)] from 0.647 to 0.700 by the year 2007, through …(c) financing demand-driven community investments for income-generation, health and sanitation, education, culture, environmental management and others impacting the HDI and environmental sustainability …”

• Highlighted Components: Community Subprojects ($36 million or 90 percent of total project cost) will support community matching grants for approximately 1,200 small-scale socioeconomic infrastructure, education, health, culture, productive, environmental and other investments aimed at raising the United Nations HDI. All 216 municipalities in Maranhão (other than Sao Luis) will be covered by this component, because of the importance of encouraging cross-sectoral integration throughout the state, getting all municipalities to focus on the HDIs, and strengthening local and municipal institutional arrangements. However, about 60 percent of total component cost will be concentrated in the 80 municipalities with the lowest HDIs.

47

Procurement Arrangements: In early 2003, an independent procurement review was conducted of the predecessor project and concluded that the procurement procedures followed under the project complied with the loan agreement. The State Technical Unit (NEPE) procurement staff was well-versed in Bank procurement policies and procedures. It was anticipated that all procurement financed by the project would be carried out by the beneficiary community associations.

Procurement of goods and works under subprojects costing less than $50,000 implemented in remote areas would be carried out mainly through direct contracting and community participation. This procurement method is appropriate because most subprojects: (i) would be small and/or implemented in scattered or remote areas and therefore it will be difficult to obtain competitive proposals; (ii) can be managed directly by rural communities, who will also contribute directly to the work through the donation of unskilled labor and local materials; (iii) will be selected on the basis of willingness of the beneficiary communities to contribute to and physically supervise works execution; and (iv) would provide means by which communities could play an active role in the local development process.

Additional procurement of goods estimated to cost less than $100,000 equivalent, up to an aggregate amount of $1.5 million, would follow national shopping procedures. Contracts estimated to cost more than $100,000 equivalent would be awarded on the basis of NCB procedures satisfactory to the Bank. Works contracts (other than contracts costing less than $50,000 in scattered or remote areas) estimated to cost less than $100,000 equivalent, up to an aggregate amount of $1.0 million, would be procured under lump sum, fixed price contracts awarded on the basis of quotations obtained from a minimum of three qualified local contractors in response to a written invitation. The invitation shall include a detailed description of the works, including basic specifications, the required completion date, a basic form of agreement acceptable to the Bank, and relevant drawings, where applicable. The award would be made to the contractor who offers the lowest price quotation for the required work, and who has the experience and resources to complete the contract successfully. The standard bidding documents for NCB agreed between the Bank and the federal government of Brazil would be used. No ICB is anticipated for any goods or works under the project.

48

Nature of Procurement Problems ISR 12: This ISR provides an update on recent developments related to the findings of the Procurement Post Review (100 percent coverage) carried out in August 2008. The task team leader has reviewed the PAD/LA for consistencies and reported to the SM accordingly. Following the conclusions of the Post-Procurement Review (evidence of systematic misapplication of Bank procurement guidelines, and red flags of possible fraud/corruption), several measures were taken: (i) INT was promptly notified of the Post-Procurement Review findings, (ii) the Borrower was notified (at the highest level) and given a chance to explain findings (letter attached), and (iii) the Borrower was asked not to approve further community subprojects until a corrective action plan was agreed to with the Bank. The borrower's reply to the findings (from October 17) did not significantly change the conclusions of the Post-Procurement Review. Consequently, the Bank will declare misprocurement. Given the above situation, project implementation has basically stalled.

ICR: The Bank’s Post-Procurement Review revealed prima facie evidence that bidding quotations did not meet Bank requirements. The Mid-Term Review found overly-complex procurement procedures and lack of understanding of procurement rules and processes by associations. NEPE was found not to be providing adequate supervision and guidance to communities on problems to be avoided in procurement, and there was a lack of rigorous analysis by NEPE of problematic subproject documentation in order to make corrections. The Mid-Term Review recommended that NEPE (i) prepare procurement procedures designed to facilitate associations’ involvement, including model documents; (ii) send technicians to the subprojects to review their status; and (iii) expand the use of the MIS as a procurement control and supervision instrument. NEPE was also tasked with updating the Operational Manual, which at the time of the Mid-Term Review had no specific section on procurement. NEPE’s response to these recommendations was slow and incomplete and was in any case overtaken by issues related to the imminent closing of the project on December 31, 2008. Since procurement issues were first flagged only in April 2008, by then it was too late to take meaningful corrective actions.

Concurrent Bank missions on financial management and regular field supervision at the time noted that (i) these procurement irregularities were not related to financial management practices, and (ii) the investment subprojects identified in April 2008 physically had been completed to the satisfaction of the beneficiaries. In other words, the problems identified were related specifically to the misapplication of procurement procedures.

49

The type of problem found in Maranhão (systematic misapplication of Bank procurement guidelines and confusion about how Brazilian Law 8666 on procurement must be applied under the project) has not been identified in the other CDD projects under implementation in Northeast Brazil. Nevertheless, one important lesson from this experience is that the application of Bank-approved procurement procedures for CDD projects requires considerable initial and periodic training of project implementation agencies which, in turn, need to transmit this training to beneficiary community associations. In the particular case of Brazil, this requires more pro-active engagement on the part of the Bank to periodically clarify to implementing agencies when and how Bank-approved procedures, as specified in the loan agreement and operational manual, should be used by communities, and how these may or may not relate to Brazilian Law 8666 on procurement.

IEG Comments - Built on good experience with an earlier project. - Problems arose with compliance with procedure. No evidence of systematic

fraud. Subprojects completed to the satisfaction of beneficiaries. - Importance of post reviews to ensure compliance. - Need for clear guidance to implementing agencies when conflict between

Bank and government procedures.

51

ICT Procurement: Summary of Quality Assurance Group, IEG, and Bank Reports

IEG reviewed evaluative and other lessons that emerge from earlier reviews of ICT that mention procurement, including a Quality Assurance Group assessment, an IEG evaluation, a Bank study on Financial Management Information Systems (FMIS), and the Bank’s ICT Strategy. Three of these were completed in the past two years, indicating that the effectiveness with which the Bank approaches ICT investment is an important topic in the Bank.

The Quality Assurance Group assessment reviewed 23 projects with a combined procurement of ICT components valued at $680 million. 17 Its procurement discussion focuses primarily on resource and skills issues, and procurement planning. With respect to the latter, a key finding is that the special procurement requirements of project ICT components are not properly recognized. This is treated as a project design issue. The report concludes that the root of the problem is weakness in the procurement strategy, as less than one fourth (23 percent) of projects are rated as satisfactory or better in this respect. Today, the Quality Assurance Group assessment is somewhat dated (it covers projects that were under implementation eight or more years ago) but it points toward a broad spectrum of skills and resource issues, and weak procurement planning, that are constraints in achieving project outcomes.

The IEG ICT evaluation covered the Bank’s work in the ICT sector in several broad areas (IEG 2011b). A part of the evaluation dealt with ICT applications such as the use of ICT for delivery of public services. Procurement was assessed for its contribution to their effective delivery. The evaluation offered the following observations and recommendations regarding procurement.

• Procurement is a major constraint to implementing ICT projects. Procurement delays, attributable in part to risk-averse procurement specialists, slow down project implementation, in some cases stalling projects for years.

• The Bank’s procurement rules are designed for infrastructure projects; for example, separating contracting from professional (consulting) services. They are cumbersome and not suited for a rapidly evolving sector like ICT. The Bank needs to better adapt its procurement rules to meet the needs of ICT projects.

• The Bank lacks necessary ICT procurement-related expertise, leading to inconsistencies of approach.

52

• Change management is inherent in projects that have a significant ICT component (modernization of a tax system, for example). This further complicates specifying system requirements and managing implementation. This is a common theme of this study, reported by task team leaders (Indonesia) with direct experience, as evidenced in IEG’s field visits.

The Bank’s recent ICT strategy followed the IEG ICT evaluation and reflects many of its findings and recommendations. In particular it notes a lack of ICT-related procurement expertise at the Bank, inconsistency of approach, and a procurement regime derived from infrastructure projects, not well suited for ICT procurement. Additional points raised by the strategy include:

• A possibly higher incidence of corruption in information technology procurement than in other project components, based on a study in the Europe and Central Asia Region.

• A need to build ICT-related procurement skills within the Bank, but also to build capacity in client agencies, particularly in areas related to developing technical specifications and bidding documents.

• Acknowledgement of recent improvements and agreed areas for future change. The former include the 2011 revisions to the guidelines, which provide for two-stage bidding, framework contracts and continuation of engagement with a brand name supplier. Proposed future actions include the update of the Bank’s ICT standard bidding document, revisions to procurement specialist training materials, and allocation of large complex ICT to the most experienced procurement specialists.

The FMIS study reviewed 55 closed and 32 active FMIS treasury-related projects covering the period 1984 and 2010 (World Bank 2011d). All include large ICT systems, implementation of which was a primary focus of the study. The study pointed toward 18–24 months of typical delay caused by lengthy procurement processes, stretching in rare cases to over two years. 18 This contributed to major delays and dissatisfaction on the part of the borrower. It also pointed towards technical capacity constraints on the Bank’s project teams and the project implementing unit staff of the borrower. The study recommends ensuring ICT expertise on the Bank project team and building capacity of the project implementing unit, with the latter focused on building organic capacity. The study also pointed to the need for appropriate procurement planning and recommends combining procurement into a smaller manageable number of packages, typically

53

three or fewer. Finally, it alluded, in one of its case study countries (Turkey) where procurement using country procurement systems was completed in six months. 19

Box B.1. FMIS Packages The Bank’s FMIS study reported from its case studies that, in most instances, three variants describe the design of procurement packages and procurement methods.

Option 1 (Georgia, Moldova, and Tajikistan): One single-responsibility contract package using two-stage ICB was bid covering the implementation of all ICT software, hardware and network components.

Option 2 (Albania, Azerbaijan, and Russian Federation): Two linked contracts were bid simultaneously involving (1) two-stage ICB procurement of a contract for the development of application software, including demonstration, and (2) single-stage ICB bidding of a contract for the installation of all hardware and network equipment. To ensure compatibility, implementation of the second contract depends on the inputs to be provided by the software developer under the first contract. The second contract was delayed due to this linkage.

Option 3 (Kyrgyz Republic and Ukraine): Two separate independent contracts were bid simultaneously involving (i) two-stage ICB bidding of a contract for the development of applications software and installation of central hardware, including demonstration, and (ii) single-stage ICB bidding for the installation of standard field hardware, engineering support systems and network equipment.

The study concluded with suggestions to improve the design of FMIS procurement:

It is advisable to have two separate independent packages (Option 3). The study found that this reduced the complexity of ICT implementation and resulted in faster implementation. Another possibility is to use Option 1 for a well-defined and relatively small-scale implementation of the FMIS system.

Bidders should be required to comply with all ‘mandatory’ requirements listed in the bidding documents. This ensures that minimum technical requirements are met. Technical evaluation/merit points should be considered only when there is a need to define and evaluate ‘desirable’ requirements that will add value to the minimum mandatory requirements—where there is quantifiable value expected from offers that exceed the minimum requirements.

Where the technical requirements are well defined, the weight of price should be kept as high as possible (80 percent or more) to benefit from price competition, while ensuring the delivery of good quality solutions based on mandatory requirements. The weight of technical evaluation for desirable features can be up to 20 percent, if there is a need to consider some optional high-value technical requirements in addition to mandatory requirements. IEG agrees that the maximum level of 20 percent is reasonable. A higher level leads to more subjectivity in the evaluation and is also an indicator that technical requirements are not well understood.

Source: IEG 2008a.

55

PPP Procurement: Comparison of the Bank and Other Organizations Over the past few years, multilateral development banks (MDBs) have been working to elaborate procurement requirements for PPP projects through their Heads of Procurement. There is an established working group to study PPP-related procurement matters that provides guidance on the matter of “acceptable practices,” and harmonizes approaches among the MDBs. Outputs are meant to guide the different MDBs as their policies develop. They include:

• Procurement & PPP Transactions Guidance for MDB Public-Sector Engagements (2012). This guide aligns closely with the Bank Guidelines and the Guidance Note, although with less detail (for example, in treatment of bidding procedures). It elaborates the role of the MDBs in providing advisory services and guarantees.

• Procurement Principles Applicable to Private Sector Transactions—Guidance for MDBs (2012). As with the guide described above, it reflects existing Bank practice, but expands into other areas. For example, it emphasizes the importance of due diligence, recognizing that private sector projects are often not subject to the same oversight as public sector projects (they might, for example fall outside the jurisdiction of a national procurement law). Typical due diligence requirements and documentation are detailed in attached checklists. The guide also addresses the need for arms-length arrangements (between a client and its affiliates, who might also act as contractors) and provides guidance on guarantees and unsolicited proposals.

The approach of other MDBs is documented in the heads of procurement PPP working group guidance notes described above. As with the Bank, MDB policies are typically spelled out in the procurement guidelines of the organization, and supplemented by interpretation notes and guides. With the exception of the Black Sea Trade and Development Bank, requirements of the other MDBs are very similar to the Bank’s.20 When it comes to on-lending arrangements, where the beneficiary is a PPP party, MDBs leave procurement to the intermediary to oversee, saying that procurement must follow well-established methods and commercial practices “acceptable” to the organization, or similar wording. For selection of concessions, all MDBs specify open and competitive tendering, and all provide for the exception to open tendering by requiring competitive tendering for downstream procurement. There are a few variations between the organizations:

56

• The European Bank for Reconstruction and Development requires a formal competitive process but defines it in a separate concession policy, rather than in its guidelines.

• The African Development Bank (AfDB), Inter-American Development Bank (IDB), and Islamic Development Bank (ISDB) all specify ICB for selection of the concession, as did the Bank prior to the 2011 revision to its guidelines. The European Investment Bank (EIB) specifies “formal international” tender.

• Except for the Bank, ADB, and ISDB, remaining MDBs employ a market comparison to contract terms, saying that costs must be in line with the market and the project budget, and that contract terms should be fair and reasonable.

• EIB invokes compliance to local legislation. It also requires that European Union treaty conditions be met where a concession is not awarded through formal international tendering. These conditions include adequate advertising, international competition, and a fair and nondiscriminatory process.

There is very little variation in the PPP procurement policies of the MDBs, as reflected in their procurement guidelines. Furthermore, through the heads of procurement working group, the MDBs have worked to harmonize such things as interpretations of “acceptability,” further aligning their practices. The policy similarities are unsurprising. Over the past 10 years there has been a push to align procurement guidelines. Differences that exist (AfDB, IDB, and ISDB specifying ICB for selection of concessions, for example) probably reflect one organization getting ahead of the other, rather than any fundamental disagreement. With policies so similar, it is differences in application and interpretation of those policies that most likely differentiate approaches to PPP procurement.

In contrast to the Bank, the International Finance Corporation (IFC) does not have documented requirements regarding procurement in its PPP investments. IFC reviews a concession agreement (and key contracts in a typical transaction) as part of its overall assessment of credit, development and reputational risks in the transaction, rather than as part of a self-standing procurement exercise. IFC takes a commercial approach, assessing if the PPP arrangement was the result of an open and competitive process, prices are reasonable, the project complies with local laws, the project is financially viable, and the parties to the project are reputable, competent, and financially sound. This reflects the commercial orientation of IFC.

Unlike the World Bank Group, which separates private and public sector lending operations between two different organizations (IFC and IBRD), most other MDBs have an in-house private sector lending window. Their operations include

57

departments dedicated to private sector lending, yet they manage to accommodate this private sector lending function within the terms of procurement policies that are almost identical to the Bank’s. Within these organizations, procurement policies, including PPP-specific provisions tend to be developed by public sector procurement specialists (MDB representatives on the heads of procurement PPP working group are all public sector procurement people, except for some participation of IFC). It is understandable then that the PPP-related procurement policies of other MDBs, like the Bank, address public accountability concerns and are not fully private-sector oriented, as are the IFC’s.

Box B.2. Public Accountability frames Government Regulations for PPP Transactions Like the MDBs, governments, and those who may course financing through them with a sovereign guarantee, are constrained by their public accountability. A guide produced by the World Bank Institute gives examples of PPP procurement requirements for six countries (Brazil, Chile, Egypt, Mexico, the Philippines, and South Africa) and the European Union. These examples describe the legal requirements typically contained in a PPP or similar law. Also described are the methods used in the different countries for establishing qualifications of clients, bidding processes to be followed, conditions of negotiations, and basis of awarding concessions (such as price and technical capability). Refer to Table 3.5.1 of the WBI guide for a detailed description of the different PPP procurement procedures.

Sources: World Bank Institute, IEG.

Of the different country approaches described in the World Bank Institute (WBI) PPP reference guide countries following the European Union procedures have the most options (World Bank 2012j). The options allow open or restricted bidding (restricted to a minimum number of prequalified bidders), single or multistage bidding (where full proposals are invited at one time, or different aspects of the proposal are invited in sequence), and different approaches for negotiation, ranging from no negotiations at all to simultaneous negotiations with all bidders throughout the bid and evaluation process. In all cases, European Union regulations allow for contracts to be awarded to the best financial proposal (technically compliant) or the best combination of technical and financial scores. The following variations of country policies as compared to the Bank’s are noted:

• All countries, including those following European Union procedures, use a competitive tendering process to arrive at a PPP concession contract. All permit prequalification as an option, except Philippines, which states it is the “norm” and South Africa, which mandates it (and further limits the number of prequalified bidders to between three and four). Prequalification restricts participation in the bidding to prequalified firms. This is not at odds with Bank policies; the Bank Guidelines mandate competitive bidding and its

58

guidance encourages prequalification. However, the prequalification process itself must be advertised and opened to all eligible bidders, and may not limit the number of bidders as does South Africa.

• All countries require a single-stage bidding process, except Egypt and those using the European Union competitive dialog procedures. South Africa allows the addition of a second stage to bidding where the single stage process does not produce a clear preferred bid (“best and final offer” approach). All of these arrangements are fully in line with the Bank’s guidance, which provides for either single- or two-stage bidding, at the option of the government.

• Negotiation, to arrive at a concession contract, is an area where there appears to be considerable divergence. The policies of three countries (Brazil, Chile, and Mexico) are silent on the topic. The Philippines and South Africa do not allow negotiation, apart from exceptions (where there is only one bid, for example), consistent with Bank policy. Egypt and the European Union provide for a “competitive negotiation” procedure, as an option. The Bank’s guidance note (and also the MDB guidance) does not accept competitive negotiation, where contract award is based on negotiations with several bidders. It is therefore at odds with the policies of Egypt and any country following European Union procedures. It is not clear if the other countries, whose policies are silent on the matter, accept the practice or not.

• Except for Brazil and the Philippines, all countries, including those applying European Union policies, have two options for deciding who is awarded the concession contract. One option is to award on the basis of best financial offer among technically compliant offers (“lowest evaluated bid” in Bank terminology). The second is to use a point scoring system that combines financial and technical proposals to come up with an overall best offer. Brazil and Philippines do not allow the second option. The Bank, in its policies and guidance, accepts both methods, as does the guidance from heads of procurement. The WBI document notes that South Africa applies a preference for black-owned bidders. Similar preferences (for domestic bidders, for example) are not uncommon--the Bank allows some in its public procurement policies and would likely accommodate certain country-specific preference schemes when assessing “acceptability” of the procurement system.

The WBI guide addresses the matter of unsolicited proposals, proposals from a private sector entity to government to enter into a PPP. These are seen positively, as a way to encourage the private sector to promote its ideas and expertise to aid the delivery of public services. Seven countries were surveyed (Chile, Indonesia, Italy, Korea, the Philippines, South Africa, and the U.S. state of Virginia). All accept

59

unsolicited proposals, but then conduct a competitive bidding if the proposal is worthwhile. To incentivize such proposals, the proponent is given an advantage during evaluation, compensated for its development costs if it loses the bidding, or given an opportunity to match a competing winning bid. The heads of procurement group guidance recommends this approach. The Bank’s guidance is vague, saying that awarding a contract to an unsolicited proposal may be considered as an exception under the guideline provisions for direct contracting, or that the Bank may require a competitive process, possibly “adding necessary flexibility.”

61

Appendix C. Procurement Risk Concepts and Thresholds Risk Concepts

Box C.1. Procurement Risk in Relation to the Bank’s Principles of Procurement—Process and Outcome Risk Transparency in procurement process sees to the first injunction in the Bank’s Articles: that loan funds be used for the purposes intended. Sometimes less emphasized (even though measures to explicitly address fraud and corruption in procurement have become more visible) is the need for transparency in contract management and execution, so transparency in outcome is also needed. Transparency (widespread advertising of tenders) also serves the equity principle by ensuring tender information is broadly disseminated. The economy principle has been intended to embody a prime purpose of procurement, namely to secure for the Borrower optimal value for money in the use of Bank loan funds, as the outcome of a competitive tender, against the risk that many factors may work to impede that optimal outcome. While the achievement of an optimal outcome may depend on an effective procurement process, the optimal economic result itself is an outcome concept. As increasingly recognized today, “economy” on its own may not be adequate, especially if narrowly interpreted, for a best value outcome. The efficiency principle relates exclusively to the effectiveness of the procurement process (in pursuit of the economic outcome) and addresses the risk that the procedural requirements of procurement policy may be such as to be overly cumbersome, costly and slow to an extent that could slow project implementation. To that extent highly inefficient procurement processes or systems may be a factor which impacts project outcomes, but the efficiency principle relates exclusively to process. Too seldom emphasized in Bank discourse is the fact that there is a key risk-efficiency trade-off embodied in the balancing of efficiency and economy, which is dealt with in more detail in the main text.

The equity principle requires that, in the execution of its procurement policies, the Bank ensure that suppliers from all eligible countries be given equal knowledge and an equal opportunity to bid on the tenders for contracts under projects being financed by the Bank. Clearly, the equity principle cannot be extended to outcomes, since there can be only one winner per bid. To that extent, the equity principle relates also only to process.

The local development principle historically supported the purpose of World Bank lending to bring development to developing countries markets. (As data from procurement outcomes in recent years show an increasing degree to which Part II country suppliers have won ICB tenders on Bank projects —now in excess of 60 percent — suggest that the risk that favoring local suppliers may conflict with the economy principle have probably declined.

Procurement Principle Process Outcome Risk Addressed Transparency X X Loans not used for purposes intended Economy X Procurement gives sub-optimal value for money, Efficiency X PR process too cumbersome, slow, delays outcome

and project implementation Equity X Uneven playing field between eligible bidders Local development - Goods, contractors, consultants - Local procurement systems

X

X

Local suppliers, contractors shut out by international bidders Local PR systems not compatible with Bank policy

62

63

Box C.2. Procurement Risk Management—Different Levels in the Bank Procurement risk is dealt with at different levels in the Bank, including the entity level, country and sector levels, and, most importantly, at the level of individual project transactions. The extent to which procurement risk is treated as a separate set of risks, as against being integrated into more general operational risk, varies both across levels and within them.

At the level of the Bank as a whole (entity level): There are three key functions at the entity level which view procurement risk in real time, as part of ongoing operations:

• Integrated Risk Management Report: This report, which is the Bank’s most aggregated report on risks facing the institution as a whole is based on a survey of Bank-unit “appetites and tolerances” for 41 identified risk items. One of these items is “FM, Procurement and Disbursement risk,” which views procurement risk mainly from the perspective of how it may impact the Bank’s overall disbursement rate. The Chief Risk Officer unit is currently engaged in a brainstorming pilot analysis which is attempting to seek answers to a number of procurement risk related questions, with a view to establishing the appetites and tolerances for this risk factor. Some key questions being explored include: How is procurement risk managed? Beyond the taxonomy of risk management tools, how much procurement risk should the Bank take on? Does the Bank follow a risk-return model in procurement where higher risks maybe tolerated where returns to development (measured either by IRR, progress towards PDOs, or IEG outcomes) are higher? How should procurement risk be flagged? Has the bar for procurement protection been set too high or too low? It is anticipated that this process could be completed by end FY13, but this will depend on senior management (VPU/EVPU) sign-off, and could also be potentially impacted by the arrival of a new Bank CFO in March.

• The Chief Procurement Policy Officer: This is the Bank’s chief authority on procurement policy, and in that capacity is the apex of the Bank’s central and operational (Regional and Network) based procurement specialists who administer the function throughout the Bank, including a significant contingent of decentralized specialist staff located in the Bank’s field offices in many countries. This team represents the core of the Bank’s procurement implementation capability, it upholds procurement policy, it advises operational staff at the level of project transactions and their procurement requirements, it acts to settle complaints and disputes, and it offers a training component to both Bank staff and staff within borrowing countries involved in implementing Bank projects. In that sense this officer commands a team that is the backbone of the Bank’s capability in managing procurement risk at all levels of the institution. He or she also serves as an ultimate arbiter on procurement policy matters—often involving real time disputes on specific procurement contracts—where these cannot be resolved at a unit or regional level.

• The Operations Procurement Review Committee (OPRC): The OPRC plays a key oversight role by reviewing and clearing the highest-value and most complex contracts financed under Bank-supported operations. All contracts above a defined threshold level must be reviewed by the OPCR, and some complex or sensitive contracts may be reviewed, regardless of their value. The thresholds for mandatory review were increased in January 2009 (unchanged since 1992) to (i) $50 million or more for civil works; (ii) $30 million or more for goods; (iii) $15 million or more for consultancy services; (iv) $5 million or more for all direct contracting and single-source contracts with firms; and, (v) $1 million or more for single-source contracts for individual consultants.

64

Source: IEG interviews with Bank staff.

Box C.3. Procurement Risk Management Relative to Other Operational Risks in Bank Projects An important question in managing procurement risk at the level of individual projects is how this risk is viewed in comparison with other operational risks facing Bank projects. The Bank addresses this issue by distinguishing four categories of risk that affect projects, two of which (Country Risk and Sector Risk) are external to the projects, while the remaining two (Inherent Risk and Control Risk) refer specifically to factors arising from the project itself. Risks are rated at four levels: high, substantial, moderate, and low. The categories are as follows: External to the Project

• Country Risk: Country risk relates to the general standing of the country in terms of business environment, operational risk, strength of institutions, stability of the economy, status of procurement reform, record on governance and general risk of fraud and corruption. Country risk assessments, including relevance to procurement (for example, as linked to CPIA results, and/or governance and fraud and corruption risk) should be contained in the Country Assistance Strategy for each borrowing country.

• Sector Risk: This relates to risks generic to specific sectors (including those derived from strength of institutions, investment types and commercial risks) that should be reflected in Sector Strategy Notes compiled by the networks.

Specific to the Project • Inherent Risk (Project Risk): For a given project the relevant elements of Country, Sector

and Project risk, including the overall business environment and procurement capacity of the country, design of the project, its technical complexity, its operational and/or commercial objectives, market conditions, technology and financial risks, all combine to form the Inherent Risk of the project. These are independent of specific procurement risk factors, which are treated separately (see below) but, once identified, procurement risk should be an integral part of overall project risk.

• Control Risk: Control risk is related to the internal control structure and the capacity of the implementing agency for the project to competently handle three aspects of procurement execution: Governance, Capacity in policies and procedures (knowledge of Bank Guidelines and systems), and Oversight of procurement implementation. In brief, Control Risk reflects the capability of the implementing agency to manage procurement process risk. Part of this capability involves the capacity to develop a dynamic Procurement Risk Mitigation Plan containing specific measures to address key procurement risk factors, and to monitor their application over time. In standard Bank practice, all Project Appraisal Documents (PADs) contain a procurement annex (written by the procurement specialist, but drawn from material produced by the borrower) built around the two key dimensions of control risk: An implementing agency capacity assessment; and the quality and effectiveness of the Procurement Risk Mitigation Plan.

• Procurement Risk: Procurement risk is the risk deriving from the specific nature of the supply and service contracts that have to be procured to implement the various components of the project, as guide by the Bank’s or other procedural guidelines, with due regard to the four overriding principles contained in Box C.1. (As described below, there is a need to clarify this usage within the risk taxonomy currently used by the Bank)

• Residual Risk: The Residual Risk is the level of procurement risk remaining in the project (in the judgment of the procurement specialist) after implementation of the procurement risk mitigation plan.

Sources: OP/BP11.00, the P-RAMS template; OPCS Reports, for example, Use of Country Procurement Systems Proposed Piloting

65

Program. OPCS May 20, 2008 Annex D.

Box C.4. Managing Procurement Risk in Projects—Summary of Tools and Procedural Devices Available to Procurement Staff to Manage Procurement Risk

• Project Design: Project design is mainly driven by project content and objectives, but quite often project components are chosen because their procurement features set them apart from other components. Many projects also have project implementation components designed to finance the enhancement of implementation capacity of the implementing agency. Projects with highly decentralized objectives often have procurement delivery as part of the project design. • Procurement Plan: Written by the Borrower, under Bank supervision, and mandatory for every project, this lays out the whole procurement strategy and content and highlights the risks to be addressed. It is intended to be updated at least annually, and in that sense is both a planning and a dynamic monitoring tool. • Selection of Implementing Agency: The Borrower may propose but the Bank must agree on who will be the implementing agency. The choice is most often between a Ministerial unit or agency, and project implementation/management unit (project implementing/managing unit) required by the Bank if it deems the Ministry unit to be lacking in capacity.

• Staffing: It is very common for measures to be taken to enhance the skills and experience of the implementing agency staff (whether in the project implementing unit or another) by either or both training and hiring of consultants to the agency as a means of enhancing capacity to implement procurement.

• Procurement Method: Within the broad thresholds set Bank-wide by OPCSOR (and within that framework separately by the Regional RPM) project staff are free to define which procurement method will be used for which project components, based on their judgment of risk and efficiency in achieving an economic outcome. ICB + Prior Review (and/or NCB) remains the default procurement method for all larger contracts (both goods and works), but newer forms of lending (particularly in the social sectors) have increased the share of other methods.

• Use of Pre-/postqualification: In large, complex projects with substantial technology dimensions task team leaders/procurement specialists may choose to require pre-qualification of bidders, to reduce know-how risk in procurement.

• Thresholds for Procurement Method and Review: Maximum Prior Review thresholds are set institutionally (in Annex C of BP 11.00), for goods, works and services, with different levels for each risk category of the implementing agency. Regional Project task team leaders/procurement specialists can set lower thresholds where they believe risks are greater. Thresholds are a key tool for risk management at the project level. PS may also increase the sample of contracts

• Timing of Procurement: In large and/or complex procurements task team leaders/procurement specialists may elect to conduct advanced procurement (sometimes with retroactive financing) in order to ensure procurement does not delay project implementation. Frequently pre-appointment of procurement staff to project implementing/managing unit may be a condition of project approval or effectiveness. • Frequency of Supervision: In higher risk projects procurement specialist may call for more frequent than annual supervision.

• Use of website: In pursuit of the Transparency principle many projects now use web sites as a means to display procurement tenders, awards and outcomes. E-procurement is being encouraged but is not yet widespread.

• Financing of Incremental Operating costs: As a means to ensure that project implementation is properly resourced projects are often designed to include the salary and other operating costs of implementing agencies (excluding civil servant salaries where these units are staffed by government staff). • Addressing Fraud and Corruption: The requirement to address ethical behavior and fraud and corruption risk in projects was introduced into the Procurement Guidelines in 1996, but was not widely practiced until a decade later, following the governance and anti-corruption program and IDA internal controls review. Various tools are available:

o Ethics Pledge: Some projects require an ethical conduct pledge from project counterparts. o GAAP/IS: Recent IL reform and the governance and anti-corruption have introduced Implementation Support (IS) and Governance and Accountability Action Plan to focus and resource implementation and explicitly address transparency and fraud and corruption risk. o Civil Society Oversight: Increasing use is being made in project design of a role for CS in oversight, often involving fraud and corruption risk and using the project website, among other means. o Establish Complaints process is a common means of providing access to red flags and indicators of fraud and corruption risk.

66

to be post-reviewed (generally 20 percent); they may call for Post-Procurement Reviews annually or twice a year, and may call for Independent Procurement Reviews.

• New Tools –P-RAMS/ORAF: Mandatory since July 2010 the P-RAMS provides a detailed checklist in the implementing agency capacity assessment of questions around 11 risk factors, and a Mitigation Measure Action Plan aimed at dynamically assessing all forms of procurement risk, including fraud and corruption risks and intends to inform the ORAF and the PAD.

Source: IEG review.

67

Thresholds Analysis – “Methods” Thresholds

SETTING METHOD THRESHOLDS TO OPTIMIZE COMPETITION

The Implementing Agencies of borrowing countries can use a variety of procurement methods in World Bank funded projects. The method selected depends on a number of factors including the type and value of good, work or service procured and the potential interest of domestic and foreign bidders. 21 The Bank sets monetary threshold above which certain methods have to be used. In the case of goods and civil works these thresholds indicate, among others, above which contract value ICB shall be used.22 Contracts below this value are most prominently procured through national competitive bidding (NCB).

Monetary procurement method thresholds are set for each borrowing country by the regional procurement managers (RPMs), drawing on analyses of procurement risk undertaken as part of Country Procurement Assessment Reviews (CPARs) or other means, and including foreign bidders’ interest in participation in domestic markets; conditions, size and depth of the market and capacity of the local industry.23

ICB thresholds therefore show a significant degree of variation across countries and within regions. More sophisticated clients and middle-income countries tend to have higher thresholds. The maximum ICB threshold used by the Bank is $25 million, for civil works contracts in Brazil. In East Asia and the Pacific ICB thresholds for civil works vary from $100,000 in Kiribati to $20 million in China within the same region. Some country-specific thresholds are set with a monetary range for risk categories low to high, as in India, where ICB is used for works contracts starting from $10 to 20 million.

One feature distinguishing ICB and NCB lies in the requirements for the publication of Specific Procurement Notices. Paragraph 3.4 of the Bank’s procurement guidelines stipulates that borrowers may limit advertising to the local press for NCB processes. Foreign firms cannot be barred from competing for NCB contracts; the different treatment of advertising for NCB and ICB reflects the position that foreign firms would not normally bid for NCB contracts given their value and nature.24

With its procurement guidelines the Bank aims to allow borrowing countries to buy high quality goods and services as economically as possible and expresses its position that this objective is best achieved through transparent, formal competitive bidding. For the procurement of equipment and civil works, ICB is the procurement method the World Bank encourages its borrowers to use in the majority of cases.25

68

The setting of ICB threshold is critical given the expected benefits in competition and economy in the tender process. There is, however, a possible tradeoff with processing time and efficiency. ICB procurement tends to take longer than contracts awarded with NCB procedures, as IEG’s analysis of processing times in procurement in Chapter 4 suggests.

ANALYSIS OF DISTRIBUTION OF ICB CONTRACTS

IEG has attempted an analysis similar to the one undertaken by the AfDB (as described in Chapter 3). Using contract data from its budget tracking system, AfDB has shown that in works contracts foreign bidders entered bids on contracts for mostly those contracts in the top quintile (20 percent) of contract value. The AfDB concluded, accordingly, that to set the ceiling contract for ICB at any level lower than 80 percent of highest value would involve no increase in competitive bidding but would involve significantly more review work.

To analyze the trends in methods used for procuring civil works in World Bank-funded projects, IEG used recent years of the Bank’s Form 384 database on prior reviewed contracts financed by the Bank.26 Between 2007 and 2012 the Bank financed 3,618 prior reviewed civil works contracts procured through ICB (out of 12,572 Civil Works contracts total). This gives an average of around 50 prior reviewed civil works contracts procured through ICB per month. On the other hand 9,453 goods contracts were procured through ICB during the same period, an average of around 130 contracts per month.

IEG undertook an aggregate analysis for procurement of civil works contracts only, but suggests a closer look at cluster of countries with similar parameters to determine the impact of current method thresholds. Based on their value civil works contracts procured through ICB between 2007 and 2012 were divided into quintiles and disaggregated into national and foreign contractors.

Participation of foreign firms, in terms of numbers of contracts, is relatively low in the first four quintiles, ranging from 15.5 percent in the first quintile to 24.2 percent in the fourth quintile (Table C.1). These results vary only minimally when looking at the distribution by total value in those quintiles, though they are slightly higher than what is found in AfDB’s analysis; that is, more foreign firms win contracts in the lower quintiles of World Bank funded projects.

Table C.1. Distribution of Civil Works Contracts Procured through ICB (2007–12) By number Quintiles Value of

contracts ($m) National contractors

(nos) % Foreign contractors (regional

and nonregional) (nos) %

69

1 0 - 0.6 612 84.5 112 15.5 2 0.6 - 1.5 574 79.3 150 20.7 3 1.5 - 4.1 570 78.8 153 21.2 4 4.1 - 11.0 549 75.8 175 24.2 5 >11.0 436 60.3 287 39.7 Upper Percentiles 90th percentile >23.3 189 52.4 172 47.7 95th percentile >41.1 75 41.7 105 58.3 By value Quintiles Value of

Contracts ($ millions)

National Contractors (total $ millions)

% Foreign Contractors (Regional and Non-Regional) (total $ millions)

% Average Contract Value ($ millions)

1 0 - 0.56 169.0 85.3 29.1 14.7 0.3 2 0.56 - 1.54 569.0 78.9 152.0 21.1 1.0 3 1.54 - 4.11 1,500.0 78.7 406.0 21.3 2.6 4 4.11 - 11.00 3,710.0 75.7 1,190.0 24.3 6.8 5 >11.00 13,300.0 50.6 13,000.0 49.4 36.3 Upper Percentiles 90th percentile >23.30 9,460.0 46.0 11,100.0 54.0 56.9 95th percentile >41.10 5,990.0 39.9 9,010.0 60.1 83.3 Source: IEG analysis. The findings of IEG’s analysis suggest that about 80 percent of prior reviewed ICB contracts for civil works in the lowest four quintiles, with a maximum value of $ 11 million, were won by national firms.

Although the results of IEG’s analysis are broadly similar to findings of the AfDB, there are some differences: in the highest quintile of contract sizes (with values greater than $ 11 million) in World Bank projects more contracts tend to be won by national contractors than in the case of the AfDB. In the case of the World Bank the majority of these high value contracts were won by national firms, with foreign firms winning only 40 percent.27 The picture changes for the World Bank in the highest five percent of contracts (with values greater than $41 million), where more contracts were awarded to foreign firms.

It has to be noted that the AfDB paper assumes a uniform level of ICB threshold for all countries while the World Bank uses country-specific thresholds that vary significantly as described above. Also, in practice, at the World Bank, the ICB procurement method may frequently be used even below the threshold set for the country; indeed the analysis suggests that average contract values are relatively low for the first two quintiles where 1,448 contracts with an average value below one million were procured through ICB (see also section on prior threshold analysis earlier in this chapter). In FY12 half of the civil works contracts procured through ICB used this procurement method for contract values below the threshold set on the country level.

70

If the upper limit of the fourth quintile was to be considered as the ICB threshold for civil works for the contracts in this aggregate analysis, the number of contracts with mandatory ICB use would have reduced significantly. This would represent an improvement in efficiency in terms of procurement processing time. In a previous section IEG analyzed a sample of contracts and extracted average processing times for procurement. The results found suggest savings of 93 days on average per contract when moving from ICB to NCB. However, given the fact that they are established on the country level, analysis of method thresholds for cluster of countries with similar characteristics is key. It would be important to quantify the trade-off between procurement efficiency and risk especially for groups of countries with lower ICB thresholds as explanatory factors might not be uncovered with an aggregate analysis.

A caveat to the above analysis is the known limitations, in terms of quality and reliability of Form 384 data. Once the Bank’s proposed electronic no objection goes fully into effect, such limitations will pose fewer problems.

71

Thresholds Analysis – “Prior Review” and “Clearance”

DATA SOURCES

Data Sources – Bank-Wide Prior Review and Clearance Thresholds

Data for the Bank-wide prior review and clearance threshold analysis were compiled from archived and current OP/BP 11.0 files and corresponding memoranda on the Bank’s intranet. There is no central repository for this information and it may be incomplete.

Prior review threshold information from 1997 by clearance level were the earliest available to IEG (Sanchez 1997). Reference to Bank-wide maximum prior review thresholds are found in 2008 and 2009.28 Clearance thresholds were revised in 2002.29 Further revisions of some clearance thresholds are found in the 2008 prior review threshold updates.30 Mandatory review thresholds for RPMs and the OPRC were further updated in 2009, as were regional and country level revision of prior review thresholds. Some regions, for example, Africa, adopt further levels of delegation for clearance (with corresponding thresholds) between procurement specialists, senior procurement specialists and hub coordinators.

Data Sources—Regional Prior Review and Procurement Method Thresholds

Regional and country level information on previous years’ prior review and method thresholds is not maintained on a uniform basis across all regions and is readily available only for a few regions. Historical threshold information was readily available for the East Asia and Pacific Region (from 2006), the Europe and Central Asia Region (from 2006), the South Asia Region (from 2008), and for Africa (from 2004). Historical threshold information was not readily available for the Latin America region or for the Middle East North Africa Region. Current regional threshold information was taken from the extranet, and as this is not up to date, it was supplemented by internal sources.31

Data Sources - Project Level Prior Review Thresholds and Project Level Risk

Project Appraisal Documents and procurement plans were scanned for project level prior review thresholds for goods and works, project risk, including residual project risk, procurement risk and residual procurement risk, on a sample basis for 59 projects, based on overall country selectivity criteria. IEG also drew upon additional project level prior review data in the Middle East and North Africa dashboard and in the procurement plan execution system (SEPA), adding eight more projects to the sample. IEG used the residual procurement risk rating when available, or, in its

72

absence, the procurement risk rating. Threshold data sources available to and used by IEG are summarized below:

Table C.2. Prior Review and Procurement Method Thresholds Available to IEG Region Prior Review Threshold Information and Source Procurement Method Threshold Information and Source AFR Selected countries – 2004/2005 (RPM); 2012 (RPM) 2010 (external procurement website)

2012 (RPM) EAP 2006, 2011(EAP RPO website); 2012 (external

procurement website) 2006, 2011(EAP RPO website); 2012 (external procurement website)

ECA 2006, 2007 (four revisions in 2007), 2010, 2011 (ECA RPO website); 2011 (external procurement website)

2006, 2007 (four revisions in 2007), 2010, 2011 (ECA RPO website); 2011 (external procurement website)

LCR Brazil, Mexico, Colombia – 2007 (RPM) Brazil, Mexico, Colombia – 2007 (email from LCR RPM), 2012 (external procurement website)

MNA 2011 (external procurement website) 2011 (external procurement website) SAR 2008 (SAR RPO website); 2012 (external procurement

website) 2008 (SAR RPO website); 2012 (external procurement website)

Source: IEG. Note: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LCR = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Africa.

Data Sources—Governance Indicators

Governance indicators used in the analysis include Transparency International’s Corruption Perceptions Index, the Kauffmann and Kraay Governance Indicators, and the World Bank’s County Policy and Institutional Assessment (CPIA). The Index scores provide a score, ranking a country on how corrupt its public sector is perceived to be on a scale of 1 to 10 (with 10 indicating no corruption).32 Two Kauffmann and Kraay Governance indicators were used for this analysis: Government Effectiveness and Control of Corruption. The overall CPIA score and two sub indicators were used for this analysis: subindicator 13 Quality of Budget and Financial Management and subindicator 16 Transparency, Accountability and Corruption in Public Sector.33

METHODOLOGY

Sample Construction – Correlations with Governance Indicators

Two samples were constructed, one from 2008 and one from 2011, for the comparison of Bank thresholds with external measures of country risk/country governance. The nonstandardized nature of the setting of thresholds across regions, and gaps in historically available information that varied considerably across regions are caveats to the sample construction.

73

Box C.5. Threshold Analysis – Sample Construction 2011 Sample:

Europe and Cental Asia: Data from thresholds effective as of 2011 were used. Prior review thresholds are provided by procurement type (goods/works/consultants) and by procurement methods (ICB/NCB/Shopping).

East Asia and Pacific: Data from thresholds effective as of 2011 were used. Prior review thresholds are provided by procurement type (goods/works/consultants) and by procurement methods (ICB/NCB/Shopping).

South Asia: Data from thresholds effective as of February 2012 were used. Prior review thresholds are provided by procurement type (goods/works/consultants). Previous thresholds dated from 2008.

Latin America and the Caribbean: Current (2012) thresholds were used. It was assumed that prior review thresholds were the same as NCB/ICB method thresholds in the absence of separate information.

Middle East and North Africa: Current prior review thresholds were used (2012). Prior review thresholds are provided by procurement type (goods/works/consultants) and by procurement method (ICB/NCB/Shopping).

Africs: 2010 procurement method (ICB/NCB) thresholds were used as 2011 prior review thresholds.

2008 Sample:

Africa: RPM guidelines from 2005 outlining prior review thresholds for projects with three levels of risk were used for the 2008 sample.

East Asia and Pacific: Data from thresholds effective as of 2006 were used. Prior review thresholds are provided by procurement type (goods/works/consultants) and by procurement methods (ICB/NCB/Shopping).

Europe and Central Asia: Data from thresholds effective as of 2007 were used. Prior review thresholds are provided by procurement type (goods/works/consultants) and by procurement methods (ICB/NCB/Shopping).

Latin America and the Caribbean: Threshold information for Brazil, Colombia, and Mexico. Other countries in the region were not included in the 2008 sample because data were not available.

Middle East and North Africa: Historical threshold data unavailable –not included in the 2008 sample.

South Asia: Data from thresholds effective as of 2008 were used. Prior review thresholds are provided by procurement type (goods/works/consultants).

Sample Construction Issues:

Diverse practices across Bank regions in terms of how prior review thresholds are set led to the construction of subsamples by IEG to maintain homogeneity. In the 2011 sample, thresholds from the Middle East and North Africa are divided into four categories by risk level of implementing agency (mirroring the format for Bank wide prior review thresholds established in 2009). Africa countries included in the 2008 sample had three risk levels (high, medium, and low risk), which also mirrors the format of Bank wide prior review thresholds from 2008. Some countries from South Asia used a range for their prior review threshold (for example, the 2008 prior review threshold for goods is “300,000 to 500,000”). IEG divided the thresholds into equivalent risk categories – for example, the 300,000 threshold was interpreted as the threshold for a high risk implementing agency and the

74

500,000 threshold was interpreted as a low risk implementing agency.

For Europe and Central Asia and East Asia and Pacific Regions: Some thresholds were also set in terms of the first few projects reviewed (for example, Turkey 2011 prior review threshold for NCB goods is “First 2 & >500k”). In such cases the numerical threshold was considered the effective threshold and the “First 2” prior review requirement is not reflected in the analysis and correlations.

Source: IEG analysis. Correlations between country level prior review thresholds and governance indicators were undertaken in clusters, reflecting different practices in setting thresholds across regions. For each coefficient, the vector of country level risk thresholds was correlated with each vector of country level governance indicators. A subgroup of countries was constructed to include those that did not differentiate thresholds by risk level of implementing agency. These countries were pooled with the overall sample, using three sets of assumptions, in the 2008 subsample, namely, that risk thresholds in these countries corresponded, in turn, to the high, medium or low risk implementing agency thresholds. Similarly, for the 2011 subsample, four series were constructed to accommodate high, substantial, moderate and low risk implementing agencies. Samples were also divided by category and method: ICB goods and ICB works, NCB goods, NCB works, to ensure consistent comparisons.

RESULTS – ALL COUNTRIES

• There are no strong correlations. The majority of the correlation coefficients are positive but weak/moderate in strength. There are a number of negative correlations, all very small in magnitude.

• Correlations for NCB goods and works are lower than for ICB goods and works.

• In most cases when comparing the correlation coefficients across the different levels of risk, the strongest coefficient is on the high risk thresholds, with lower coefficients on the substantial, moderate or low risk coefficient.

75

Table C.3. Correlations between Country Thresholds and Indices of Country Governance

CPIA - Overall

CPIA Subindicator 13. Quality of Budget

and Financial Management

CPIA Subindicator 16: Transpar. Account.and

Corrup.in Pub. Sec

Transparency International -

Corruption Perceptions

Index

Kauffmann and Kraay -

Regulatory Quality

Kauffmann and Kraay - Control of Corruption

2008 ICB goods H (68) 0.51 (68) 0.45 (68) 0.32 (67) 0.46 (68) 0.57 (68) 0.25 M (66) 0.41 (66) 0.33 (66) 0.34 (65) 0.44 (66) 0.47 (66) 0.29 L (68) 0.13 (68) 0.09 (68) 0.19 (67) 0.28 (68) 0.18 (68) 0.22 ICB works H (68) 0.46 (68) 0.41 (68) 0.18 (67) 0.30 (68) 0.43 (68) 0.16 M (66) 0.47 (66) 0.42 (66) 0.22 (65) 0.30 (66) 0.42 (66) 0.17 L (68) 0.44 (68) 0.38 (68) 0.19 (67) 0.30 (68) 0.40 (68) 0.18 2011 ICB goods H (117) 0.33 (117) 0.32 (117) 0.20 (123) 0.12 (123) 0.28 (123) 0.15 S (110) 0.30 (110) 0.29 (110) 0.20 (116) 0.11 (116) 0.25 (116) 0.14 M (110) 0.16 (110) 0.14 (110) 0.12 (116) 0.04 (116) 0.13 (116) 0.05 L (117) 0.03 (117) 0.03 (117) 0.09 (123) 0.00 (123) 0.05 (123) -0.01 ICB works H (116) 0.41 (116) 0.36 (116) 0.24 (120) 0.24 (122) 0.40 (122) 0.20 S (109) 0.35 (109) 0.30 (109) 0.23 (113) 0.23 (115) 0.35 (115) 0.21 M (109) 0.29 (109) 0.36 (109) 0.20 (113) 0.18 (115) 0.29 (115) 0.13 L (116) 0.22 (116) 0.32 (116) 0.18 (120) 0.13 (122) 0.23 (122) 0.08 2008 NCB goods H (51) 0.39 (51) 0.39 (51) 0.25 (51) 0.27 (48) 0.45 (48) 0.14 M (49) 0.08 (49) 0.01 (49) 0.18 (49) 0.18 (46) 0.19 (46) 0.26 L (51) -0.11 (51) -0.19 (51) 0.06 (51) 0.04 (48) -0.02 (48) 0.20 NCB works H (54) 0.39 (54) 0.36 (54) 0.00 (54) 0.11 (54) 0.35 (54) -0.04 M (52) 0.38 (52) 0.36 (52) 0.01 (52) 0.14 (52) 0.34 (52) 0.04 L (54) 0.30 (54) 0.26 (54) 0.02 (54) 0.15 (54) 0.25 (54) 0.10 2011 NCB goods H (32) 0.29 (32) 0.09 (32) 0.26 (37) 0.14 (37) 0.44 (37) 0.29 S (28) 0.07 (28) -0.04 (28) 0.12 (33) 0.16 (33) 0.14 (33) 0.06 M (29) -0.1 (29) -0.14 (29) -0.02 (34) -0.08 (34) -0.09 (34) -0.09 L (32) -0.11 (32) -0.14 (32) -0.05 (37) -0.09 (37) -0.12 (37) -0.10 NCB works H (33) 0.38 (33) 0.26 (33) 0.27 (38) 0.21 (38) 0.51 (38) 0.35 S (28) 0.13 (28) 0.06 (28) 0.14 (35) 0.07 (35) 0.22 (35) 0.12 M (29) 0.03 (29) -0.02 (29) 0.06 (35) 0.00 (35) 0.08 (35) 0.03 L (33) 0.00 (33) -0.07 (33) 0.01 (38) -0.03 (38) 0.00 (38) -0.04 Source: IEG analysis. Note: All countries for which risk level data are available. For those countries and years which did not differentiate country risk by implementing agency, successive correlations have been undertaken assuming, in turn, that such risk levels correspond to each level on the three- (2008) or four- (2011) point scale for risk used in other countries.

76

RESULTS – SUBSAMPLE OF COUNTRIES (AFRICA AND LATIN AMERICA AND THE CARIBBEAN)

IEG also analyzed results separately for the group of countries, in two regions especially, for which country threshold data do not differentiate thresholds by risk level of implementing agency (that is, there is only one threshold by procurement method/type for the country).

Table C.4. Correlations between Country Thresholds and Governance Indices – Single Risk Threshold Countries

CPIA – Overall

CPIA subindicator 13. Quality of Budget and Financial Management

CPIA Subindicator 16: Transpar., Account. and Corrup. in Pub. Sec

Transparency International - Corruption Perceptions Index

Kauffmann and Kraay - Regulatory Quality

Kauffmann and Kraay - Control of Corruption

2008 ICB goods (43) 0.56 (43) 0.51 (43) 0.44 (43) 0.52 (43) 0.61 (43) 0.34 ICB works (43) 0.50 (43) 0.46 (43) 0.27 (42) 0.33 (43) 0.44 (43) 0.23 2011 ICB goods (102) 0.34 (102) 0.33 (102) 0.21 (105) 0.12 (105) 0.28 (105) 0.17 ICB works (101) 0.41 (101) 0.36 (101) 0.26 (103) 0.26 (104) 0.41 (104) 0.22 2008 NCB goods (26) 0.46 (26) 0.42 (26) 0.39 (26) 0.36 (25) 0.52 (25) 0.26 NCB works (29) 0.45 (29) 0.44 (29) 0.02 (29) 0.19 (29) 0.40 (29) 0.10 2011 NCB goods (20) 0.45 (20) 0.20 (20) 0.41 (22) 0.29 (22) 0.73 (22) 0.52 NCB works (22) 0.63 (22) 0.50 (22) 0.46 (24) 0.41 (24) 0.88 (24) 0.64 Source: IEG analysis. Note: These countries do not differentiate thresholds by risk level of implementing agency (ie there is only one threshold for the country), at the country level. Most consist of countries in the Latin America and the Caribbean and Africa Regions.

• Although correlations for this subset of countries was generally higher than for all countries taken together, only a few are over 0.5.

• As in the overall group, correlations for ICB goods and works are stronger overall in 2008 than in 2011. For NCB goods and works there is variance – for some indicators the correlation is higher in 2008, for others it is higher in 2011.

• The indicator that has the strongest correlations by year, procurement category, and method is the Kauffmann and Kraay Regulatory Quality Indicator, and may best reflect the implicit criteria in terms of county governance/capacity that RPMs use in determining prior review thresholds.

These data are displayed in Figure C.1 for the years 2008 and for one governance indicator, the CPIA score. These offer more granular country level insights and show

77

that while the highest prior review thresholds do tend to be found in countries that have relatively high CPIA scores, there are many thresholds that are very low among countries that have high CPIA scores, notably Armenia, Bhutan, Georgia, Macedonia, and Thailand from the 2008 graph and Chile, Georgia, Macedonia, Peru, Thailand, and Uruguay in the 2011 graph. Graphs based on other procurement methods and governance indicators show a similar pattern.

Figure C.1. CPIA Scores and Correlation with Prior Review Thresholds (ICB Goods, 2008)

Source: IEG compilation. A caveat to the analysis is that conclusions must take into account the quality of the underlying data, which is heterogeneous by region and not prepared on the same basis. Moreover, the dates at which revisions have occurred appear to be ad hoc, and have not been uniform across regions. In some regions, prior review threshold are also the method threshold (Latin America and the Caribbean, Africa, and some countries in the Middle East and North Africa), and in others this is not the case but relates to the challenges of interpreting threshold data. Sample sizes are small because of the absence of better historical prior review threshold information.

78

Figure C.2. CPIA Scores and Correlation with Prior Review Thresholds (ICB goods, 2011)

79

Post-Procurement Reviews

Table C.5. Post Procurement Reviews: Project Sample

Country Project ID Project Title PPR FY Contract Value

($) Africa Tanzania P055120 Transport Sector Support 2011/12 64, 458 Tanzania P111155 Zanzibar Urban Service Project 2011/12 694,848 Tanzania P111556 East Africa Public Health Laboratory 2011/12 109,705 Tanzania P114866 Secondary Education Development 2011/12 306,052 Ethiopia P101556 Electricity Access Additional Financing 2012 East Asia and Pacific Indonesia 7866 ID Empowerment of Urban Areas 2010 Lao PDR P120909 Upland Food Security Improvement 2012 989,981 Philippines P066076 Judicial Reform Support 2011 41,800 Europe and Central Asia Turkey 7539-TU Fourth Export Finance Intermediary Ln. 2012 12,227,940 Ukraine P095203 Second Export Development Project 2011 4,682,626 Azerbaijan P100582 Real Estate Registration Project 2011 Azerbaijan P083126 Real Estate Cadastre and Registration 2012 Middle East and North Africa Morocco P110833 Rural Roads II 2011 Morocco 7351 MOR Rural Potable Water and Sanitation 2011 Djibouti Morocco Tunisia Yemen, Rep. of

Consolidated Strategic Action Plans to Implement Follow-up Activities Addressing Findings of PPRs and

IPRs

2009

South Asia Bangladesh Consolidated PPR/IPR Summary contained in FY2011

Report covering 34 projects and 174 contracts

2011

9.7m India Consolidated Report on PPRs covering 24 projects and

1596 contracts

2012

113.84m Note: In some cases (Bangladesh, India, and Yemen) consolidated reports were compiled by the region, in others there were no consolidated reports and the analysis is based on individual PPR completions. PPR = Post-Procurement Review.

APPENDIX 3D POST PROCUREMENT REVIEWS

80

Table C.6. Instances of Noncompliance in PPR/IPR Reports on Yemen (FY09–10)

Coverage: Projects PPR IPR 1 Project in each of the following sectors

FY 09: Public Sector Governance (4 contracts), Water and Sanitation (11 contracts), Transport (4 contracts), Urban Development (2 contracts), Social Protection (11), Rural Development sector ( 2 projects: 8 contracts) FY10: Education (2 contracts), Transport (5 contracts), Environment (8 contracts), Private Sector (2 contracts) Social Protection (16 contracts), Water and Sanitation sector (2 projects; 16 contracts) Total Number of contracts: 89

FY09: Health sector (17 contracts) FY10: Agriculture and Rural Development sector (2 projects; 20 contracts) Public Sector Governance sector (22 contracts) Total Number of contracts: 59

SUMMARY OF PROBLEMATIC FINDINGS A: Relating to the Borrower/Implementing Agency PPRs Inaccurate cost estimates (referential sources of unit prices not reliable); Recurrent overwriting of award decisions by the country’s High Tender Board (HTB) after the Bank NOs were granted; Weak procurement and contract administration capacity in project implementing units (exacerbated by high turnover of procurement staff). IPRs (i) Generalized average procurement capacity and poor procurement filing systems and contract management that in many cases led to extensive time overruns; (ii) insufficiently comprehensive ToRs, Schedule of Requirements and technical specifications; (iii) unauthorized restriction in bidding documents for arithmetical errors exceeding 3% of bid price resulted in disqualification of lowest priced/substantially responsive bidders; (iv) unjustified use of the SSS method and unauthorized use of CQS and SSS in cases where the estimated contract value exceeded the thresholds established in the LA for the use of these methods; (v) lack of procurement-related documentation on file (e.g., proof of advertisement in local newspapers, financial proposals and basis for calculation of consultants’ fees and reimbursable expenses, copies of performance securities when their furnishing was required in the contracts; (vi) changes in the substance of bid (i.e., performance security, delivery period, payment terms, etc.) occurred after the bids’ receipt deadline; (vii) unguaranteed advance payments over 10%; (viii) amount of performance security not increased as required by the performance security provision consequently with an increase of the contract price for additional works; (ix) performance security improperly obtained in Yemeni Rials instead of the currency of the contract (US$) or a freely convertible currency; (x) required publication of evaluation results and contract award was not made; (xi) fraud and corruption and audit clauses in bidding/contract documents were not in compliance with the agreed legal provisions; (xii) modifications for aggregate increase by more than 15% of the original contract amount made without seeking the Bank’s NO;

(xvii) A case of non-compliance with GL’s requirement to specify liquidated damages for delayed completion/delivery; (xviii) a case of award made for a lower-priced option of the lowest evaluated substantially responsive bidder, not in accordance with the BER and the award recommendation accorded with the Bank; (xix)A case where items re-advertised after cancellation due to change in requirements on the bids’ submission deadline, differed from those in the corresponding bidding documents; a case where currency of payment specified in the contract was inconsistent with the contract price currency specified by the successful bidder in its bid; (xx) A case where procured items were long stored in a government location instead of distributed to the intended recipients; (xxi) A case where, during physical inspection, procured items could not be found at the intended destination (and deficiencies noted with regard to the location of other items procured which caused them to rust; (xxiii) In the particular case of one project implementing unit, prolonged lack of procurement staff and new management unaware of prior procurement activity; (xxii) poor to non-existent “after delivery” control of procured assets; (xxiii) expired Bank guarantees not returned to bidders; (xxiv) Required geographical spread of short-listed firms was not complied; (xxv) Notification of award predated the Bank’s NO; (xxvi) Unauthorized suppression of provision on qualification of the bidder as required alleging objective to develop local industry and due to low average cost of sub-projects; (xxvii) A case where, though the bidding documents were accorded NO by the Bank, payment terms were unacceptably set differently for “goods supplied from the purchaser’s country” from those for “goods supplied from abroad” (allegedly to avoid the use of LC in the case of local suppliers by increasing slightly the advance payment); (xxviii) Obligations not originally included in the RFQ (that is, furnishing of performance security in the amount of 10% of the contract price), were imposed to successful suppliers at the time the POs/contracts were

81

(xiii) physical inspection revealed that payments were made against non-executed items of works and others with poor workmanship or not in accordance with the technical specifications; (xiv) non-compliance with advertisement requirements and minimum 30-day period allowed for preparation/ submission of bids; (xv) a case of unauthorized use of the “shopping” method for works exceeding the threshold established in the LA; (xvi) a case where requirement to reduce the performance guarantee after delivery/acceptance of goods was not complied and 2% of the contract price was improperly retained as cash deposit for the duration of the warranty period;

issued; (xxix) Opening of proposals occurred a week after the deadline for receipt of proposals; (xxx) Required prior reviews by the Bank not sought by Project Management Unit staff also implementing another Bank-financed project alleging not being aware that project-specific provisions applied; (xxxi) Details relating to “liquidated damages clause” not properly specified (four kinds of deficiencies found); delayed delivery of goods or completion of works or services; (xxxii) A case where liquidated damages were not deducted from the contract price payable on account of penalties despite a liquidated damages provision in the contract; (xxxiii) A case where changes to the scope of works to reduce the contract amount were made after the Bank had accorded NO; and (xxxiv) A case where extension of bid validity was requested from a limited number of bidders instead of from all bidders as required in the GLs.

Source: Strategic Action Plan to Implement Follow-Up Activities Addressing Findings of FY09-FY10 PPRs and IPRs in the Middle East and North Africa Region.

82

Analysis of P-RAMS

DATA DESCRIPTION

The data provided for analysis covers the period since the implementation of Procurement Risk Assessment Management System (P-RAMS) in 2010 and ending in FY12 and consists of 794 eligible projects, of which 607 had at least one P-RAMS. Including multiple P-RAMS sequences, the number of eligible and actual P-RAMS rises to 917.

COMPLIANCE AND COMPLETION

For the purpose of this analysis a completed P-RAMS is one that provides information about both project and residual risk. There were 61 projects that listed their P-RAMS status as “in progress” but did not provide any information about project risk; therefore these observations were not counted as completed. Compliance rates were calculated by project, looking at the ratio of projects with at least one P-RAMS completed against the number of eligible projects (investment lending projects that were in the Project Concept Note phase on or after July 19, 2010).

RISK MANAGEMENT AND DYNAMIC RISK MANAGEMENT Overall procurement risk is determined after the analysis of procurement risk by taking into account the 11 risk factors contained in the Procurement Risk Assessment Questionnaire—as well as other possible project risk factors (political climate, capacity, and so forth). After project risk is identified, the person completing the P-RAMS designs a risk mitigation plan with steps to address the risks defined in the previous stage, and enters each mitigation measure into the Mitigation Measures Action Plan in the template. The risk that may remain after the risk mitigation measures are carried out is called residual risk.

First and Last P-RAMS: “Bookend Analysis”

To analyze the risk management component of the P-RAMS, the sample of P-RAMS was tabulated based on initial project procurement risk. Of the 607 P-RAMS completed, 66 percent were categorized as having either high or substantial project procurement risk. Only 7 percent of P-RAMS exercises reported low initial project procurement risk. Risk was examined at the start and end of the available P-RAMs sequences for the project, comparing project risk to residual risk. For projects with

83

two or more P-RAMS exercises the project risk for the first P-RAMS was compared to the residual risk for the last P-RAMS, that is, using a “bookends” approach.

Table C.7. Residual Risk by Project Risk for All Projects with P-RAMS

PROJECT RISK One Sequence

(N = 393) Two Sequences

(N = 65)

Three Sequences (N =

13)

Four or Five Sequences

(N = 11) ASSESSED RISK : High Freq. % Freq. % Freq. % Freq. % 134 21 2 4 RESIDUAL RISK High 18 13% 5 24% 0 0% 0 0% Substantial 91 68% 14 67% 2 100% 4 100% Moderate 25 19% 2 10% 0 0% 0 0% Low 0 0% 0 0% 0 0% 0 0% ASSESSED RISK: Substantial Freq. % Freq. % Freq. % Freq. % 124 23 7 3 RESIDUAL RISK High 1 1% 0 0% 0 0% 0 0% Substantial 34 27% 5 22% 3 43% 3 100% Moderate 86 69% 18 78% 4 57% 0 0% Low 3 2% 0 0% 0 0% 0 0% ASSESSED RISK: Moderate Freq. % Freq. % Freq. % Freq. % 116 16 3 2 RESIDUAL RISK High 0 0% 0 0% 0 0% 0 0% Substantial 1 1% 1 6% 0 0% 0 0% Moderate 87 75% 9 56% 2 67% 2 100% Low 28 24% 6 38% 1 33% 0 0% ASSESSED RISK: Low Freq. % Freq. % Freq. % Freq. % 19 5 1 2 RESIDUAL RISK High 0 0% 0 0% 0 0% 0 0% Substantial 0 0% 1 20% 0 0% 0 0% Moderate 1 5% 0 0% 0 0% 0 0% Low 18 95% 4 80% 1 100% 2 100% Source: IEG calculations from P-RAMS database provided by OPSOR. Note: These figures are for the “bookends” of risk, the initial risk rating for the first P-RAMS and the residual risk rating for the last P-RAMS. (n = 482).

The main findings are:

• The exercise tends to show greater difference in risk levels (as measured by the difference in initial and residual risk) for projects with high and substantial initial project procurement risk. Seventy-eight percent of projects (249 out of 318) with an initial assessed risk of high or substantial had a difference in residual risk rating of at least one risk category.

84

• Most projects with moderate risk tended to not see a difference in project residual risk, although 25 percent of projects (35 out of 137) with an initial risk rating of moderate had a residual risk rating of low.

• Projects with a low initial risk rating remained with low residual risk rating 93 percent of the time (though two saw an increased residual risk rating).

• The data does not show an increased decline in risk over multiple P-RAMS. For example, zero projects with four or five P-RAMS that had an initial risk level of high saw a reduction to moderate or low residual risk. This finding is replicated for projects that have an assessed risk level of substantial as well.

Examination of Each P-RAMs in a Series: “Sequence Analysis”

The last unexpected finding raises some questions as to the value added of the dynamic P-RAMS exercise. A closer examination was therefore undertaken looking at the detailed sequencing of risk from the first P-RAMS to the last P-RAMS for each given project where two or more sequences existed. Results are indicated in Table C.8.

Table C.8. P-RAMS Risk Ratings for a Sample of Projects with Two or More P-RAMS

1st P-RAMS 2nd P-RAMS 3rd P-RAMS 4th P-RAMS

ProjectID ProjectRisk ResidualRisk ProjectRisk ResidualRisk ProjectRisk ResidualRisk ProjectRisk ResidualRisk

P094360 High Substantial High Substantial High Substantial High Substantial

P123923 High Substantial High Substantial High Substantial Substantial Moderate

P100530 Substantial Moderate Substantial Moderate Substantial Moderate Substantial Moderate

P120349 Moderate Moderate High High Moderate Moderate Moderate Moderate

P122319 Low Low Low Low Low Low Low Low

P125689 High Substantial High Substantial High Substantial P123201 Substantial Moderate Substantial Substantial Substantial Substantial P126537 Moderate Low Moderate Low Substantial Moderate P122492 High Substantial High Substantial P122008 Moderate Moderate Moderate Moderate

Source: IEG tabulation from P-RAMS database provided by OPSOR. Note: Selected by IEG to show the variation in patterns between project and residual risk for projects with multiple sequences.

• Only one project showed the expected pattern of consistently reduced

residual risk with additional P-RAMS. • The most common pattern seen was the same project risk rating through

every P-RAMS in the sequence as well as the same residual risk rating through every P-RAMS.

• The data do not show an increase in reduction of risk over multiple P-RAMS. For example, zero projects with four or five P-RAMS that had an initial risk level of high or substantial saw a reduction to moderate or low residual risk.

85

Of the 65 projects that had two P-RAMS, 21 showed no difference between initial project procurement risk and residual risk in any project phase. Four of these 21 projects with no change in risk were categorized as low risk projects; it can be assumed that the risk mitigation plan could not have changed the residual risk rating. Of the remaining 44 projects that did see some change in risk level between initial project procurement risk and residual risk, 39 projects had identical P-RAMS risk ratings in both sequences with the residual risk always being noted as lower than the project risk (for example, an initial risk rating of high and residual rating of substantial, for each project phase).

At first glance it seems counterintuitive that a project would have a residual risk rating of moderate in one sequence followed by a project risk rating of substantial (more than moderate) in the next sequence. One possible explanation for this is that the risk mitigation plan from the previous phase was not completely implemented and the risk was not reduced as planned. This could also reflect too frequent changes in P-RAMS sequences. Another possibility is that changes in risk might be expected to occur over the course of implementation, and projects with P-RAMS have not been under implementation for any length of time, due to the newness of the tool (see Table C.9, which shows that only a few projects are under supervision). A third possibility is the inertia in terms of changing entries, and perhaps limited scrutiny of P-RAMS sequences. These potential explanations for the two P-RAMS sequence projects could apply to the three-P-RAMS projects, too.

Table C.9. Sequence and Phase Tabulations Sequence 1 2 3 4 5 Total

Phase Freq % Freq % Freq % Freq % Freq % Freq % Project Concept Note 215 40 0 0 0 0 0 0 0 0 215 32 Quality Enhancement Review

67 12 42 47 0 0 0 0 0 0 109 16

Decision Meeting 97 18 20 22 13 52 0 0 0 0 130 19 Negotiations 60 11 17 19 8 32 10 100 0 0 95 14

Supervision 104 19 10 11 4 16 0 0 1 1 119 18

Total 543 81 89 13 25 4 10 1 1 0 668 Source: IEG calculations from P-RAMS database One issue with analyzing the “spread” of risk rating data in the P-RAMS is that all of the measures are subjective, assigned by the procurement specialist or task team leader on the project. A complementary analysis looking at whether the risk mitigation measures were carried out and effective would be useful in addition to examining subjective assessments of risk provided in the P-RAMS. Table C.9 shows the distribution of P-RAMS through the project cycle, as of FY12.

86

CONCLUSION

Overall it remains unclear whether the P-RAMS tool is performing as was intended. Though the program has yet to achieve the desired 100 percent compliance, it seems that goal will be achieved in the very near future. An analysis of the most important element of the P-RAMS – the risk measurement and mitigation shows mixed results. If residual risk over the cycle of the project is expected to decline, the analysis shows it has yet to occur. If however, risk is basically static and the goal of the P-RAMS is to mitigate risks where possible then the exercise is more successful (see the “bookend” versus “sequence” analysis in preceding pages). The usefulness of P-RAMS ultimately will depend on how much its findings are used for risk analysis. Further complementary qualitative analysis is required before a more complete conclusion can be drawn.

Box C.6. INT Guidance to Staff on Procurement Risk

INT’s draft paper on potential red flag issues, mostly alluding to procurement, offers a number of suggestions to operations staff:

87

Establish an integrity due diligence checklist for implementing agencies: National procurement systems could be strengthened by adding integrity checklists as part of Bid Evaluation Reports. OPCS could set standards, and the regions could offer training to local implementing agency staff in operating the checklists.

Define Bank Due Diligence in Prior Reviews: Especially as management is now suggesting that transaction-level fraud and corruption due diligence be de-emphasized, it becomes important that the general principles and what to look for in the prior review of the most risky transactions be more clearly defined in a revised OP/BP11.00.

Incorporate a Red Flag Checklist into Post-Procurement Reviews: The suggestion is to amend the 2002 Office Memorandum which establishes the basis for handling integrity risks in Post-Procurement Reviews to change the basis for the review from contract size to contract risk rating; provide guidance on detection of high-risk transactions (for example, single bidder, close to threshold amounts, existence of procurement complaint); attaching a red flag checklist; ensuring that where consultants are used to conduct Post-Procurement Reviews they are trained in red flag methodology.

Disclose Past Performance in PADs and ICRs: The suggestion is that particularly in the case of successor projects implemented by the same project implementing unit, the procurement performance of contracted companies be reflected in the ICRs of one project and the PAD of the ensuing project. A scorecard could be developed containing such items as: actual cost compared to cost estimates at bidding; percentage of bids with only one or two bidders (not clear if this refers to the bidders or the project implementing unit); number of transactions with overruns above 6 months; number transactions with procurement complaints.

Avoid Conflicts of Interest in Recruiting and Deploying STCs: Project staff should avoid giving unchecked procurement discretion to STCs who may or may not be in a potential COI situation.

.

89

Appendix D. Procurement Tracking Systems, Time Analysis, and Africa Region Review Systems

BANK-WIDE INFORMATION – EARLIER VERSIONS OF FORM 384

Earlier versions of the Form 384 template indicate a stronger emphasis on collecting procurement process data, including a number of dates regarding the bidding process as well as contract co-financing information. Before 2001 the Operations Procurement – Form 384 selection screen provided data entry tabs such as Activity Plan/Monitor, Project Plan/Monitor, Evaluation Report, Contract Evaluation, Form 384 and Project Information. A big part of the data populating the form was captured from information entered via these tabs earlier in the procurement process. The actual Form 384 data entry was divided into two separate sections for goods and works and for consultant services, each providing sections for entry of basic data, supplier/firm data and currency data. A number of contract dates were collected such as bid opening/proposal submission, evaluation no objection, contract received, contract signature and Form 384 date. In addition, fields for delivery completion target/actual dates, completion percentage, exchange rate and co-financing information were provided. Users were asked to enter the country codes of the next three lowest bidders/firms considered for contract award in addition to supplier/firm data of the winning bid.

The 2001 revision of Form 384 came embedded in a general overhaul of the procurement information system aimed at reducing the amount of data entry in the system. Operations Procurement – Form 384 selections screens were trimmed down to Project Information, Form 384, Post Review and Activity Planning, with some of the tabs capturing extended information to pick up data entered elsewhere prior to the system revision. Cofinancing data were moved to their own tab and the following fields removed: delivery completion target and actual dates, completion percentage, completion as of date and one of three country codes to identify next lowest bidders. While aiming for simplification of data entry, the incentive for procurement data collection seemed to have remained a priority in 2001 as additional fields were opened for data entry: implementing agency, procurement method, borrower bid reference number, bid opening date, domestic preference indicator, award affected by preference indicator, bid evaluation report no objection date, prequalification of bidders required indicator, fixed price contract indicator, two stage bidding indicator, bid number, bidding description and contract description. In the

90

consultant services section no fields were removed from data entry but new fields were added such as the check box for misprocurement and allow for contract signature date before no objection date. Overall it seems that while a few key variables have been dropped from bank wide data collection in the 2001 revision, other indicators have been added reflecting a possible shift in specific monitoring needs but not a general reduction of data collection.

This seems to have changed significantly with the 2009 revision when Form 384 became a single data entry screen, delinked from procurement bids/activities information and simplified with a broad stroke including the removal of all but key data entry fields necessary for basic procurement reporting. Form 384 in its latest version is divided into four sections: basic data, contract price, supplier firm/individual details, and contract comments. No objection date and contract signature date are the only contract dates collected in addition to basic information like procurement method and sector allocation.

It is no longer necessary to enter the actual currency payment terms with the contract information. While the form no longer requires bidding activity other than the winning firm or individual, the most noticeable element added is the integrated anti-money laundering check which must be executed in order to complete Form 384. On entry of data the new Form 384 provides a list of suppliers that have already been cleared for the respective project and, if the supplier is not listed, the anti-money laundering check process is activated. Discussions with Bank’s procurement personal suggest that the 2009 revisions and simplifications of the Form 384 were in line with a general move away from extensive control on the contract accounting side within the Bank. However, recent Bank efforts indicate an increased interest to put in place an integrated bank-wide procurement information system and invest in more comprehensive data collection.

REGIONAL PROCUREMENT TRACKING SYSTEMS – NATURE OF REPORTS PRODUCED

Latin America and the Caribbean: SEPA—System Reports and Borrower Interaction34

As a management information system SEPA allows users to create reports, approve procurement plans and inform the parties concerned about the status of execution thereof. The system provides a platform for interaction between the task team leader and the implementing agency, while at the same time providing an audit trail of the process. The borrower enters and updates the procurement plan according to the loan or grant agreement, the Bank is then prompted by a system-generated notification to furnish its no-objection or provide comments.35 Once approved, the procurement plan is automatically updated on the SEPA website.

91

SEPA helps both the Bank and its borrowers to ensure that detailed procurement plans are prepared, and to identify variations between planned and actual procurement activities. Upload of information into SEPA usually starts with the first procurement plan of a project that covers the first 18 months.36 According to the Bank’s guidelines, procurement plans are to be updated by the Borrower every year. Contracts previously awarded and those to be procured in the following twelve months are required to be included (World Bank 2011e). SEPA staff indicated that most executing agencies update their procurement plan in SEPA on average once per month. Procurement plans, updates and modifications are subject to the Bank’s prior review and no objection before they are posted and have to be published on the Bank’s external website. Procurement plans approved in SEPA are automatically published on the SEPA public website. They have to, however, be manually migrated to show up in the public documents on the Bank’s external website (http://www.worldbank.org), which SEPA staff do on a regular basis.

Data in SEPA are populated by the borrower. Some date fields, such as estimated dates for procurement steps, and estimated amounts, are mandatory for procurement plan approval while others, including actual dates and amounts, hinge on the readiness of the borrower to enter additional information. Actual dates of contract execution do not require new approval by the Bank and can be entered by the borrower at any time.

Africa: The Procurement Cycle Tracking System—PROCYS

The principal function of PROCYS has been that of a management information system that measures responsiveness of participants in the procurement process, and monitors aggregate trends in countries and sectors to identify bottlenecks. Unlike SEPA it channels communications including their attachments through the system recording dates automatically. It is now partially integrated with other Bank systems, in the sense that the filing of no objection notices and related documents into the Bank system is also automated. It generates several managerial reports, including monthly no-objections (by country/by stage/average time taken); quarterly reports on RPM/OPRC No-Objections, status reports on all RPM/OPRC cases, and reports on average time taken per project. Duration of process time is tracked starting from initiation, i.e. when the first request arrives by email to PROCYS, to approval, that is, issuance of no objection.

Box D.1. PROCYS Workflow The process is initiated when the borrower sends a request for procurement review to PROCYS via email; the Bank task team receives an email notification with a link to the request in the system. The task team leader/team reviews and where necessary submits it to the appropriate level of

92

Middle East and North Africa: The Procurement Portfolio Dashboard

The procurement portfolio dashboard is available to Bank staff for download from the region’s intranet site in the form of Excel spreadsheets, available from October 2008. Database macros allow users to produce management reports filtered for information for the entire region, for a selected country department, country, or procurement staff.37 A core element of the dashboard is information on project-level procurement activities planned and executed by procurement specialists, such as tracking compliance with scheduled procurement related activities, including procurement post reviews, procurement plan updates, P-RAMS status and complaints. Project-level data further include committed, disbursed, and cancelled project amounts, procurement method and prior review thresholds, procurement risk ratings, and task team leader and procurement specialist names. Data related to specific contracts or procurement stages are not collected.

A main element of the dashboard is information on project level procurement activities planned and executed by procurement specialists, such as tracking compliance with scheduled procurement related activities, including procurement post reviews, procurement plan updates, P-RAMS status, and complaints. Project-level data further include committed, disbursed and cancelled project amounts, procurement method and prior review thresholds, procurement risk ratings, task team leader and procurement specialist names, among others. Data related to specific contracts or procurement stages are not collected. However, the summary reports on the dashboard are not presented in a way easily accessible for interested users, as they lack explanatory information such as indication of the time period covered and general legend.

SUMMARY

The preceding brief summary illustrates that at present from the management information system point of view it is not possible for central procurement staff to track procurement processes Bank-wide, in terms of parameters relating to efficiency of implementation. Only Form 384 is uniform Bank-wide, but it provides a very

procurement specialist. The comments or no objection are then communicated to the borrower by the task team. Both Bank staff and clients can retrieve project information via the web-based system. PROCYS stores communications with a date and time stamp. However, the unique transaction request number cannot be linked to contract identifiers used in other Bank systems tracking contract data, such as the Bank-wide Form-384. At the time a no-objection is issued, for any stage of the procurement process, all communications including attachments are filed under the project number.

Source: IEG.

93

limited amount of information, on a subset of contracts, and Bank and borrower contributions are not separated. Moreover its use as information of contract level disbursement has been reduced. Form 384 does not provide a platform that is accessible to borrowers or any persons external to the Bank, and its internal use is not management friendly as it does not process any automated reports.

Other regional systems each have their aims and objectives, and reflecting these, advantages and disadvantages. Although the procurement anchor is aware of these difficulties, it has apparently been difficult to develop a system that is integrated with the Bank’s other platforms.

Table D.1. Comparison of Procurement Monitoring Systems by Region

Bank-wide: Form 384

LCR / ECA: Procurement Plan Execution System SEPA

AFRICA: Procurement Cycle Tracking System PROCYS

MENA: Procurement Portfolio Dashboard

Unit of Entry TTL or certified project team member

Borrower enters and updates Procurement Plan for the project

Database as clearinghouse for communications between the borrower and the bank on procurement processes, initiated via email by borrower

Updated monthly by staff in headquarters

Individual contracts subject to prior review

Monitor Procurement Plan execution

Individual contracts subject to prior review

Centralize information based on information in Bank systems and information on projects by team

Contract award information on contracts above the prior review threshold

Approve Procurement Plan through the system; provide public access to basic information

Track response times of Clients, Task Team, Procurement Specialist

GENERAL PROJECT LEVEL DATA Country yes yes yes yes Loan / Credit No yes yes yes Project ID yes yes yes yes Project Status (Pipeline /Active / Closed) yes yes

Project Amount

Total Committed (IBRD + Grants)

Loan Amount/ Credit/ Grants AND Local counterpart amount recorded; including origin currency and exchange rate

Total committed and Grants

Disbursed Disbursed in FY and Total

Undisb. Total undisbursed balance

Cancelled Total cancelled

Aggregated Contract Value Value of PPR Contracts Reviewed

Key Project Dates

Loan Effective Date PCD Review Date

Project Original AND Actual Closing Date Appraisal Date

Project actual closing date Approval, Effective and Closing Date

94

Bank-wide: Form 384




Estimated time per procurement stage P-RAMS Date

TTL at least one and up to three TTL indicated TTL name per project TTL name per Project

Procurement Specialist up to three PS indicated PS name per project Primary and Secondary PS

Agreement Type (IBRD, IDA, other) yes

Lending Instrument (DPL, P4R, IL, other) yes

Sector Information Major Sector and Sector Board Major Sector and Sub-Sector General sector code per

project

Implementing Agencies Borrower, Executing Agency, Sub-Executing Agency

Number of Implementing Agencies

PROCUREMENT PROJECT LEVEL DATA

Project Level Prior Review Threshold

Threshold amounts set by the Bank AND by the Borrower

Amounts and Location of Prior Review Threshold (Proc Plan or Legal Agreement)

Project Level Procurement Method Threshold Threshold amounts for use

with each procurement method

Amounts and Location of Review Threshold (Proc Plan or Legal Agreement)

Procurement Plan Status

Procurement Plan entered and updated in SEPA by executing agency; upon approval automatic publication on SEPA external website; historic procurement plans available

Number for Procurement Plans status current, outdated and not required, date of current and due date next Procurement Plan

Project Procurement Risk Rating

Proposed ISR Rating, Last ISR Rating from SAP, Procurement Risk Rating, # Projects at Risk

Project P-RAMS Status

#P-RAMS in progress, archived and required; Share of projects per risk categories H, S, M, L and “no info”

Procurement Flag yes

Contracts covered only prior review contracts all contracts in Procurement Plan

only contracts marked as prior review

all contracts in Procurement Plan

Project Level Procurement Post Reviews (PPR) (scheduled / completed)

indicates if PPR required for project

PPR completion actual vs planned by country and by person, PPR/IPR action plan implementation, Number of Post Reviews and Supervision Missions

95

Bank-wide: Form 384




PROCUREMENT CONTRACT LEVEL DATA Contract Description yes yes yes yes

Contract ID Contract ID assigned by SAP and Borrower country reference

Contract Reference Number each contract transaction is assigned a request number

Individual Contract Value Total contract amount and amount used

Estimated contract value and actual contract amount and payments

Estimated contract value

Procurement Method (ICB, NCB; for consulting QCBS, QBS,FBS,LCS,CQS, SSS)

yes yes yes

Procurement Category (Goods, Civil Works, Cons. Services)

yes yes yes

Clearance Level (PS, RPM, OPRC) yes

Bidders Nr of bids received and firms considered, country of nationality of winning bid, bidder eligibility status (Anti-Money Laundering, list of debarred firms, etc.)

Awarded: name, country of nationality, bid amount/total amount paid; Shortlisted: name, country of nationality & amount offer

Key Dates and Elapsed Times Contract award date Borrower issue of specific

procurement notice Date of first action

Signature of contract Borrower first submission to Bank of draft bid (preQ) documents*

Date of most recent action

Form-384 sent date Bank final no objection to draft bid (PreQ) documents*

Borrower delay (Average time taken by borrower beyond normal processing time)

Borrower issue of Bid (PreQ) documents*

Task team time (average time taken by task team to review steps )

Bid Opening* Procurement time (average time taken by PS/RPM to review steps)

Borrower submission to Bank of Bid evaluation report*

Average Response Time for No Objection

Bank No-Objection to Bid Evaluation Report*

Number of Stages Given No Objection

Signature of contract*

idle time (duration between today's date - date of last action, usually defined if pending with borrower between two stages)

Contract Completion* Total time client

*stages vary by procurement method and category

Total time with TTL / PS / Client

Estimated and actual contract dates can be entered

Average time with TTL / PS / Client

Total days of process between Bid Opening and Signing of Contract

# iterations (Number interactions between TTL and borrower / TTL and PS/RPM)

96

Bank-wide: Form 384




Management type reports

Procurement Plan monitoring on contract level for selected project

Average Response Time for No Objected Cases

Reports for the entire region, by country, by CMU, by person (PS)

Number and amount of ex post contracts initiated in specific time period by category for selected project

Number of stages given No Objection

Number and Amounts of supervision, lending closed and total projects

Report of Management Processes with elapsed times between stages on contract level for selected country

Response times by sector for selected country

Post Review Schedule (Planned and Actual)

Portfolio Review Report with aggregate amounts for processing stage by project, country and province

Support Effort Hours per case Cumulative PPR Progress

Number of contracts by category, method and stage (expected, in process, completed, canceled) for selected country and project

Monthly reports on No-Objections (by country/by stage/average time taken)

Complaints: Number of minimum, maximum, average days pending for status pending and closed

Quarterly reports on RPM/OPRC No-Objections

General Procurement Plan report indicating total number of projects and procurement plans by country for selected project status (in preparation, active, closed) and year

PROCYS also provides an external website for clients for real-time status of all their requests sent to the Bank

Public reports

Publication of Procurement Plans on SEPA public website

Aggregated Reports of contract values by procurement method/category for selected values (country, project, sector, province)

97

Procurement Efficiency at the World Bank – Elapsed Time Analysis

DATA COLLECTION AND SAMPLE CHARACTERISTICS

Information on processing times of procurement contracts in the Bank is not available in any comprehensive centralized format that can be used by Bank management to track the procurement process for all regions or contracts. Data are highly decentralized and typically maintained by individual country procurement offices. Although procurement checklists are required to be maintained, which would serve as summary sheets, and could deliver dates on processing steps, these are often not available. IEG therefore conducted its own data collection for this analysis, linked to its field visits, and based on a template tracing critical dates in the “no-objection” process.38 Detailed contract processing data at the country/contract level are in some cases maintained by the procurement coordinator’s offices and sometimes by sector units. Each regional office has its own format in terms of specific fields of information tracked, date conventions and, clearly, currency units.

To construct a representative stratified sample IEG suggested a specific list of prior review contracts to each field visit country office, by procurement method and category, by size, and across a sample of fiscal years from 2007 to the present. However, information received is concentrated in 2010–12 with the year 2011 representing 32 percent of all observations in the sample. The response rate from country offices was highly variable in quantity and quality, ranging from 107 contracts received from Azerbaijan to four from Tanzania.

Often, not all date fields were filled. Of 502 contracts received, only 201 provided the requested information on all date fields from Issue of the Specific Procurement Notice to Contract Signature. 39 By far the most frequently reported dates were the no objection to the bid evaluation report and the contract signature which are also provided to the Bank-wide Form 384. In regions where a central data repository for the procurement process has not been established, staff in the field indicated that dates for procurement steps prior to the no objection to the bid evaluation report have to be extracted from procurement documents and communications with country clients. It appears that information on all steps in the procurement process is not readily available in all countries. Clearly this affected the scope and quality of analysis that could be undertaken, as well as reliability of results obtained and will have to be a caveat to their interpretation.

The request for contract sample data sent to the field visit countries was separated by key processing dates for Goods and Works (ICB, ICB with preQ; NCB, NCB with Pre-Q), Goods and Works (ICB/NCB Two Stage) and large values of consultant

98

services (quality- and cost-based selection, quality-based selection, Fixed Budget, least cost selection, consultant qualifications section, single source selection). For purposes of this analysis the respective dates for each procurement step were merged across ICB/NCB, ICB/NCB two-stage and consultant services selection allowing for global data set as well as sub set analysis. Due to a low response rate for information on contracts involving ICB/NCB two-stage and pre-qualification (evaluation reports), data provided for these groups was integrated with the overall ICB/NCB data. For all contracts dates for final Bid Evaluation Reports were used to construct time intervals. 40 Contract values were converted to dollars using monthly average exchange rates.41 The fiscal year of the contract was determined based on the date of the Bank’s No Objection to the Bid Evaluation Report.

MEASURES OF AVERAGE ELAPSED TIMES

IEG’s analysis focuses primarily on two overall elapsed times: borrower issue of specific procurement notice to contract signature and first submission to the Bank of draft bid (preQ) documents to the Bank’s no objection to the bid evaluation report. IEG also attempts to selectively review two intermediate steps in the procurement process, to the extent that data permit: first submission to the Bank of draft bid (preQ) documents to the Bank’s final no objection to the draft bid (PreQ) documents and borrower submission to bank of bid evaluation report to the Bank’s no objection to the bid evaluation report. Because of data limitations, results for different intervals may refer to different subsets in the data set.

SELECT FINDINGS – AVERAGE TIME

The average days between the borrower’s first submission of the draft bidding documents to the Bank’s final no objection to bid evaluation report of the contract was 253 days. However, the variable displayed a high dispersion from the average, with a standard deviation of 160. Though 50 percent of the contracts included in this analysis go from the submission of the bidding documents to the No Objection to the bid evaluation report in less than 208 days, IEG’s analysis indicates the existence of contracts with significant longer duration, for two contracts even more than 900 days, driving up average processing times. Figure D.1 shows the distribution of data points for borrower issue of specific procurement notice to contract signature and first submission to the Bank of draft bid (preQ) documents to the Bank’s no objection to the bid evaluation report compared to the same intervals without those contracts that make for the highest five percent of elapsed times.

99

Figure D.1. Elapsed Time Distributions of Procurement Intervals

0 500 1,000 1,500 2,000

All Observations

0 200 400 600 800

Excluding upper 5 percent

Submission BD to Bank-No Objection BER Issue Specific Procurement Notice-Signature Contract

Source: IEG analysis of sample contract data. Notes: Two intervals are examined in each box: (1) Borrower Issue of Specific Procurement Notice to Contract Signature and (2) First Submission to the Bank of Draft Bid (preQ) Documents to the Bank’s No Objection to the Bid Evaluation Report The vertical line in each box denotes the median; horizontal lines denote “adjacent values.” The offset of the median towards the left and the ‘long tail’ of the distribution to the right describe the skewed distribution. Dots indicate outliers. More than a quarter of the processing time from submission of bidding documents until contract award is due to the preparation of the bid evaluation report after bid opening. In the case of consultant services more than 40 percent of processing time is dedicated to this step in the procurement process, likely because for the quality- and cost-based selection procurement method two documents need to be prepared by the borrower; an evaluation of technical proposals and, after the subsequent opening of financial proposals, a combined evaluation report. Both reports require a no objection by the Bank. This remains true even after excluding those contracts that bring in the highest five percent of elapsed times per category. Results in terms of processing times by category, method, and sector are given in Figure D.2.

100

Figure D.2. Average Elapsed Times in Procurement Steps from Submission of Bidding Documents by Borrower to Contract Signature by Category, Method, and Sector (days)

Source: IEG analysis of sample contract data.

SELECT FINDINGS: VARIATION IN ELAPSED TIMES

Comparing the arithmetic mean and the mode of some of the distributions, the former is typically higher, due to a proportion of contracts that are more time consuming. A key finding is that the procurement process from the time the borrower issues the specific procurement notice to signature of the contract takes more time, on average, for procurement method quality- and cost-based selection used for the selection of consultant services, as compared to ICB and NCB. On average, the processing times to procure consultant services are more than 100

101

percent greater than for contracts awarded using NCB. And differences are not merely that of the average time taken, but also in the number of outliers.

Figure D. 3. Average Elapsed Days in Procurement Steps from Submission of Bidding Documents by Borrower to Contract Signature, by Method (days)

Source: IEG analysis of sample contract data. Note: With controls for outliers. The distribution of elapsed times for the intermediate procurement step from submission of draft bid documents to the bank’s no objection to the final bid evaluation report shows the existence of extreme values for all procurement methods, with especially high values for ICB and quality- and cost-based selection contracts (Figure D.3). Consultant services and ICB contracts differ only by a few days in average processing time taken for this interval. The contracts that contribute the upper five percent of processing time for consultant services selected by the

102

quality- and cost-based selection method take more than three times longer than the bottom 50 percent of contracts.

REGRESSION ANALYSIS AND RESULTS

Dependent Variable: Elapsed Time Interval (1) Issue of the Specific Procurement Notice to Contract Signature

IEG undertook a simple regression analysis to obtain possible explanations for factors affecting elapsed times in procurement, considering alternative dependent variables in the regression specification. A first interval analyzed is the time taken from the Issue of the Specific Procurement Notice to Contract Signature. Explanatory variables included contract attributes: (contract value, procurement method, category of good or service being procured and major sector); control variables included country specific variables (gross domestic product, poverty rate). Results indicate a statistically significant positive relationship between contract value and processing time, robust to the introduction of controls (Table D.2). Coefficients denote marginal effects, and results suggest (specification 5) that for every $10 million increase in contract value, the number of days for processing increases by 14.6, independent of all other contract attributes and across all countries and regional offices. Results also indicate significantly lower elapsed times for NCB contracts compared to ICB.

Dependent Variable: Elapsed Time interval (2) – Submission of Draft Bid Documents to No Objection to the Bid Evaluation Report

Data on this time interval were available for 213 contracts.42 The positive relationship between contract value and elapsed time remains statistically significant, and robust to the addition of controls (Table D.3). Marginal effects are broadly similar - the regression coefficient in Specification (4) implies that for every $10m increase in contract value, the number of days for Bank No Objection increases by 10.6 days, independent of all other contract attributes and across all countries and regional offices. The coefficient for method—NCB—as compared to other methods, is also significant, and the coefficient implies that it takes the World Bank, on average, 65 days less to issue no objection for NCB method contracts as compared to ICB.

The second specification also adds country governance as a control variable, in the form of the CPIA. Results indicate that country level governance factors have a powerful influence on elapsed time, with lower elapsed times for countries with higher CPIA scores.

Table D.2. Analysis of Processing Time from Issuing of Notice to Contract Signature (1) (2) (3) (4) (5) Variables SPN to Contract SPN to Contract SPN to Contract SPN to Contract SPN to Contract

103

Signature Signature Signature Signature Signature Contract 1.42e-06*** 1.55e-06*** 1.52e-06*** 1.47e-06*** 1.46e-06*** Value (1.12e-07) (1.34e-07) (1.41e-07) (9.90e-08) (1.01e-07) Social Sector 3.442 -18.05 -27.32 (51.06) (51.58) (52.15) Infrastructure 95.80** 65.00 64.89 Sector (44.10) (47.22) (52.47) Economic -65.66 -46.46 -49.65 Sector (46.38) (45.66) (47.34) Civil Works -8.500 52.32 48.90 Contracts (56.46) (61.05) (66.33) Consultant 135.1** 112.9* 70.51 Contracts (63.38) (66.28) (75.10) Goods 14.62 37.19 34.09 Contracts (59.56) (60.09) (63.01) NCB Method -112.1*** -113.5*** (39.27) (39.52) Cons. Method 57.45 93.17 (63.32) (66.99) GDP 0.0121* (0.00626) Average -0.725 Poverty Rate (4.488) Constant 243.9*** 198.1*** 185.5*** 196.4*** 158.0* (13.67) (31.30) (60.12) (59.08) (91.97) Observations 272 272 272 272 272 R-squared 0.074 0.191 0.305 0.342 0.346

Source: IEG analysis of sample contract data. Note: Robust standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1. N = 272. All country and regional controls have been included in all specifications. GDP = gross domestic product; SPN = specific procurement notice.

One caveat to the interpretation of these results is that the distribution of residuals in the regressions exhibits deviation from the normal distribution, possibly due to some high leverage data points in the 'long tail' discussed in previous sections. Although the non-normality of residuals does not affect the consistency of an estimate, it may affect its statistical significance.

Table D.3. Analysis of Processing Time from Submission of draft Bid Documents to Bank’s No Objection to Final Bid Evaluation Report (1) (2) (3) (4) (5) Variables Processing Time for

Bank No Objection Processing Time

for Bank No Objection

Processing Time for Bank No Objection



NCB Method -68.19*** -47.92** -57.44** -79.32*** -64.61* (23.05) (21.92) (76.47) (26.42) (27.69) QCBS 28.41* 70.70*** 76.47*** 1.35 21.90 Method (24.00) (21.40) (22.36) (33.38) (34.87)

104

Contract 9.73e-07*** 1.16e-06*** 1.12e-06*** 1.00e-06*** 1.06e-06*** Value (2.34e-07) (2.13e-07) (1.96e-07) (1.75e-07) (1.78e-07) Social Sector -59.92 -43.55 -36.42 (39.03) (40.75) (41.27) Infrastructure -42.50 -36.87 -21.86 Sector (34.65) (36.00) (35.25) Economic -27.70 -44.30 -58.11 Sector (39.97) (42.73) (44.20) Civil Works -69.02* -51.14 Contracts (35.41) (35.93) Goods -99.35*** -76.20* Contracts (37.35) (40.35) GDP Level 0.0352*** (0.00962) Average -1.816 Poverty Rate (2.819) CPIA -335.33*** score (112.82) Constant 234.1*** 286.7*** 292.6*** 366.3*** 1,533*** (13.25) (40.59) (42.34) (54.00) (410.6) Observations 213 213 213 213 213 R-squared 0.153 0.382 0.392 0.406 0.445

Source: IEG analysis of sample contract data. Note: Specifications (2) to (5) include country and regional level controls. Robust standard errors in parentheses, *** p<0.01, ** p<0.05, * p<0.1. GDP = gross domestic product. N = 213.

105

Efficiency of Procurement Management in the Africa Region

Further insights into procurement efficiency are afforded by IEG’s analysis of datasets made available by the Africa Procurement unit. Generated by PROCYS, the system tracks each step in a procurement process that involves interaction between the key players—borrowers, Bank task team leaders and Bank procurement staff—to monitor the responsiveness and efficiency of each party in the process.43

ELAPSED TIMES IN THE PROCUREMENT PROCESS

Each step in a procurement process that involves an exchange between participants in the procurement process is an entry in PROCYS. In 2012, 3,043 requests were logged; a number that has increased over time. This likely reflects the fact that use of PROCYS in terms of country coverage has increased, especially since 2010.44 On average, there are two to three iterations both between client and task team leader, and task team leader and procurement specialist. In terms of numbers of days of elapsed time, in each iteration, the client takes the most time with an average over the 2009–12 period of almost 11 days; while the task team leader and procurement specialist take around 8 days each. There is evidence of increased efficiency in all parties; the number of days per iteration has shrunk considerably for clients, from 14 in 2009 to 5 days in 2012; from 10 to 5 days for task team leaders and from 8 to 7 days by the procurement specialist.

Table D.4. Africa: Numbers of Iterations and Average Response Times, 2009–12 Year Number of

records Average number of iterations Average number of days taken per iteration

Client - TTL TTL – Procurement specialist

Client TTL Procurement specialist

2009 Q1 360 2 2 16 10 9 Q2 404 2 2 14 11 11 Q3 522 2 2 11 8 5 Q4 672 3 2 17 10 6

2010 Q1 655 3 2 12 8 9 Q2 606 3 2 10 10 8 Q3 626 3 3 12 8 6 Q4 653 4 3 13 10 9

2011 Q1 671 4 3 13 8 9 Q2 719 3 3 11 9 6 Q3 661 3 3 15 10 10 Q4 726 3 2 9 8 6

2012 Q1 723 3 2 8 7 8 Q2 898 3 2 6 6 7 Q3 772 2 2 5 5 6 Q4 647 2 1 1 3 5

APPENDIX 4C EFFICIENCY OF PROCUREMENT MANAGEMENT IN THE AFRICA REGION

106

Avg/Total 10,315 3 2 10.3 8.0 7.5 Source: PROCYS database. Note: TTL = task team leader.

Aside from average elapsed time, IEG reviewed the distribution of elapsed time. Since the average numbers of days taken by client, task team leader, and procurement specialist was considerably greater than the median of their distributions, it suggests the presence of outliers that take much longer times per iteration.45 For instance, the average number of days taken by procurement specialist per iteration is seven, but half the records (the median) take less than three days and at the 75th percentile, it requires eight days to process a contract.

Table D.5. Africa: Response Times for Selected Procurement Categories by Clearance Level, 2009–12 Average numbers of days per iteration

Procurement Category Clearance Level PS RPM OPRC

(Number of records) Goods, works, and nonconsulting services 2690 221 40 Consulting services 4333 250 4

Client Goods, works, and nonconsulting services 11 16 8 Consulting services 10 13 10

TTL Goods, works, and nonconsulting services 7 7 13 Consulting services 7 10 5

Procurement specialist Goods, works, and nonconsulting services 7 8 16 Consulting services 6 8 25

Source: Africa procurement office PROCYS database. Note: OPRC =Operational Procurement Review Committee; PS = procurement specialist; RPM = regional procurement manager; TTL = task team leader. Clearance at the OPRC level experienced the largest delays in terms of time taken by Bank staff (task team leader and procurement specialist) with 27 days per iteration. However, clients do not take longer to respond to OPRC, with just 8 days per iteration, in contrast to 16 days on their iterations at the RPM level. There are also clear differences in the numbers of client and task team leader iterations, depending on the clearance level, with an average of five iterations for requests at the OPRC level, four at the RPM level, and three at the procurement specialist level.

To estimate overall Bank performance (as separated from the client), IEG aggregated the time taken by the task team leader and the procurement specialist. Best and worst performers are indicated in Table D.6.

Table D.6. Africa: Procurement—Best and Worst Performers, 2009–12 Country Number

of records

Average number of days taken by client

(per iteration)

Average number of days taken by

TTL (per iteration)

Average number of days taken by

procurement specialist (per

iteration)

Average number of days

taken by TTL +PS (per iteration)

The longest numbers of days Guinea-Bissau 30 9 32 7 39 Burkina Faso 220 19 13 23 36


107

Tanzania 878 14 9 12 21 The least number of days Mauritius 17 15 2 3 5 Gabon 44 9 3 4 7 Mozambique 776 8 6 2 7 Source: Africa procurement office PROCYS database. Note: PS = procurement specialist; TTL = task team leader. During the 2009–12 period, there are no significant differences in response times between procurement categories for any of the three sets of players in the process. In terms of specific products procured, clients spend the most time on “plant design, supply and installation” with 18 days on average, based on 93 requests recorded from 2009 to 2012. For task team leaders, it takes the most time—16 days—to process “Retroactive No-Objections,” based on 23 cases during the period of analysis. For procurement specialists, “output and performance-based road contracts” took the most time to process: 51 days; followed by “information systems,” with 15 days. It should be noted that the products procured did not seem uniformly defined. For example, in some cases procurement of consulting services was indicated to be related to firms or individuals, in other cases it was not specified.

In terms of procurement method, the time taken by clients to respond to ICB-related requests was 15 days on average; compared to 8 days for NCB and 12 for quality- and cost-based selection requests. Response times for these methods are similar for task team leaders and procurement specialists; however, two-stage ICB takes 3 more days, per iteration, for task team leaders to process than for procurement specialists. Along with shopping, two-stage ICB takes the most time to process (11 days) for task team leaders than any other procurement method. For procurement specialists, the methods that take the most time are NCB, with 10 days; followed by ICB and quality- and cost-based selection that take 9 days.46 The procurement methods that entail the shortest processing times are “force account” and “commercial practices.” Although there are few of these cases in our sample, 20 records for “force account” and only 2 for “commercial practices,” the times that clients, task team leaders and procurements specialists take for request processing are lower than the rest. They vary in between zero and four days.

BORROWER DELAYS AND IDLE TIME

As is clear from the table below, borrower delays in between steps in the procurement process are even greater than the actual time taken by all parties to process the different requests. And delays appear to vary by quarter. In 2012, for example, borrower delays in between steps averaged 44 days; while the total time taken to process all requests was 27 days. The 27 days refer to the average time per iteration taken by the borrower (15 days), task team leader (6 days) and by


108

procurement staff (6 days). Borrower delays vary by quarter but they are always above Bank response times, on average, by more than three times.

Table D.7. Africa: Borrower Delays and Bank Response, 2010–12

Year-Quarter

Borrower delay (days)

between iterations

Response times (days) within iterations Borrower

time Task team

time Procurement

staff time Bank

response time

Total response

time 2010 Q1 18.3 9.2 3.8 5.1 8.9 18.1

Q2 38.1 16.1 5.5 4.6 10.1 26.2

Q3 49.6 19.3 9.3 5.9 15.2 34.6

Q4 38.6 12.4 5.7 7.3 13.0 25.4

2011 Q1 53.5 12.8 10.6 6.4 17.0 29.8

Q2 30.9 11.8 6.6 6.5 13.1 25.0

Q3 39.8 14.1 5.8 5.4 11.2 25.2

Q4 39.1 14.8 6.5 5.8 12.3 27.1

2012 Q1 38.1 15.7 6.9 6.1 13.0 28.7

Q2 63.0 17.5 6.9 7.2 14.1 31.6

Q3 38.1 15.1 6.6 6.0 12.6 27.6

Q4 38.4 11.4 4.8 5.8 10.6 22.0 Source: Africa procurement office PROCYS database. Note: Delays recorded in PROCYS are defined as the average time taken by the borrower beyond normal processing time (70 days for consultants and 90 days for goods, works, and services. Three clearance levels were indicated in the database: procurement specialist, RPM, and OPRC. Borrower delay was similar at the RPM and OPRC levels but much lower with clearance at the procurement specialist level.

Table D.8. Africa: Borrower Delays by Clearance Level, 2010–12 Clearance level Number of records Borrower delay (days)

PS 1,026 40 RPM 68 69 OPRC 4 65 Not available 494 40 Total 1,592 42 Source: Africa procurement office PROCYS database. Note: OPRC = Operational Procurement Review Committee; PS = procurement specialist; RPM = regional procurement manager. According to the most recent report from the Africa procurement unit, the countries with the largest borrower delays are Chad (66 days), Cameroon (63 days), and Ethiopia (62 days).47 The largest Bank response times per iteration correspond to South Africa (21 days), Burkina Faso (19 days), and Central African Republic (16 days).


109

Figure D.4. Procurement Processing Delays and Bank Response by Country

0 10 20 30 40 50 60

ChadCameroon

EthiopiaGhana

Ivory CoastSenegalUganda

KenyaTanzaniaRwanda

Eastern AfricaRepublic of Congo

ZambiaAfricaBenin

AngolaLiberia

DRCBurkina Faso

BurundiMozambique

NigeriaTogoMali

MadagascarSudanNiger

MalawiCentral African Republic

LesothoSouth Africa

GambiaCongo

SwazilandMauritania

Cape VerdeGabon

Central AfricaSierra Leone

Sao Tome and PrincipeComoros

Botswana

Procurement Staff

Task Team

Borrower Delay

Source: Africa procurement office database. Note: Delays recorded in PROCYS are defined as the average time taken by the borrower beyond normal processing time (70 days for consultants and 90 days for goods, works, and services).

110

Bibliography Aberdeen Group. 2008. The E-Procurement

Benchmark Report: Driving Year-Over-Year Superior Performance. Boston: Aberdeen Group.

ADB (Asian Development Bank). 2002. “Understanding Public Procurement.” Governance Brief, Asian Development Bank. Manila: ADB.

———. 2007. Environmentally Responsible Procurement—A Reference Guide. Manila: ADB.

AfDB. (African Development Bank). 2011. “Evaluation of Paris Declaration Implementation at the African Development Bank.” Operations Evaluation Department. Tunis: AfDB.

———. 2012. “Determination of Financial Thresholds for Modes of Procurement of Goods, Works and Non-Consulting Services”. African Development Bank Group, Procurement Policy and Fiduciary Services Department. January 2012. Tunis: AfDB.

AfDB, IDB (Inter-American Development Bank), World Bank, and Curtin University of Technology. 2007. International Survey of Procurement. Perth.

Agaba, Edgar, and Nigel Shipman. 2007l “Public Procurement Reform in Developing Countries: The Uganda Experience.” In G. Pig and K.V. Thai (eds.) Advancing Public Procurement: Practices, Innovation, and Knowledge Sharing (Boca Raton, FL: Pr. Academics Press), 373-91.

Al-Azar, Rima. 2011. “A Tale of Two CDDs: Lessons Learned for Mitigating the Risk of Fraud and Corruption in CDD Operations.” PowerPoint presentation, World Bank, Washington, DC.

Andrews, Matt. 2013. The Limits of Institutional Reform in Development: Changing Rules

for Realistic Solutions. Cambridge: Cambridge University Press 128-130.

APEC (Asian Pacific Economic Cooperation). 1999. “Government Procurement Experts Group Non-Binding Principles on Government Procurement”. August 1999. Singapore: APEC.

ASIL (American Society of International Law). 2013. Will the World Trade Organization Finally Tackle Corruption in Public Purchasing? The Revised Agreement on Government Procurement. http//www.asil.org/insights130415.cfm.

Blue, Richard, and John Eriksson. 2011. “Evaluation of the Implementation of the Paris Declaration: United States Government Synthesis Report”. Arlington, VA: The United States Agency for International Development.

Burger, Philippe, and I. Hawkesworth. 2011. How to Attain Value for Money: Comparing PPP and Traditional Infrastructure Public Procurement. OECD Journal on Budgeting, Volume 2011/1. Paris, France: OECD.

Casartelli, G., E. Wolfstetter. 2007. "Selection and Employment of Consultants: the Effectiveness of the World Bank's Policy" Working Paper. Washington, DC: World Bank.

Commonwealth of Australia. 2008. Commonwealth Procurement Guidelines, December 2008. Issued under Regulation 7 of the Financial Management and Accountability Regulations, 1997. Canberra: Commonwealth of Australia.

Debevoise, E.W., and C. Yukins. 2010. “Assessing the World Bank’s Proposed Revision of Its Procurement Guidelines,” The Government Contractor, 52, (21).

Dehn, Jan, Ritva Reinikka, and Jakob Svensson. 2003. “Survey Tools for

111

Assessing Performance in Service Delivery.” In Francois Bourguignon and Luiz Pereira da Silva (eds.), Evaluating the Poverty and Distributional Impact of Economic Policies. New York: Oxford University Press and World Bank.

Dener Cem, Joanna. A Watkins and William. L. Dorotinsky. 2011. “Financial Management Information Systems—25 Years of World Bank Experience on What Works and What Doesn’t.” World Bank, Washington, DC.

De Silva, S. 2000. “Community-based Contracting; A Review of Stakeholder Experience.” Washington DC: World Bank.

DfID (Deparment for International Development). 2011. “Approach to Value for Money.” London: Department for International Development.

European Union, World Bank, and Inter-American Development Bank. 2009. Perú Informe de la gestión de las finanzas públicas Basado en el marco de referencia para la medición del desempeño (PEFA). Lima: European Union, World Bank and Inter-American Development Bank.

Gartner Group. 2012. “Survey Shows Why Projects Fail.” http://www.gartner.com/id=2034616.

GSDRC. 2010. Helpdesk Research Report: Public Financial Management and Frontline Service Delivery. http://www.gsdrc.org/docs/open/HD653.pdf,.

IDB (Inter-American Development Bank). 2010. “Guide for Acceptance of the Use of Country Procurement Systems.” June 2010. Washington DC: IDB.

IAEA (International Atomic Energy Agency). 2011. Agency for Financial Regulations and Rules AM VI/1:

Procurement of Goods and Services, February 16. Mimeo.

IEG (Independent Evaluation Group). 1977. “The Role and Use of Consultants in Bank Group Projects.” Report no. 1824, Washington, DC: World Bank.

———. 1981. “An Interim Report on Procurement Issues in Bank-Financed Projects.” Report no. 3557, Washington, DC: World Bank.

———. 1990. “A Review of Procurement in Policy-Based Lending.” Report no. 8341, Washington, DC: World Bank.

———. 2002. India Country Assistance Evaluation. Washington, DC: World Bank

———. 2005. The Effectiveness of World Bank Support for Community-Based and –Driven Development: An OED Evaluation. Washington, DC: World Bank.

———. 2006a. Debt Relief for the Poorest—An Evaluation Update of the HIPC Initiative. Washington, DC: World Bank.

———. 2006b.Engaging with Fragile States: An Evaluation of World Bank Support for Low-Income Countries Under Stress. Washington, DC: World Bank.

———. 2006c. Hazards of Nature, Risks to Development – An IEG Evaluation of World Bank Assistance for Natural Disasters. Washington, DC: World Bank.

———. 2006d. Small States: Making the Most of Development Assistance – A Synthesis of World Bank Evaluation Findings. Washington, DC: World Bank.

———. 2007a. The Development Potential of Regional Programs – An Evaluation of World Bank Support of Multicountry Operations. Washington, DC: World Bank.

———. 2007b. Water Management in Agriculture—Ten Years of World Bank Assistance, 1994-2004. Washington, DC: World Bank.

http://www.gsdrc.org/docs/open/HD653.pdf

http://www.gsdrc.org/docs/open/HD653.pdf


112

———. 2008a. Country Financial Accountability Assessments and Country Procurement Assessment Reports: How Effective Are World Bank Fiduciary Diagnostics? Washington, DC: World Bank.

———. 2008b. Public Sector Reform: What Works and Why? An IEG Evaluation of World Bank Support. Washington, DC: World Bank.

———. 2008c. The World Bank in Indonesia 1999-2006--Country Assistance Evaluation. Washington, DC: World Bank

———. 2009a. Bangladesh Country Assistance Evaluation 2001-2008. Washington, DC: World Bank

———. 2009b. Improving Effectiveness and Outcomes for the Poor in Health, Nutrition and Population--An Evaluation of World Bank Group Support since 1997. Washington, DC: World Bank

———. 2009c. Review of IDA Internal Controls: An Evaluation of Management’s Assessment and the IAD Review. Washington, DC: World Bank.

———. 2009d. The World Bank Group Guarantee Instruments 1990-2007 –An Independent Evaluation. Washington, DC: World Bank

———. 2010a. Cambodia – An IEG Country Assistance Evaluation, 1999-2006. Washington, DC: World Bank

———. 2010b. Climate Change and the World Bank Group. Phase II: The Challenge of Low-Carbon Development. Washington, DC: World Bank.

———. 2010d. IDA Internal Controls: Evaluation of Management’s Remediation Program--Report on Management’s Implementation and IAD’s Review of the Five-Point Action Plan. Washington, DC: World Bank.

———. 2010d. Mozambique Country Program Evaluation. Washington, DC: World Bank.

———. 2010e. Poverty Reduction Support Credits: An Evaluation of World Bank

Support. Washington, DC: World Bank.

———. 2010f. The World Bank in Nigeria 1998-2007—Country Assistance Evaluation. Washington, DC: World Bank

———. 2011a. “Aid Effectiveness 2005-10: Progress in Implementing the Paris Declaration.” World Bank, Washington, DC.

———. 2011b. Capturing Technology for Development – An IEG Evaluation of World Bank Group Activities in Information and Communication Technologies. Washington, DC: World Bank.

———. 2011c. Trust Fund Support for Development – An Evaluation of the World Bank’s Trust Fund Portfolio. Washington, DC: World Bank.

———. 2011d. World Bank Country-Level Engagement on Governance and Anticorruption: An Evaluation of the 2007 Strategy & Implementation Plan. Washington, DC: World Bank.

———. 2011e. World Bank Progress in Harmonization and Alignment in Low-Income Countries. Washington, DC: World Bank.

———. 2012a. The Matrix System at Work: An Evaluation of the World Bank’s Organizational Effectiveness. Washington, DC: World Bank.

———. 2012b. “The World Bank and Public Procurement. An Independent Evaluation.” Approach Paper. Washington, DC: World Bank.

———. 2012c. The World Bank Group’s Response to the Global Economic Crisis—Phase II. Washington, DC: World Bank.

———. 2013. “Comparison of the International Instruments on Public Procurement.” Background paper. Washington, DC: World Bank.

IMF (International Monetary Fund). 2012. Update on the Bank’s Business Modernization: Results, Openness,

113

and Accountability Spring 2012.”IMF World Bank Development Committee.

Kivistö, T., V.M. Virolainen, and E. Tella. 2003. Consortia purchasing and logistics in Kuopio area (Finland)--Lessons learned from a 4-year project. IRSPP conference, Budapest, and in KV Thai et al. (eds.), (2005): Benchmarking municipal public procurement activities in Finland. In Challenges in Public Procurement: an International Perspective.

Korpela, J., and M. Tuominen. 1994. Benchmarking Logistics Performance with an Application of the Analytical Hierarchy Process. Lappeenranta, Finland: Lappeenranta University of Technology.

Kruger. K, 2009. “Ban on negotiations in tender procedures – Undermining the best value for money.” Journal of Public Procurement, 4.

Mackie, Andrew, and Giovanni Caprio. 2011. Assessing the Impact of the FEFA Framework. Washington, DC: PEFA Secretariat.

Ministry of Foreign Affairs of Denmark. 2008. “Evaluation of the Implementation of the Paris Declaration: Phase One Synthesis Report.” Denmark: Ministry of Foreign Affairs.

OECD (Organization for Economic Cooperation and Development). 2003. Mainstreaming the Procurement Function into the Public Expenditure Policy and Effectiveness Dialogue. Paris: OECD.

———. 2005a. “Paris Declaration on Aid Effectiveness--Ownership, Harmonization, Alignment, Results and Mutual Accountability.” Paris, France: OECD.

———. 2005b. “Strengthening Procurement Capacities in Developing Countries.” Volume 3 of the OECD-DAC Guidelines and Reference Series, Harmonising Donor Practices for Effective Aid Delivery. Paris: OECD.

———. 2006a. “The Challenge of Capacity Development: Working Towards Good Practice.” DAC Reference Document, Paris.

———. 2006b. “Methodology for Assessment of National Procurement Systems.” Paris: OECD.

———. 2009. “Compendium of Country Examples and Lessons Learned from Applying the Methodology for Assessment of National Procurement Systems: Sharing Experiences.” OECD Journal on Development 9(4).

———. 2011a. International Engagement in Fragile States: Can’t We Do Better? Paris: OECD.

———. 2011b. A Practical Guide for Transforming Procurement Systems. Paris: OECD.

———. 2011c. Strengthening Country Procurement Systems, prepared for the 4th High Level Forum on Aide Effectiveness, November 29 – December 1, 2011, Busan, Korea. OECD/DAC Working Party on Aid Effectiveness Task Force on Procurement.

OECD-DAC and World Bank. 2004. “Mainstreaming and Strengthening Public Procurement: A Strategic Framework.” Draft report of the 3rd Joint Round Table on Strengthening Procurement Capacities in Developing Countries. Washington, DC, and Paris.

Pallas, Christopher L., and Jonathan Wood. 2009. “The World Bank’s Use of Country Systems for Procurement: A Good Idea Gone Bad?” Development Policy Review 27 (2): 215-230.

PEFA Secretariat. 2011. Independent Evaluation of the Public Expenditure and Financial Accountability Program. Washington, DC: PEFA.

Pollit, C., and G. Bouckaert. 2004. Public Management Reform: A Comparative


114

Analysis. New York: Oxford University Press.

Sanchez, A. 1997. The World Bank Procurement Function – Adjusting to Emerging Needs. Washington, DC. World Bank. Retrieved from: http://intresources.worldbank.org/INTPROCUREMENT/Resources/Procurement-Strategy.pdf.

Singh, J., Dilip R. LImaye, Brian Henderson, and Xiaoyu Shi. 2012. “Public Procurement Of Energy Efficiency Services: Lessons from International Experience.” Washington, DC: World Bank.

Tonkin, Christine. 2012. “Procurement Governance Reform at the International Atomic Energy Agency.” Mimeo.

Thai, K.V., et al. 2005. “Benchmarking Municipal Public Procurement Activities in Finland.” In Challenges in Public Procurement: An International Perspective. Academic Press.

UNCITRAL (United Nations Commission on International Trade Law) Secretariat. (1994, 2011) “UNCITRAL Model Law on Public Procurement.” UNCITRAL, Vienna, Austria.

Volcker, Paul, Gustavo Gaviria, John Githongo, Ben W. Heineman Jr., Walter Van Gerven, and Sir John Vereker. 2007. Independent Panel Review of the World Bank Group Department of Institutional Integrity. Washington, DC: World Bank.

World Bank. 1997. “The World Bank Procurement Function- Adjusting to Emerging Needs.” SecM97-854, Washington, DC: World Bank.

———. 2002a. “Fiduciary Management for Community-Driven Development Projects: A Reference Guide.” Washington, DC: World Bank.

———. 2002b. Guidelines for Country Procurement Assessment Reviews. Washington, DC: World Bank.

———. 2003. “Standard Bidding Documents: Supply and Installation of Information Systems.” Washington, DC.

———. 2004. “Country Procurement Issues Paper—Rwanda.” Washington, DC: World Bank.

———. 2005a. “Africa Capacity Building Task Force Report: Forging Effective States and Engaged Societies.” Washington, DC: World Bank.

———.2005b. “Expanding the Use of Country Systems in Bank-Supported Operations: Issues and Proposals.” Operations Policy and Country Services, Washington, DC.

———. 2005c. “Meeting the Challenge of Africa’s Development: A World Bank Group Action Plan.” Africa Region Department, Washington, DC: World Bank.

———. 2006. “Quality of Information and Communication Technology Components in Bank Projects.” Quality Assurance Group Assessment, Washington, DC: World Bank.

———. 2007a. “Detailed Implementation Review. India Health Sector. 2006-2007.” INT, Washington, DC: World Bank.

———. 2007b. “Mandatory Provisions for NCB (for Bank/IDA financed contracts).” Washington, DC: World Bank.

———. 2007c. “Operational Approaches and Financing in Fragile States.” Washington, DC.

———. 2007d. “Strengthening World Bank Group Engagement on Governance and Anti-Corruption.” Washington, DC: World Bank.

———. 2008a. “Enhanced Support to Staff Working in Fragile and Conflict-Affected States.” Washington, DC: World Bank.

http://intresources.worldbank.org/INTPROCUREMENT/Resources/Procurement-Strategy.pdf



115

———. 2008b. “Use of Country Procurement Systems in Bank-Supported Operations: Proposed Piloting Program.” R008-0036/5. Washington, DC: World Bank.

———. 2009a. “Guidance to Staff for Using the Company Risk Profile Database (CRPD).” World Bank, Washington, DC.

———. 2009b. “Handling Suspected and Alleged Fraud and Corruption in Procurement: OPCS and INT Protocol.” Washington, DC: World Bank.

———.2009c. “Piloting Program in Use of Country Systems: First Progress Report.” Operations Policy and Country Services. Washington, DC: World Bank.

———. 2010a. Financial Management Sector Strategy FY11-13. 6, 9. Washington, DC: Financial Management Sector Board.

———. 2010b. “Guidance Note on the Operational Risk Assessment Framework (ORAF) – Risks to Achieving Results.” Operations Policy and Country Services, World Bank, Washington, DC.

———. 2010c. “Guidance Note: Procurement Arrangements Applicable to Public-Private Partnership Contracts Financed under World Bank Projects.” Washington, DC: World Bank.

———. 2010d. Monitoring Report 2010. PEFA Secretariat Report. World Bank, Washington, DC. http://pefa.org/sites/pefa.org/files/attachments/ENG-%20MR-10-FinalReport-May192011.pdf.

———. 2010e. “Piloting Program in the Use of Country Procurement Systems—Second Progress Report.” Washington, DC: World Bank.

———. 2010f. “Procurement Risk Assessment and Management System User Guide.” OPSOR, May 2010, Updated

July 2011, November 2011, January 2012. Washington, DC: World Bank.

———2010g. “Public Procurement of Energy Efficiency Services: Lessons from International Experience.” Washington, DC: World Bank.

———. 2010h. “Republic of Moldova Country Procurement Assessment Report.” Washington, DC: World Bank.

———. 2011a. “Community-Driven Development Projects: A Review of Governance Issues.” OPCS Document, World Bank, Washington, DC.

———. 2011b. Country Partnership Strategy Progress Report for the Republic of Moldova for the Period FY09-FY13. Washington, DC: World Bank.

———. 2011c. “Diversity Benchmarking Study in Selected Countries.” OPSOR, Mimeo. Washington, DC: World Bank.

———. 2011d. “Financial Management Information Systems—25 Years of World Bank Experience on What Works and What Doesn’t.” Washington, DC: World Bank.

———. 2011e. “Green Procurement Policy Framework in Selected Countries.” OPSOR, Mimeo. Washington, DC: World Bank.

———. 2011f. “Guidelines: Procurement of Goods, Works, and Non-Consulting Services under IBRD Loans and IDA Credits and Grants by World Bank Borrowers.” Washington, DC: World Bank.

———. 2011g. “Strengthening Country Procurement Systems.” Prepared for the 4th High Level Forum on Aide Effectiveness, November 29 – December 1, Busan, Korea. Washington, DC: World Bank.

———. 2011h. “Strengthening World Bank Engagement on Governance and Anti-Corruption: Procurement Companion Piece.” Washington, DC: World Bank.

http://pefa.org/sites/pefa.org/files/attachments/ENG-%20MR-10-FinalReport-May192011.pdf




116

———. 2011i. World Development Report: Conflict, Security, and Development. Washington, DC: World Bank.

———. 2012a. “Analysis of World Bank Completed Cases of Fraud and Corruption from the Perspective of Procurement.” Integrity Vice Presidency, Washington, DC: World Bank.

———. 2012b. “Applying the Principles for Good International Engagement in Fragile States and Situations to Strengthen and Transform Public Procurement Systems.” Background note, OPSOR. Washington, DC: World Bank.

———. 2012c. “The Bank's Procurement Policies and Procedures: Initiating Discussion and Approach Papers--Proposed Review.” Report submitted to the Audit Committee and to CODE, Washington, DC: World Bank.

———. 2012d. “Fiduciary Management for Community-Driven Development Projects: A Reference Guide.” Washington, DC: World Bank.

———. 2012e. Financial Management and Procurement in World Bank Operations: Annual Report for FY11. Washington, DC: World Bank.

———. 2012f. “Green Procurement in Selected Environmental Policy Framework: Review of the World Bank‘s Procurement Policies and Procedures.” Background Paper. Washington, DC: World Bank.

———. 2012g. “Guidance Note for Design and Management of Procurement Responsibilities in Community-Driven Development Projects.” Washington, DC: World Bank.

———. 2012h. “Investment Lending Reform: Modernizing and Consolidating Operational Policies and Procedures.” OPCS, Washington, DC: World Bank.

———. 2012i. “OPCPR Procurement Risk Assessment and Management System

User Guide.” Washington, DC: World Bank.

———. 2012j. “Procurement and PPP Transactions Guidance for MDB Public-Sector Engagements and Procurement Principals Applicable to Private Sector Transactions—Guidance for MDBs.” Washington, DC: World Bank.

———. 2012k. “Procurement Risk Assessment and Management System User Guide.” Washington, DC: World Bank.

———. 2012l. “Project Appraisal Document on a Proposed Credit in the Amount of SDR 26.1 million ($40.0 million Equivalent to the Republic of Moldova for the Moldova Education Reform Project).” Washington, DC: World Bank.

———.2012m. “Public Procurement of Energy Efficient Products: Lessons from Around the World.” Energy Sector Management Program. Washington, DC: World Bank.

———.2012n. “Review of the World Bank’s Procurement Policies and Procedures: Achieving Social Diversity Through the use of Supplier Preferences in the Procurement Process.” Background paper. Washington, DC: World Bank.

———.2012o. “Review of the World Bank’s Procurement Policies and Procedures, The Use and Impact of the Bank’s Policy of Domestic Preferences.” Background Paper. Washington, DC: World Bank.

———. 2012p. “Update on the Bank’s Business Modernization: Results, Openness, and Accountability Spring 2012.” DC2012-0005. Washington, DC: World Bank.

———. 2012q. “Virtual VP Review of the Procurement Policy Review Proposal--An Approach Paper.” Washington, DC: World Bank.

117

———.2013a. “Approach for Private Sector Procurement in Projects financed by IBRD/IDA, including Public Private Partnerships.” Washington, DC: World Bank.

———. 2013b. “Audit of the Bank’s Operational Framework for Using Investigation Results in Bank-Funded Projects.” Report No. IBRD FY12-16. Washington, DC: World Bank.

———. 2013c. “Procurement in World Bank Investment Operations. Phase I: A Proposed New Framework.” OPCS. Washington, DC: World Bank.

———. 2013d. “Red Flags Learning Tool.” Integrity Vice Presidency. http://go.worldbank.org/D1K227ET81.

———. 2013e. “Situations of Urgent Need of Assistance or Capacity Constraints: Simplified Procurement Procedures and Guiding Principles.”

———. 2013f. “Supporting Countries in Strengthening their Public Procurement Systems: Second Generation Reforms” (draft). Operations Risk Management Department. Washington, DC: World Bank.

———.2013g. “Use of Country Procurement Systems.” Background Paper, Washington, DC, World Bank.

World Bank Institute. 2012. “Public-Private Partnership Reference Guide, Version 1.0.” Washington, DC: World Bank.

World Trade Organization. 1996. “Agreement on Government Procurement (GPA).” Geneva: World Trade Organization.

.

119

Endnotes Appendix A 1 Others apply some value for money principles (such as life-cycle costing, for example) to varying degrees, often without calling it “value for money.” Some agencies use the term but without defining a distinct procurement process around it—for example, the United Nations procurement guide refers to value for moeny, but only in limited terms and makes no recommendations to operationalize VfM in procurement policies and practices. 2 Getting Value for Money from Procurement: How Auditors Can Help, Office of Government Commerce and National Audit Office, UK, 2001; Regularity, Propriety and Value for Money, Treasury Officer of Accounts, HM Treasury, UK, November 2004; Procurement Efficiency and Value for Money Measurement, Office of Government Commerce, 2005; and An Introduction to Public Procurement, Office of Government Commerce, 2008. 3 “Best Value” was ascribed its modern regulatory definition by FAC 97-02, Sept. 30, 1997, as part of the rewrite of Federal Acquisition Regulation Part 15. 4 Directive 2004/18/EC of the European Parliament and of the Council of March 31, 2004 on the Coordination of Procedures for the Award of Public Works Contracts, Public Supply Contracts and Public Service Contracts. The EU Directive says, “…Where the contracting authorities choose to award a contract to the most economically advantageous tender, they shall assess the tenders in order to determine which one offers the best value for money.” 5 MEAT must be used when competitive negotiation procedures are applied (Article 29). 6 Commonwealth of Australia, Commonwealth Procurement Guidelines, December 2008. Issued under Regulation 7 of the Financial Management and Accountability Regulations 1997. 7 APEC Government Procurement Experts Group Non-Binding Principles on Government Procurement. August 1999. 8 Procurement in World Bank Investment Operations. Phase I: A Proposed New Framework. OPCS, March 29, 2013. 9 Although negotiation after contract award is not generally accepted good practice, with the possible exception of in the United Kingdom.

10 By using benchmarks, for example, as illustrated in a simple example in Chapter 4. 11 See Chapter 6 for a description of competitive negotiation. 12 The proposal is currently under legislative review, between the Council and the European parliament. In December 2012, the Council adopted a compromise text, which is now the basis for negotiations between council and parliament. There is some expectation, at the European Union, that the new directive may be adopted before the end of 2013. Implementation dates are unclear. 13 Governance in public procurement is increasingly on the agenda of international organizations as well, including the World Trade Organization, as witnessed by proposed amendments to its Government Procurement Agreement (Chapter 5).

Appendix B 14 The 2000 Bank document Community-based Contracting—A Review of Stakeholder Experience highlighted local shopping among the very simple contracting procedures. 15 Examples of incentives were, for example, savings reverting back to communities for financing additional facilities.

ENDNOTES

120

16 Although this conclusion may not have been surprising, since the sample had purposively focused on successfully completed subprojects. Construction cost-savings ranged from 12 percent to 56 percent compared to original estimates. At the same time, the quality and sustainability of the completed projects was found to be high.

consultant. 18 A Mongolia case study noted that procurement of the ICT system went through three procurement processes over two years The study reports on three FMIS related projects in Mongolia (P051855, P077778, P098426), with the first project approved in June 1998. Two were active at the time of the report (2011).

19 The case study covers one Bank funded project (P035759), approved in 1995 and closed in 2002, and a subsequent Government-funded component. The cost of the Bank project was $78.5 million. In 1999, the Government and the Bank agreed to restructure the project, and the government initiated the Public Expenditure Management (PEM) component as a separate activity funded from the its own budget. Project funds were used to develop a customs information system and improvement of debt management system, together with related advisory support. The PEM component was implemented as a government led activity in parallel to the Bank project, with a cost of approximately $15.7 million. 20 Unlike the other MDBs, the Black Sea Trade and Development Bank takes a very much hands off approach. It only requires procurement methods that “lead to sound selection of goods, works and services at fair market prices.”

Appendix C

21 Resource Guide: Procurement Methods http://web.worldbank.org/WBSITE/EXTERNAL/PROJECTS/PROCUREMENT/0,,contentMDK:20109695~pagePK:84269~piPK:60001558~theSitePK:84266~isCURL:Y,00.html.

22 Most countries also set monetary thresholds for the use of shopping procurement method. 23 World Bank Procurement Method Thresholds http://web.worldbank.org/WBSITE/EXTERNAL/PROJECTS/PROCUREMENT/0,,contentMDK:21038090~isCURL:Y~menuPK:2926825~pagePK:84269~piPK:84286~theSitePK:84266,00.html. 24http://web.worldbank.org/WBSITE/EXTERNAL/PROJECTS/PROCUREMENT/0,,contentMDK:21038037~isCURL:Y~menuPK:2926825~pagePK:84269~piPK:84286~theSitePK:84266,00.html.

Bank guidelines do make publication of Specific Procurement Notices in UNDB and DG market electronic portal mandatory for ICB processes. 25 Procurement of goods, works and non-consulting services under IBRD Loans and IDA Credits and Grants by World Bank Borrowers, Paragraph 1.3, January 2011.

26 The field “amountUSE” of Form 384 was used as contract value. 27 When looking at the distribution by value a higher share is won by foreign firms, though still more than half of contracts in the highest quintile go to national firms (Table 1, “Distribution by Value”). 28 In 2008, three risk levels were defined by implementing agency (high, medium, or low). In 2009, Bank-wide maximum prior review thresholds were revised and categories expanded into high, substantial, moderate, or low. 292002 BP 11.0, Annex A Decision Authority Matrix. This document specifies clearance thresholds for task team leaders/PS/PAS, RPA, Regional Vice Presidents, and OPRC.

http://intranet.worldbank.org/WBSITE/INTRANET/OPSMANUAL/0,,contentMDK:22819821~menuPK:51457764~pagePK:51458737~piPK:51458739~theSitePK:210385,00.html

30 Email from OPCPR retrieved from: http://intranet.worldbank.org/WBSITE/INTRANET/INTCOUNTRIES/INTAFRICA/INTAFRPRO/0,,contentMDK:21772310~pagePK:64168332~piPK:64168299~theSitePK:1938585,00.html, this specified, for example, task team leader/procurement specialist levels of justification for single source selection of consultants.



http://intranet.worldbank.org/WBSITE/INTRANET/INTCOUNTRIES/INTAFRICA/INTAFRPRO/0,,contentMDK:21772310~pagePK:64168332~piPK:64168299~theSitePK:1938585,00.html

http://intranet.worldbank.org/WBSITE/INTRANET/INTCOUNTRIES/INTAFRICA/INTAFRPRO/0,,contentMDK:21772310~pagePK:64168332~piPK:64168299~theSitePK:1938585,00.html

ENDNOTES

121

31http://web.worldbank.org/WBSITE/EXTERNAL/PROJECTS/PROCUREMENT/0,,contentMDK:21038090~menuPK:2926825~pagePK:84269~piPK:84286~theSitePK:84266~isCURL:Y~DIR_PATH:WBSITE/EXTERNAL/PROJECTS/PROCUREMENT/,00.html. 32 http://www.transparency.org/research/cpi/ 33 http://www.worldbank.org/ida/IRAI-2011.html. Bank data on CPIA scores for IBRD countries were also used.

Appendix D 34 Sistema de Ejecucion de Planes de Adquisiciones in Spanish.

35 SEPA does not handle procurement documents or attachments through the system. 36 To upload a procurement plan the government enters project information and data on the related contracts. On the project level this includes procurement method and prior review thresholds, names of implementing agencies, responsible task team leaders and procurement specialists, project amounts and key project dates, among other things. On the contract level this includes contract description, estimated amount, the percentage of the contract financed by the local counterpart to the project, and estimated dates for all procurement stages related to the contract. Over the course of execution, actual amounts and actual dates are entered into the system, after the estimated dates. Dates are entered according to procurement method and review method. After the actual contract signature date is recorded, contracting data such as bidder’s names and their bids, winning bidder and contract value can be entered. 37 In the form of Excel spreadsheets, available from October 2008. Database macros allow users to produce management reports filtered for information for the entire region, for a selected country department, country or procurement staff. 38 The data template is included in Appendix A, which contains the overall questionnaire template. 39 Information was requested on the procurement steps borrower issue of specific procurement notice, borrower first submission to Bank of draft bid (preQ) documents, Bank final no objection to draft bid (PreQ) documents, borrower issue of Bid (PreQ) documents, borrower bid (PreQ) opening date/ minutes, borrower submission to Bank of bid evaluation report, Bank no-objection to bid evaluation report, borrower submission to Bank of negotiated contract, Bank no-objection to negotiated contract, date of signature of contract. 40 For consultant services selected by quality and cost the dates of submission of / no objection to the negotiated contract were considered as the final bid evaluation report. For contracts flagged as ICB/NCB Two Stage the first stage Submission of / No Objection to Bid Evaluation Report was used, only one observation was received for the second stage. For contracts flagged as subject to Pre-Qualification the dates of Submission of / No Objection to the final Bid Evaluation Report were used. Where Bid Evaluation Report Submission / No Objection dates were not provided the “Award No Objection” date in the bank-wide Form 384 database was used if the contract information received from the field could be matched with Form 384. 41 International Monetary Fund, International Financial Statistics: http://elibrary-data.imf.org/. 42 For the regression analysis of average processing times from submission of draft bid documents to no objection to final bid evaluation report only quality- and cost-based selection procurement is considered for consultant services contracts. 43 Two databases were provided by the regional procurement management office, on user responsiveness and elapsed times. Data cover the period from 2009 to 2012; however, data for 2009 are incomplete and 2010 provides a first reasonably full dataset.

44 Numbers of iterations for 2011 were 2,778; for 2010, 2,540; and for 2009, 1,970. While 2009 was a startup year, the system is now at “steady state.” Arguably it could also suggest an increase in the number of iterations. However, data in Table B.14 suggest that this may not be the case.

http://www.worldbank.org/ida/IRAI-2011.html

ENDNOTES

122

45 The database shows some errors in recording times. For example, in 62 cases (of 10,331) the average number of days that client take to respond has a negative values. However, these are not frequent.

46 For clients, limited international bidding takes 19 days to process. Other lengthier client response times correspond to ICB and two-stage ICB with 15 and 14 days, respectively. 47 The latest monthly report produced by the Africa Procurement Unit on April 2, 2013, that includes no objections processed through July 1, 2011, and excludes 1 percent of outliers.