Embed Size (px)
Citation preview
Document of The World Bank
Report No: ICR00001918
IMPLEMENTATION COMPLETION AND RESULTS REPORT
(IDA-38910 TF-50298)
ON A
CREDIT
IN THE AMOUNT OF SDR 6.8 MILLION
(US$10.15 MILLION EQUIVALENT)
TO THE
REPUBLIC OF ARMENIA
FOR A
PUBLIC SECTOR MODERNIZATION PROJECT
October 26, 2011
Poverty Reduction and Economic Management Unit Europe and Central Asia Region
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
CURRENCY EQUIVALENTS (Exchange Rate Effective October 25, 2011)
Currency Unit = Armenian Dram (AMD) AMD 1.00 = USD 0.003 USD 1.00 = AMD 377.75
FISCAL YEAR January 1 – December 31
ABBREVIATIONS AND ACRONYMS
AAPA Academy of Public Administration IDA International Development Association CAS Country Assistance Strategy IGR Institutional and Governance Review CSC Civil Service Council IRI Intermediate Results Indicator CPS Country Partnership Strategy ISR Implementation Status and Results Report CFAA Country Financial Accountability
Assessment IMF International Monetary Fund
COC Chamber of Control INTOSAI International Organization of Supreme Audit Institutions
CPAR Country Procurement Assessment Report MTED Ministry of Trade and Economic Development
ECA Eastern Europe and Central Asia PAD Project Appraisal Document EDMS Electronic Document Management
System PDO Project Development Objective
ESW Economic and Sector Work PIU Project Implementation Unit GDP Gross Domestic Product PSMP Public Sector Modernization Project HRM Human Resources Management QAG Quality Assurance Group HRMIS Human Resources Management
Information System TTL Task Team Leader
ICT Information and Communications Technology
WAN Wide Area Network
ICU Inter-Community Unions WTO World Trade Organization
Vice President: Philippe Le Houerou
Country Director: Asad Alam
Sector Manager: William L. Dorotinsky
Project Team Leader: K. Migara O. de Silva
ICR Team Leader: Ronald Myers
ARMENIA Public Sector Modernization Project
CONTENTS
Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph
1. Project Context, Development Objectives and Design ................................................... 12. Key Factors Affecting Implementation and Outcomes .................................................. 93. Assessment of Outcomes .............................................................................................. 134. Assessment of Risk to Development Outcome ............................................................. 225. Assessment of Bank and Borrower Performance ......................................................... 226. Lessons Learned............................................................................................................ 267. Comments on Issues Raised by Borrower/Implementing Agencies/Partners ............... 28 Annex 1. Project Costs and Financing .............................................................................. 29Annex 2. Outputs by Component...................................................................................... 30Annex 3. Economic and Financial Analysis ..................................................................... 45Annex 4. Bank Lending and Implementation Support/Supervision Processes ................. 46Annex 5. Beneficiary Survey Results ............................................................................... 47Annex 6. Stakeholder Workshop Report and Results ....................................................... 47Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ......................... 47Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ........................... 56Annex 9. List of Supporting Documents .......................................................................... 56 MAP
A. Basic Information
Country: Armenia Project Name: Public Sector Modernization Project
Project ID: P060786 L/C/TF Number(s): IDA-38910,TF-50298
ICR Date: 10/26/2011 ICR Type: Core ICR
Lending Instrument: SIL Borrower: REPUBLIC OF ARMENIA
Original Total Commitment:
XDR 6.80M Disbursed Amount: XDR 6.73M
Revised Amount: XDR 6.73M
Environmental Category: C
Implementing Agencies: Public Sector Reform Commission, Civil Service Council, Academy of Public Administration, State Procurement Agency, Chamber of Control, Ministry of Finance and Economy, Ministry of Economic Development and Trade, the Office of the Government, Tax and Customs Services, and Ministry of Territorial Administration
Cofinanciers and Other External Partners: B. Key Dates
Process Date Process Original Date Revised / Actual
Date(s)
Concept Review:
02/25/2003 Effectiveness: 12/13/2004 12/13/2004
Appraisal: 01/12/2004 Restructuring(s):
Approval: 05/04/2004 Mid-term Review:
05/30/2008 05/26/2008
Closing: 03/31/2009 02/28/2011 C. Ratings Summary C.1 Performance Rating by ICR
Outcomes: Satisfactory
Risk to Development Outcome: Moderate
Bank Performance: Satisfactory
Borrower Performance: Satisfactory
C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings
Quality at Entry: Moderately Unsatisfactory
Government: Satisfactory
Quality of Supervision:
Satisfactory Implementing Agency/Agencies:
Satisfactory
Overall Bank Performance:
Satisfactory Overall Borrower Performance:
Satisfactory
C.3 Quality at Entry and Implementation Performance Indicators
Implementation Performance
Indicators QAG Assessments
(if any) Rating
Potential Problem Project at any time (Yes/No):
No Quality at Entry (QEA):
Satisfactory
Problem Project at any time (Yes/No):
Yes Quality of Supervision (QSA):
None
DO rating before Closing/Inactive status:
Moderately Satisfactory
D. Sector and Theme Codes
Original Actual
Sector Code (as % of total Bank financing)
Central government administration 70 82
Sub-national government administration 30 18
Theme Code (as % of total Bank financing)
Administrative and civil service reform 25 73
Law reform 13 1
Municipal finance 13 9
Municipal governance and institution building 24 9
Public expenditure, financial management and procurement
25 8
E. Bank Staff Positions At ICR At Approval
Vice President:
Country Director: Asad Alam D-M Dowsett-Coirolo
Sector Manager: William Leslie Dorotinsky Helga Muller
Project Team Leader: K. Migara O. De Silva Antonius Verheijen
ICR Team Leader: Ronald Myers
ICR Primary Author: Ronald Myers F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) To enhance efficiency in public sector management through: (a) piloting innovations in selected institutions; and (b) improving transparency in government decision-making and policy implementation.
Revised Project Development Objectives (as approved by original approving authority) (a) PDO Indicator(s) – see below
Indicator Baseline Value Original Target Values
(from approval documents)
Formally Revised Target Values
Actual Value Achieved at Completion or Target Years
1) The management of the Civil Service of Armenia is transparent and based on merit principles that are consistently applied throughout the system.
Merit-based recruitment and promotion system not established
Merit Principles in recruitment and promotion are applied
NA Increasing majorities of surveyed civil service applicants report recruitment and promotion is merit based. By the closing date of the project, survey results for perception of fairness and efficiency of application of merit-based principles during competitive appointments and annual evaluations of civil servants reached, after introduction of the system 70.6% in 2010 versus an initial level of 59.1% in 2005.
2) The financial management control framework and procurement procedures become more efficient, effective and transparent.
Role of, and relationship between, internal and external audit is not fully understood. Procurement system is cumbersome and not as responsive to users’ needs as it could be. 72.1% positive perception about equal opportunities in public procurement. Average duration of procurement exercise and the number of sole source contracts respectively 51 days and 1442 contracts.
E-procurement activities yield increased efficiency in procurement with greater transparency through web information system. Where e-Procurement is applied, time taken for an average procurement exercise is reduced by 25% by the end of the project. Number of sole source contracts in public procurement is reduced by 25% by the end of the project. Audit standards on for ten subject areas, one of which to be ethical standards are developed, approved by the Chairman of CoC and effectively applied.
NA Surveys of businesses indicates satisfaction with government procurement operations – by the closing date of the project survey results for positive perception about equal opportunities in public procurement reached 80.7% from 2005 baseline of 72.1%. E-procurement installed and its use to be mandatory on January 1, 2012. Even before launch of e-procurement system the time taken for average procurement exercise was reduced from 51 to 45 days (11.8% reduction) and the number of sole source contracts was reduced from 1442 to 380 (73.6% reduction). Audit standards on 10 subject areas (including ethics) developed, approved and effective.
3) The organization of Central and local administrations is more functional and effective.
55% business community satisfaction with the appropriateness and timeliness of certification and standardization services. The administrative structure of local self-government is fragmented, they lack fiscal autonomy and implementation capacity, and local self-government staff careers are not regulated yet (46.7% satisfaction from local government services).
Level of satisfaction of the Business Community with the appropriateness and timeliness of certification and standardization services is increased by 50% over project duration as measured by surveys. Improved citizens’ perception on local government management capacity, as well as on the quantity and quality of public services locally delivered, in pilot Communities and ICUs as compared to pre-project situation. Transfer of the property and land tax database to local authorities is completed for a significant number of Inter-Community Units (including rural communities). Status of municipal civil servants established in accordance with the law.
NA For central government improved human resource management, e-procurement developed, and improved capacity for policy formulation and implementation. However, level of satisfaction of the Business Community with the appropriateness and timeliness of certification and standardization services was increased only by 9.1% over project duration reaching 60%. For local administrations, transfer of property and land tax administration was completed for entire population of communities (with exception of very few rural communities). Status of municipal civil servants established in accordance with the law. Public satisfaction on local government services registered 17% increase over the baseline reaching 54.2%.
(b) Intermediate Outcome Indicator(s) – see below
Indicator Baseline Value
Original Target Values (from
approval documents)
Formally Revised Target
Values
Actual Value Achieved at
Completion or Target Years
1) Standard tests for
recruitment
processes produced
and published
Standard tests not in place Standard tests for recruitment processes produced and published
NA All agencies develop, apply, and periodically update sector specific tests approved by the CSC
2) Standard
procedures for
performance
evaluation produced
and published
Procedures for performance evaluation not defined
Performance evaluation system evaluated and finalized
NA Performance Evaluation procedures were developed, applied, linked to staff work plans, automated through EDMS, and rolled-out to the whole CS system
3) Information on vacancies in all institutions made available through the CSC web-site and information center
Vacancies not published centrally and information provision is inadequate
All vacancies are available at the CSC information center
NA All vacancies are announced through the CSC website and Information Center
4) Recruitment and attestation centers equipped to allow better oversight over recruitment and attestation
Monitoring of procedures ineffective, only through irregular site visits
Assessment centers fully functional in all institutions under the CSL and linked to the CSC
NA Assessment centers are functional in all institutions under the CSL and linked to the CSC
5) HRMIS system installed and effectively providing data to the CSC on recruitment and career planning
No electronic register of civil servants available, no centralized information on career planning collected
HRMIS functioning for all institutions including a career tracking system
NA HRMIS functioning for all institutions including a career tracking system
6) CS training curricula reformed and training programs integrated in career development
In service training organized in an ad hoc manner and not clearly linked to career development
Training records integrated in HRMIS.
NA Training plans based on performance evaluation are linked to HRMIS and EDMS
7) The E-procurement system is introduced and found to be effective by suppliers and government users
No E-procurement system functioning
Basic elements of an E-procurement system are operational
NA Core elements of an E-procurement system has been designed, tested, and transferred to government. The E-tendering system is currently under phased implementation and its use will be mandatory within the government by January, 2012.
8) The number of suppliers bidding for government contracts increases by 30 percent from pre-e-procurement system base to the end of the project
405 bidders (Not in approval document)
30% increase NA 440% increase between 2006 and 2010 (2192 bidders)
9) Time taken for average procurement exercise is reduced by 25 percent from the pre e-procurement system base to the end of the project.
51 days (Not in approval document) 25% time reduction NA 12% time reduction by 2010 (45 days)
10) The number of sole source contracts is reduced by 25 percent from pre-procurement system base to the end of the project
1442 contracts (Not in approval document)
25% reduction in sole source contracts
NA 74% reduction (380 contracts)
11) Audit standards on 10 subject areas, one of which should be ethical standards, are developed and approved by the Chairman and applied.
They do not have a set of Audit standards or an associated audit manual at the moment. The web site does not contain all reports. (No other baseline value in approval document)
At least 10 principal audit standards are written and approved by the Chairman, appropriate training developed and delivered.
NA 10 audit standards, including ethical standards, have been developed for specific areas, approved by the Chairman, and provide guidance to the staff when auditing specific areas. They will be updated to reflect international standards adopted during the recent meeting of INTOSAI in South Africa. The CoC website is operational and contains audit reports.
12) Four training course on the new standards are developed and delivered
No courses developed or delivered Courses developed and delivered
NA Courses developed and delivered
13) Action is taken on the Recommendations of the Consultancy funded under the Dutch Grant managed by the Association.
No action yet on consultancy recommendations
Government approves the strategy of external audit reforms based on the recommendations that come out of the Consultant’s report produced under the Dutch Grant Managed by the Association.
NA A Chamber of Control law, in line with recommendations from the consultant’s report recommending adoption of international standards for external and internal auditing, was passed on December 25, 2006 and became effective on June 17, 2007.
14) Electronic office and document tracking system established for pilot ministries
No electronic office and document tracking system
System running in three pilot institutions
NA EDMS piloted, rolled-out, linked to entire CS system including 15 ministries and 10 regional centers
15) Government Press Center and public information outlet developed and utilized
Information available to public at a very rudimentary level
No target set NA The Government Press Center is fully functioning enabling more efficient communication of policy with the public.
16) Information availability on product standards easily accessible and testing instruments for products available.
No web based system established Web based system for standards and technical regulations established and running
NA Consultancy delivered on certification and conformity assessment of various products. Laboratory equipment is procured, and a relevant website established.
17) Training courses and international seminars on internal audit procedures and techniques are developed and delivered
No baseline value recorded Training on internal audit procedures and practices is delivered
NA Study tour to UK was conducted and pilot internal audit units have received training. At least three training workshops were conducted in 2009 for chief auditors of central agencies and municipal administrations.
18) Two pilot internal audit units are established in a line ministry and
None established Two pilot internal audit units are established in a line ministry and community
NA Internal audit units are currently operational in three agencies.
community administration.
administration.
19) Legislation on the establishment and operation of ICUs approved, so that administrative fragmentation can be overcome
No legislation yet approved Legislation approved NA Legislation not approved for mandatory establishment of ICUs, although voluntary establishment of ICUs has been enabled.
20) Necessary legal amendments/complementary regulation, and an action plan for the transfer of the cadastre database adopted
Legal amendments and complementary regulations lacking
Transfer of the land tax database to local authorities is completed
NA Legal amendments and complementary regulations to enable local tax administration put in place. Nearly 100% of property and land tax databases have been transferred to communities.
21) Municipal civil service law approved and implemented, in congruency with the national civil service law.
No such legislation The implementation of the municipal service law has been completed in the whole country.
NA Municipal Services Law was approved December 14, 2004 and technical assistance delivered to support its implementation in municipalities
22) National training program adopted for local self-government officials
No training program exists The national training program for local self-government officials has been implemented in all ICUs and at least 80 percent of the remaining LSG units
NA Ultimately the project did not produce comprehensive national training program. However, the project developed a Municipal Management Information System rolled out to 225 municipalities (including the largest 60), and provided related training to officials
G. Ratings of Project Performance in ISRs
No. Date ISR Archived
DO IP Actual
Disbursements (USD millions)
1 06/28/2004 Satisfactory Satisfactory 0.00 2 06/29/2004 Satisfactory Satisfactory 0.00 3 12/28/2004 Satisfactory Satisfactory 0.00 4 06/09/2005 Satisfactory Satisfactory 0.50 5 06/26/2006 Satisfactory Moderately Satisfactory 0.78 6 12/04/2006 Satisfactory Moderately Satisfactory 1.64
7 12/28/2007 Moderately SatisfactoryModerately
Unsatisfactory 3.16
8 08/18/2008 Moderately Satisfactory Satisfactory 4.75 9 07/15/2009 Moderately Satisfactory Satisfactory 8.64 10 05/10/2010 Moderately SatisfactoryModerately Satisfactory 9.83 11 03/12/2011 Moderately Satisfactory Satisfactory 10.29
H. Restructuring (if any) Not Applicable
I. Disbursement Profile
1
1. Project Context, Development Objectives and Design
1.1 Context at Appraisal Since the collapse of the Soviet Union Armenia’s economic transformation has been profound. The economy is now a market-oriented one, highly open to trade and capital, technological innovation, and is based on services, light industry, and, to a declining extent, agriculture. However, when Armenia regained its independence in 1991 it immediately confronted a distorted and imploding economy, an overly centralized and ponderous public administration, and a level of physical and institutional infrastructure inadequate to compete internationally. In addition, an energy crisis, a war with neighboring Azerbaijan, and economic blockade contributed to hyper inflation and a 50% reduction in economic output, leaving nearly 60% of the population living below the poverty line. After the 1994 ceasefire agreement with Azerbaijan the government retargeted its efforts toward addressing urgent economic and social problems, initiating and implementing a remarkable macroeconomic stabilization and economic liberalization program that resulted in stable economic growth continuing for the next decade. Indeed, by the time of appraisal the economy was growing in the double digits (nearly 10% in 2001, 12.9% in 2002, and 13.9% in 2003). Private investment (supported in part by inflows from Armenian communities around the world) had increased, manufacturing exports were surging from a low base, and the private sector accounted for more than 70% of GDP. This progress had been aided by a series of reforms within the government itself. Most notably, the structure of the central administration was reorganized and rationalized through the consolidation of ministries from 21 to 18 with attendant reallocation of responsibilities, separation of policy formulation and monitoring from execution and service delivery, and passage of organic Civil Service and related civil service remuneration legislation. The Government recognized that these actions, however important, needed to be supported by “second generation” reforms. Complex and time consuming institutional capacity building efforts were required to improve the accountability, effectiveness, and transparency of public sector management. Without them the antiquated processes, still weak capabilities, and misuse of resources by the public sector would hinder sustained economic growth. While ministries had been reorganized, they too often lacked the capacity to formulate effective policies or implement them properly. Cross-ministerial coordination systems were not functioning well. Despite the laws, a government-wide Human Resource Management (HRM) system had not been established and personnel functions were weak. The level and structure of civil service wages were inadequate to sustain a professional cadre of staff. Government procurement procedures were inefficient and lacked transparency. Oversight of government financial activities through both internal and external audit functions was weak and confusingly dispersed among various agencies. Finally, the Government had enacted legislation increasing the functions and responsibilities of local authorities (RA Law on Local Self-Government 2002, RA Law
2
on Municipal Service 2004, RA Law on Local Self-Government in Yerevan 2008, etc.), even though many communities lacked even rudimentary execution capacity. While Armenia was receiving a broad array of technical and financial assistance from donors, including in the area of public sector strengthening, there were significant gaps in cross cutting government management systems and no well coordinated approach to state reform. The World Bank was among the first donors supporting Armenia’s economic and social transformation in the early years of independence. The Bank’s strategy initially focused on providing assistance towards overall economic reforms and institution building, rehabilitation of critical infrastructure, and social protection mechanisms. Over time it deepened its focus on capacity building, informed by a series of analytical work. Policy reform indicated by the reports were increasingly included in Development Policy Operations, and the Bank sought to directly support capacity building through investment projects, such as the 2000 Judicial Reform credit. The April 2001 CAS targeted the improvement in governance and public services as a key goal, directly undergirding the government’s poverty reduction strategy. The Public Sector Modernization Project (PSMP II) under review in this ICR was an investment credit aimed at supporting the CAS by increasing efficiency and transparency in public sector management. The credit targeted several cross-governmental management information systems, training to enhance the quality and effectiveness of policy management, personnel management, and local government service delivery. Complementing this horizontal approach, the project also aimed to support several core agencies responsible for the cross-cutting systems as well as some sector specific interventions. It was complemented by a series of policy based adjustment operations and extensive economic and sector-related work, as well as related activities financed by other donors.
1.2 Original Project Development Objectives (PDO) and Key Indicators (as approved) The Project Development Objective indicated in the Project Appraisal Document (PAD- Report No. 27563-AM) is to enhance efficiency in public sector management through: (a) piloting innovations in selected institutions; and (b) improving transparency in government decision-making and policy implementation. The PDO wording in the Development Credit Agreement is nearly the same. Six project monitoring indicators are laid out on page 5 of the PAD. They include:
1. The proportion of civil service appointments made through merit-based open competitions increases by 50% annually;
2. Where E-procurement is applied, time taken for an average procurement exercise is reduced by 25% by the end of the project;
3
3. The number of sole source contracts in public procurement is reduced by 25% by the end of the project;
4. Audit standards on 10 subject areas, one of which should be ethical standards, are developed and approved by the Chairman of the Chamber of Control (COC) and effectively applied, as verified by independent local audit experts;
5. Level of satisfaction of the business community with the appropriateness and timeliness of certification and standardization services is increased by 50% over project duration as measured by surveys;
6. Improved citizens’ perception on local government management capacity, as well as on the quantity and quality of public services locally delivered, in the pilot Communities and Inter-Community Unions (ICUs) as compared to the pre-project situation.
Unfortunately the Results Framework for the project was poorly designed with inconsistencies between the PDO outcomes and performance indicators, mathematically incorrect or inappropriately defined numeric targets, broadly defined indicators, lack of baseline data and targets, etc. For example, of the indicators above, only numbers 3 and 4 appear in the Results Framework and Monitoring Table in Annex 3 of the PAD. Numbers 5 and 6 did not appear in the Results Framework and were not tracked in ISRs. Other indicators appeared to have arbitrary and end of project targets without reflecting baseline information or considered appreciation of the challenges. As a consequence during project implementation the task team sought to retrofit the shortfalls in the results framework as well as use proxy indicators although the survey results reported in Annex 2 of this report are used with caution. Overall the monitoring and evaluation framework designed for the project during preparation does not meet the current standards of the Bank. Consequently for purposes of the project (and this ICR in Annex 2) the relevant PDO outcome indicators and IRIs are assumed to be those included in Annex 3 of the PAD as supplemented by additional or clarified indicators tracked in ISRs. For the purposes of clarity, performance against the indicators listed on page 5 of the PAD are included in Section F of the data sheet of this ICR and include: (i) for the mathematically hyper ambitious target for the civil service of a 50% annual improvement, the perception on fairness and efficiency of application of merit-based principles during competitive appointments and annual evaluations of civil servants reached, after introduction of the system, 70.6% from the 2005 initial level of 59.1%; (ii) the time taken for the average procurement exercise was reduced from 51 to 45 days (11.8%) even before the pending launch of e-procurement, versus a target of 25%; (iii) similarly the number of sole source contracts was reduced from 1442 to 380 (73.6%) between 2005 and 2010 before the launch of e-procurement versus a target of 25%; (iv) audit standards for ten subject areas, one of which to be ethical standards are developed, approved by the Chairman of CoC and effectively applied; (v) the level of satisfaction of the Business Community with the appropriateness and timeliness of certification and standardization services was increased by 9.1% over project duration reaching 60% versus a targeted
4
50% improvement; and (vi) public satisfaction with local government services registered a 17% improvement over the baseline, reaching 54.2%, versus an unquantified target.
1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification
There were no formal revisions of the PDO during the life of the project.
1.4 Main Beneficiaries The PAD does not explicitly list project beneficiaries as such. Based on the characteristics of the project, it could be assumed that the beneficiaries were, most directly, government officials and staff at the central and local levels benefiting from training, improved business processes and ICT equipment, and a more professional human resources environment. The following agencies were identified in the PAD as proposed recipients of credit resources—Civil Service Council, Academy of Public Administration, State Procurement Agency, Chamber of Control, Ministry of Finance and Economy, Ministry of Economic Development and Trade, the Office of the Government, Tax and Customs Services, and Ministry of Territorial Administration—although this does not cover the up to “sixty” potential benefitting state agencies plus sub-national governments also mentioned without name in the PAD. Beyond these agencies the roll out of Electronic Document Management System and performance management system expanded the impact of the project to cover the entire central and regional public administration, plus some selected municipalities. Citizens and private sector firms, of course, would be the ultimate beneficiaries in terms of improved services, better value for taxpayer resources, more transparency in government operations, and an improved poverty alleviation effort.
1.5 Original Components (as approved) This project includes five substantive components designed to support innovations in horizontal management processes and capacity building in specific agencies (although there is some overlap between horizontal and vertical elements in components iii and iv). The components were: (i) Strengthening Civil Service Management; (ii) Introducing E-procurement; (iii) Strengthening the Chamber of Control; (iv) Improving Policy Formulation and Implementation through Capacity Building; and (v) Developing Capacity in Local Governments. All components were to be supported by a range of activities including technical assistance, training, and computer hardware and software complementary to the TA and training. The project also included a sixth component to handle (vi) Project Management, Monitoring and Evaluation Surveys, and Communication on the project itself.
5
Strengthening Civil Service Management (US$2.62 Million--IDA contribution US$2.52 Million) This component focused on the development of systems and capacities to ensure a consistent application of merit-based recruitment and promotion processes across all institutions under the Civil Service Law. Key activities under subcomponents are listed below. Subcomponent 1A was to support the drafting of a Civil Service Development Strategy and the completion of remaining elements of the Regulatory Framework for Civil Service Management. Subcomponent 1B aimed to strengthen the new Civil Service Council (which oversees the recruitment process for senior positions in the state administration, coordinates the implementation of the Civil Service Law and manages the overall civil service system) through activities, which would improve the testing and attestation instruments; equip the Council’s offices; assist the Council to develop and implement a Civil Service Training Strategy; and help develop a Communications and Public Relations Strategy and establish a Public Information Center.
Subcomponent 1C would support the introduction of cross-ministerial systems for Human Resource Management and Assessment/Attestation. The introduction of government-wide IT systems for HRM would enable the CSC to monitor the consistent application of merit based procedures and develop a pro-active career management policy that would reward professional excellence through accelerated career growth.
Subcomponent 1D would assist the Government to improve the quality of the training programs of the Academy of Public Administration (AAPA) as the main provider of Civil Service training by strengthening the Academy’s ability to develop and implement training programs, upgrading the Academy’s training facilities in Yerevan, and equipping the to-be-created regional branches in two other municipalities.
Introducing E-procurement (US$0.92 Million--IDA contribution US$0.62 Million) This component sought to address some of the systemic weaknesses in the procurement system, which were identified in the May 2003 CPAR and in particular the prevalence of non-competitive practices and over-reliance on single source methods. It aimed to establish the information technology infrastructure for E-procurement and the basis for the development of a full-fledged E-procurement system. Subcomponent 2A would assist the Government in creating the material base for E-procurement in the Central Authorized Body, the State Procurement Agency as well as 60 state agencies, 44 local Treasury units and the information centers in 10 Marz administrations and the Yerevan region.
6
Subcomponent 2B would finance the acquisition, adaptation, installation, configuration, and testing of E-procurement application software, including: (i) an automated internal procurement administration system covering planning, budgeting and recording of contract awards; (ii) a contract management system for the generation and management of contractual documentation and the procurement process; (iii) system-controlled requisitions, purchase orders, and receipt and payment documents, including e-catalogues for framework contracts; and (iv) a public awareness/outreach program to maximize the awareness and utilization of the E-procurement System. Strengthening the Chamber of Control (US$0.74 Million--IDA contribution US$0.29 Million) Component 3 would provide limited and focused support to the Chamber of Control (COC), the nation’s supreme audit authority, including the development of audit standards and manuals needed to enhance its ability to provide professional performance audit reports to the National Assembly (beyond the more narrowly defined traditional financial audits), training on the new standards, some essential equipment, and an updated web site.
Subcomponent 3A would assist the CoC prepare at least 10 performance audit standards including one on ethics, an Audit Manual for the CoC’s financial compliance units, an evaluation of budgetary programs, training and an internal code of ethics for CoC staff. Subcomponent 3B would finance workstations, ICT equipment and start up, and some furniture to the agency. Subcomponent 3C would expend a small amount for local technical assistance on the further development of the CoC’s website for posting CoC reports and information on its activities. Improving Policy Formulation and Implementation through Capacity Building (US$3.55 Million--IDA contribution US$3.42 Million) To address weaknesses in the policy formulation and implementation process in the central administration, this component (Component 4) would support the development of electronic document management systems and systems for tracking policy implementation, as well as pilot management innovations to enhance transparency and accountability and address priority capacity building needs in selected agencies. (It should be noted that this component appears ambiguously named in the sense that its activities, performance indicators, and funding are focused largely on capacity building with improved policy formulation and implementation, in most cases, assumed to result from these project inputs. As elaborated in Annex 2 some improvements in policy formulation and implementation can be discerned in some sub-components.)
7
Subcomponent 4A would support the introduction of an integrated document management system to enhance the effectiveness of policy development and implementation across the state administration. Subcomponent 4B would enhance the capacity of the Ministry of Finance and Economy in the areas of revenue policy, internal audit, and the Budget and Financial Management Training Center. Specific activities would include: (i) the provision of training to strengthen policy development in the area of revenue policy, including coordination systems with the Tax and Customs Services; (ii) strengthening the Ministry’s internal audit mechanisms through training and upgrading equipment; (iii) providing equipment to effectively link the Supervision Service [internal audit unit] of the MOF, the Supervision Service of the President’s Administration, and the Supervision Service of the Office of the Government; and (iv) enhancing the capacity of the Budget and Financial Management Training Center to effectively train staff members of budget departments across the public sector. Subcomponent 4C would support the Ministry of Economic Development and Trade (MEDT) to enhance its capacities in regulatory policy-making especially in the areas of standardization, certification and metrology and strengthen the Ministry’s capacity to communicate and coordinate with external interlocutors, domestic and international, public and non-governmental. Subcomponent 4D would focus on strengthening the external communications and outreach capacity of the Office of the Government, the office reporting to the Prime Minister and assisting in the management of the government’s decision-making process. Subcomponent 4E would strengthen MEDT’s standardization, metrological and certification functions in support of Armenia’s WTO accession by focusing on improving the regulatory framework and related technical standards, improving access to international technical standards and standards procedures, and addressing the essential hardware needs through the provision of equipment for testing laboratories.
Developing Capacity in Local Governments (US$2.85 Million- IDA contribution US$2.76 Million) To improve local self-government capacity to deliver public services as called for under the 2002 Law on Local Self-Government, this component would seek to address four related issues: (i) mitigating the consequences of administrative fragmentation; (ii) enhancing revenue mobilization; (iii) implementing a HRM system for the municipal civil service; and (iv) strengthening municipal data management. Subcomponent 5A would support the development of the legal and policy framework for Inter-Community Unions [grouping administrations and services to enhance efficiencies] to mitigate the impact of administrative fragmentation.
8
Subcomponent 5B would finance international and local technical assistance and training to advise the Ministry of Territorial Administration and Infrastructure coordination on building local capacity to meet new responsibilities for property and land tax collection. Subcomponent 5C would help design and implement a human resource management system for the municipal civil service. Subcomponent 5D targeted the development and deployment of an integrated, computer-based Management Information System, and by the strengthening of the national center for technical support and systems development in Charenstsevan. Project Management, Surveys and Communication (US$0.59 Million- IDA contribution US$0.54 Million) Noting that the other components involved a set of complementary but distinct public sector management reform initiatives, this final component sought to provide a unified project governance and management and monitoring of outcomes. Subcomponent 6A would provide funding for the project manager and essential support staff and functions. Subcomponent 6B aimed to fund periodic surveys of households, firms, NGOs and public officials to collect data on key performance indicators and obtain feedback from stakeholders on the reform process, as well as finance a communication campaign to explain project objectives to selected stakeholders to help build and maintain support and ownership for the reforms.
1.6 Revised Components No change 1.7 Other significant changes (in design, scope and scale, implementation arrangements and schedule, and funding allocations) Other significant changes impacting the project were related with three extensions of closing dates with the first for a period of 15 months to June 30, 2010, the second for 4 months to October 31, 2010 and the last one for 4 months to the de facto closing date of February 28, 2011. The main reason for the extension of the closing date was allowing sufficient time for completion of procurement of ICT systems, inter alia, roll-out of EDMS and customization of e-Tendering system.
9
2. Key Factors Affecting Implementation and Outcomes
2.1 Project Preparation, Design and Quality at Entry (including whether lessons of earlier operations were taken into account, risks and their mitigations identified, and adequacy of participatory processes, as applicable) The quality at entry was moderately unsatisfactory. This judgment balances the alignment with the Government’s public administration reform program and coordination of the program with adjustment lending and ESW work on the positive side against the very broad scope plus gaps in prior preparation for the implementation of the various ICT investments on the negative side. Strengths of the program at entry Leading up to the project and supporting its design was extensive economic and sector work, including a Civil Service Assessment (1998), an Institutional and Governance Review (2000), a Public Expenditure Review (2003), a country Procurement Assessment Report (2003), and a Country Financial Accountability Assessment (2003). Significant analytical work continued through the program, and helped refine the approach of the project. The project benefitted from years of close country policy dialogue as well as strong coordination between the Country and Sector Units. The project was complemented by several other investment operations, including health, education, social services, legal and judicial services, as well as various adjustment operations. These investment operations provided the technical assistance that was relevant to the development and implementation of the broader policy changes and allowed the Public Sector Modernization Reform project to focus on its core mission. Weaknesses First, while there was no formal revision of PDOs or the basic structure of the project, by 2007, it became apparent that the implementation of the various information technology subcomponents of the operation (which represented a large share of project financing) was challenged by a variety of technological and institutional factors. A major issue was that the basic connectivity infrastructure assumed in the PAD was not in place and its design and installation would have to precede full deployment of the electronic document management system, government-wide HRM management system, and E-procurement systems if fragmented and isolated systems were to be avoided. Interoperability of some of the investments was questioned and some efforts did not fully take into account a wide variety of uncoordinated IT initiatives in Armenia under way funded by the government, donors, and even other projects of the World Bank. Moreover, counterparts to some extent may not have fully understood the scope of the project, and plans for business process reengineering had not been fully embedded in the design of the ICT components. Consequently, overall project implementation was slowed and ISR ratings downgraded. The supervision team proactively brought in outside ICT expertise to review the project and its relations to the other initiatives, and engaged in intense dialogue with counterparts.
10
A remedial program of actions was agreed upon, adjustments made in funding allocations and procurement activities, and disbursements ramped up significantly within a year. Second, the objective of including a large array of components was to respond to client demands and needs, fill gaps left by other donors, and enhance a holistic approach to state modernization, in the view of the PAD. Yet experience with projects of this nature suggests that such a broad scope often exceeds the management capacity of government. Combining government-wide horizontal reforms with vertical capacity building efforts in a few ministries, addressing organizations at the central and regional levels of government, and supporting an agency (the Chamber of Control) independent of the executive branch stretched the management capability of the project teams. The project had five substantive components and 20 sub-components, which in turn were often broken down further (e.g. Component 1 had 13 separate sub-sub components). The size of anticipated funding of sub-components ranged from $20 thousand to $2.39 million (120 times larger). Project management needed to spend considerable time in monitoring and policing the myriad activities, educating project beneficiaries as to project goals, and in general seeking to ensure some unified approach to the project. Despite budget top ups by the Country Unit to add resources above the Supervision budget norm, the Bank team itself was challenged from the beginning by the coordination responsibilities of the operation. Third, it is also somewhat of a stretch to see the project bringing a holistic approach given the individual narrowness and overall number of issues requiring counterpart attention (scientific laboratory upgrades, internal audit training, HRM software system development, land tax data base transfers to municipalities, furniture for the Chamber of Control, etc.) While such complex operations may well on rare occasions have their place depending on the commitment, capacity, and adaptability of counterparts and Bank teams, it puts a large burden on supervision teams and carries a high risk of failure from one-off unsustainable investments. Finally, the determination of performance indicators was poor. This was especially bothersome during implementation as the vague nature of the Project Development Objective statement (“improved efficiency in public sector management”, followed by clauses on two operational aspects of how it would be achieved through piloting unnamed “innovations” and transparency) gave little concrete direction. Six “main project monitoring indicators” were listed on page 5 of the PAD, but none are PDO outcomes, only four appear in some fashion in Annex 3 (Results Framework and Monitoring) as intermediate results indicators (IRIs), and these nominally measure only three of five components. Two “main project monitoring indicators” do not appear in the Results Framework in Annex 3 and were not tracked in ISRs. The three PDO indicators relating to the civil service, financial management, and central and local administrations identified in Annex 3 are very general and non quantifiable (“transparent”, “more efficient, effective, and transparent”, and “more functional and effective”) so any determination of their achievement would be subjective. In Annex 3 there were 22 intermediate results indicators presented, although some major sub-components lacked any IRIs and in other cases indicators were unrelated to project activities (e.g. E-
11
procurement). In any case the government and Bank teams struggled to square IRIs with project activities (which presumably were closer to government priorities). Ultimately the original IRIs were supplemented by additional indicators better reflecting project activities.
2.2 Implementation As some of the design shortcomings became apparent the project supervision team attempted to address them during implementation. The government, with Bank support, moved within a year to replace an ineffective project director. A locally based Bank staff was contracted to strengthen daily supervision. Ratings of the project were generally realistic, most notably the downgrades reported in 2007-8. All performance indicators mentioned in Annex 3 of the PAD were tracked, and subsequently expanded by additional indicators. Assessments provided to management in ISRs appeared to be thoughtful, appropriately targeted, and supported by comprehensive Aide Memoires. By 2007 the team candidly reported on the ICT challenges and took steps through expanded technical assistance and dialogue with counterparts to successfully address those problems. The quality of implementation of the project was assessed by a special QAG review commissioned by the Public Sector Management Office (PRMPS) in 2007. The panel was impressed with the dedication and professionalism of the supervision team, but found that the project was “overambitious, too wide in scope with insufficient counterpart understanding, and overly equipment driven”. It noted that this lack of understanding and failure to define business process reforms “suggest hasty preparation and a failure to communicate project objectives and how to get there.” Although not explicitly stated, this may suggest that the QAG team viewed the Quality at Entry as no more than Moderately Unsatisfactory.
2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization Output and outcomes monitoring and evaluation were considered important aspects of project design, including a separate sub-component of the project. Counterparts ultimately monitored the entire set of performance indicators listed in the PAD text and annex. After the mid-term review additional intermediate results indicators were agreed upon, monitored, and reported in ISRs. Having some indicators with more apparent relevance than others (which were not cancelled), however, can lead to confusion for teams or outside observers as to which were the real metrics for measuring performance. The credit also contemplated a series of monitoring and evaluation efforts establishing baseline values and tracking progress over time. However the contracted outside firm did conduct a comprehensive review of project performance based on surveys, focus groups, and data for 2005-2009/2010. It sought to establish retrospectively reliable baseline information going back to January 2007 for Components 1, 2 and 4 (arguably the period when project implementation began in earnest) based on records as well as input from survey and focus group respondents. Surveys for Component 5 sought to reflect
12
information going back to 2005. While comprehensive, interesting, and indicative, confidence in the baseline values would be greater had the studies and surveys been done closer to project launch. In the event the 2009/2010 monitoring and evaluation effort appears to have been comprehensive and useful, providing input to project reporting and this ICR.
2.4 Safeguard and Fiduciary Compliance
The financial management arrangements in the PIU (Foreign-Financed Project Management Center), which included accounting, budgeting and planning, financial reporting, funds flow, internal control, and external auditing were developed early on and maintained an acceptable quality throughout the life of the project. Internal controls were judged to be strong and capable of providing reliable and timely information on the projects. Financial monitoring reports were always submitted by due dates and assessed as acceptable to the Bank. Audited financial statements of the project were always received by due dates with unqualified (clean) opinions issued. According to the Bank’s procurement team member, as well as periodic reports, the government implementation unit did an excellent job in preparing the terms of reference and bidding documents and procurement of the contracts generally proceeded without problems. Monitoring of project procurement and contract execution was careful, timely and detailed. Five post review missions were conducted with approximately 70% of contracts subject to post review. There was only one area of a major bidding problem when the first bid for development of the E-tendering contract solicited proposals far in excess of available resources. The proposed work program was therefore revised and re-bid successfully (an interruption in implementation leading to the last credit extension).
2.5 Post-completion Operation/Next Phase The project supported major improvements in core government management information systems (HRM, EDMS, and E-procurement) that enable more efficient policy formulation and implementation. It also contributed to modernization efforts in selected ministries and agencies, units, and a broad swath of local governments. An indication of the government's assessment of the project’s success and need to continue ongoing modernization efforts was the request for a follow up Public Sector Modernization II project which was approved on March 16, 2010. This operation builds on a narrowed set of initiatives of the first operation. The government’s project implementation unit has remained intact which should both facilitate smooth implementation of the new operation and provide institutional memory for any follow up actions required for investments under PSMP I.
13
3. Assessment of Outcomes
3.1 Relevance of Objectives, Design and Implementation This project’s objectives are fully relevant to country priorities and consistent with the Country Assistance Strategy and subsequent Country Partnership Strategy as highlighted in the two documents dating back to 2001. The Bank’s current Country Partnership Strategy for Armenia, approved on May 12, 2009 and covering FY09-12 aims at the two main pillars of mitigating vulnerability (related to the impact of the global economic crisis which hit Armenia hard due to its increased integration into the international economy) and strengthening the country’s medium term competitiveness. Improving public sector efficiency is highlighted as a core structural reform under the competitiveness pillar. The CPS notes that progress during the previous CAS period was real, especially in capacity building in selected public agencies such as the Ministry of Finance, HRM improvements, and strengthening local governments, but that more was needed in other agencies and in combating corruption and improving transparency. Consequently, the CPS highlights five areas of public sector strengthening: (i) tax and customs administration; (ii) a more integrated approach to public expenditure management; (iii) civil service reform deepening; (iv) expanding the scope of judicial reform; and (v) a comprehensive and coherent effort to strengthen conflict of interest controls at all levels of the government. The follow up Public Sector Modernization operation reflects the government’s commitment to continuing the reform effort and confidence in the Bank as a partner in this effort. This new operation aims at enhanced performance of public sector management for better service delivery by: (i) strengthening institutional capacity in policy formulation; (ii) maximizing the efficiency of human resources; and (iii) developing information systems for internal work flow and external communication.
3.2 Achievement of Project Development Objectives Despite the generalness of the Project Development Objective and design challenges for the indicators of performance, the project appears to have materially achieved its PDO. As detailed below, there have been clear improvements in public sector management efficiency stemming from: (i) the roll out (and not just piloting as called for in the PDO) of innovations such as the electronic document and human resource management systems in the bulk of Armenian government agencies (which has been facilitating other process and structural reforms, such as work planning and performance appraisal for the civil service, tracking communications online between government and citizens, introducing e-licensing, decentralizing recruitment of junior civil servants, launching functional reviews, and business process reengineering in a number of ministries) plus initiation of a modern E-procurement system; and (ii) improved transparency (both as an input to efficiency and end in itself) in operations through initiatives affecting government media and web site communication with citizens, enhanced public access to data not only on
14
their specific requests but also government policies and budget execution, and strengthening formal oversight through as internal and external audit organizations. With regard to the PDO outcome indicators listed in Annex 3 of the PAD, the weight of evidence suggests that: (i) merit and transparency are being applied and monitored by the Civil Service Council; (ii) financial management has been improved through strengthened external audit capacity (though full exploitation of this capacity may be spotty) and increased transparency of budget execution. Introduction of electronic public procurement by end 2011 will further enhance relevant project outcomes; and (iii) substantive improvements in local government operations and business processes in the central public administration. The electronic document management system enables real time effective workflow and communication within central and regional public administration. The work planning and performance management system, fully supported by Electronic Document Management System, enables developing and implementing policy in-line with Government strategies, measuring and motivating performance at organizational and individual levels. That said, difficulties in both the articulation of the PDO indicators and data collection implications need to be recognized in assessing the evidence that was collected: PDO Outcome Indicator 1: The management of the Civil Service of Armenia is transparent and based on merit principles that are consistently applied throughout the system. Annex 3 of the PAD called for surveys of citizens and public officials, plus supervision reports to provide the evidence. The operative goal for the end of the project was stated in Annex 3 as “Merit principles in recruitment and promotion are applied.” Evidence: Data was gathered in 2010 through surveys of applicants to civil service jobs. The information was broken down by year and method of hiring (competition or attestation alone), and collected from 537 people who may or may not have been hired ultimately. The summary table indicates that an increasing majority of applicants believe that recruitment is merit based. Separate survey data collected on efficiency of periodic qualification tests captured an increasing perception of efficiency, to which the project contributed through building capacity for the elaboration of more sector and job relevant questionnaires. The tests were the only source to inform merit-based career decisions before the implementation of performance appraisal system also supported by the project. The practice of periodical testing is expected to be discontinued in a couple of years after full roll-out of the performance planning and management system. (No information was gathered on the fairness of promotions in part due to the difficulty of devising a neutral and valid measurement method and this goal was not tracked in ISRs. Also the ISR dropped the goal of “increasing the proportion of civil service appointments made through merit-based open competitions by 50 percent annually” stated on page 5 of the PAD as such a goal was mathematically illogical.)
15
Question: In your opinion are the appointments of the civil servants in Armenia merit-based?
2007 2008 2009-2010 Yes 35.8% 37.6% 43.8% Rather Yes 25.7% 27.6% 32.3% Rather No 18.2% 16.0% 12.4% No 13.1% 11.2% 8.9% DK 7.2% 7.6% 2.6% Total 100% 100% 100%
Question: In your opinion is the periodical testing of the civil servants in Armenia an appropriate method for assessing relevance of professional skills for civil service position?
2007 2008 2009-2010 Yes 23.0% 21.9% 29.2% Rather Yes 26.7% 29.2% 35.5% Rather No 24.2% 18.1% 18.6% No 10.3% 14.9% 9.3% DK 15.8% 15.9% 7.4% Total 100% 100% 100%
PDO Outcome Indicator 2: The financial management control framework and procurement procedures become more efficient, effective and transparent. Annex 3 of the PAD called for surveys of businesses and officials (without target values) plus it listed project inputs relating to E-procurement, internal audit offices, CoC audit standards and manual as targets. ISRs at first made operational the PDO indicator by calling for increasing business firm satisfaction with public procurement. Later ISR identified a satisfaction rate of 80%. ISRs also called for approval of ten audit standards by the CoC plus staff trained on the new standards. Following the revision of indicators the requirement of staff training was dropped (reportedly because staff had already been trained). Evidence: As outlined below and in Annex 2, which covers implementation of the various components the specific inputs on E-procurement, internal audit, CoC audit standards and manuals, the input target values were largely met. Business satisfaction with public procurement was assessed through surveys of firms (sample of 341 firms) covering period 2005-2009/2010. The firm names were obtained from the State Procurement Service, data from official procurement bulletins, and other available sources. The following table and graph summarize the survey results, and indicate a satisfaction rate of about 80% and an increase over three years in the perceived fairness of public procurement. No similar perception based measures were obtained for wider financial control functions due to difficulty to identify proper respondents and smaller
16
scale project interventions limited to selective initial capacity building efforts that are being expanded through follow-on operations. Estimates of equal conditions and procurement competition results fairness for all participants of state procurement competition.
Yes Rather
yes Rather
no No DK Total
Are there equal conditions for all participants of state procurement competitions?
51.3% 29.4% 5.9% 7.7% 5.7% 100.0%
In your opinion are the state procurement competition results fair?
46.3% 30.2% 6.5% 6.2% 10.8% 100.0%
How has fairness of state procurement competitions changed during last 3 years?
PDO Outcome Indicator 3: The organization of Central and local administrations is more functional and effective. This statement is from the beginning of Annex 3: Results Framework and Monitoring. However later in the same annex the PDO indicator is presented as Local self-government in Armenia is empowered with sufficient organizational structure, technical capacity and fiscal autonomy to exercise their responsibilities as established by Law. The ISRs track the first statement, but the specific operational targets changed from earlier ISRs’ “Citizens and government can more effectively monitor policy decisions and policy implementation as measured by surveys; government institutions use results to improve availability of information throughout the project” to in later ISRs “83% of business community is satisfied with the appropriateness of services, a 15% increase in general satisfaction with local government services of respondents over baseline”.
16.7%
30.8%
3.8%
48.7%
0%
20%
40%
60%
Increased Almost did notchange
Decreased DK
17
Evidence: It is difficult to provide clear evidence in line with this PDO Outcome Indicator. First of all it is not clear what the word “organization” refers to in the PDO Outcome Indicator sentence—an organization chart, the mobilization of government resources, or the government as an organization itself. Secondly there are two PDO outcome indicators listed in Annex 3. Thirdly neither indicator is quantifiable as written. Fourthly the operational targets listed in ISRs evolved over time and narrowed onto local government (borrowing the concept of increased satisfaction with local government services from the last “main project monitoring indicators” listed on page 5 of the PAD text and adding a 15% numeric target). Finally, no surveys were conducted to explicitly ask businesses and citizens their views of improvements in services —rather there was a survey of officials and citizens in the four pilot Inter Community Union municipalities on the nature of government services. This provides baseline information going forward. There was no baseline information to judge project impact in these communities. In any case the project was focusing on internal public administration capacity building and it is unclear how soon citizens would have perceived improvements in service delivery at the local level. In sum, given the conceptual and data collection flaws for this PDO indicator reliance must be placed more on the intermediate results indicators linked to components to identify improvements in overall government administrative effectiveness. Those efficiency improvements under the substantive components (which are further detailed in Annex 2 of this report) include the following: Human Resource Management: With the support of the project Armenia has developed the policy and material basis for a modern HRM system. An electronic based human resources management information system has been created and installed in all 45 agencies subject to the Civil Service Law. It contains and interactively manages a comprehensive system of electronic personnel files, annual performance evaluations, and linked staff training plans. Recruitment of staff is more competitive and transparent as all vacancies are listed on the Civil Service web site. Selection has been improved through agency relevant testing using centrally approved methods (unlike previous practice where agency scope for discretion in selection criteria was broader). The program of periodic attestation is being integrated into the HRMIS. Training for public employees (both on site and long distance) has been significantly upgraded. The capacity of the Civil Service Council to oversee the civil service system has been strengthened, as well as the capacity of the Academy of Public Administration. Introducing E-procurement. The component aimed to create the material base within government for an E-procurement system as well as the procuring and installation of such a system. The component’s aims were partially achieved, and represented a major effort within the project, including the last extension of project closing date in order to complete a critical E-procurement sub-component process. By the end of the project the core elements of an E-procurement system had been designed, tested, and transferred to government. The E-tendering system is currently under phased implementation—it was loaded onto servers by end May 2011, will be publicly launched in October 2011, and its use will be mandatory for all government agencies as of January, 2012. The E-tendering system represents the heart of an E-procurement system, involving in electronic gathering
18
and posting of government tenders for goods and services, transmission of information to registering suppliers, and submission of bids (specifications and costs) all accessible on a government web-site. Strengthening the Chamber of Control. The stated objectives of this financially small component were partially achieved. The project provided support to the CoC for the development and internal review of modern performance audit standards covering ten areas, as well as a code of ethics for CoC staff. All have been approved by the Board of the Chamber including its Chairman and are in effect. An audit manual for financial compliance was similarly developed, approved and applied. The audit standards and methodologies formally guiding CoC activities are in line with international practices. The anticipated equipment and furniture purchases were made and the web site of CoC was upgraded and is now available to the public at www.coc.am. The quality of CoC operations, however, does not yet fully reflect the capacity building inputs. Improved Policy Formulation and Implementation through Capacity Building. This component (largely funding ICT investments) was generally designed to improve the information base for policy formulation and timelier and more effective policy implementation, (As indicated previously this title is ambiguous if not misleading as improved information inputs or electronic capacity are not sufficient for improved policy formulation, adoption, or implementation. So too for the entire project--the technical assistance and funding provided can enhance public sector efficiency (in terms of speed, coverage, cost, etc.) without necessarily improving the effectiveness or equity of government. Although project component design explicitly would not guarantee immediate results to improve policy formulation and implementation, some improvement was achieved in some areas due to the teams’ efforts to bring planned piecemeal efforts together to ensure synergies and alignment with general component targets. The successful creation, under this component, of an Electronic Data Management System (EDMS) digitizing government operations and moving toward a paperless administration is a key project outcome. The related software that was developed—termed Mulberry—is used by 8000 staff daily in 32 government bodies. EDMS is: (i) impacting the speed of government activity, accuracy of information, and internal consistency; (ii) permitting widespread business process reengineering and functional reforms in agencies; (iii) freeing government officials to focus more on better informed policy deliberations and less on routine transaction processing and monitoring; and (iv) transforming citizen/government interaction (electronic application for government services, tracking of government responses to those requests, public oversight of budget expenditures, etc.). EDMS has become the backbone of Armenia’s E-government effort, underlying the reforms in human resource management and capacity building supporting other components of the project, as well as other public administration modernization efforts fully funded by the government and/or other donors. It can be expected that improved document management and streamlined business processes will also contribute to the Chamber of Control’s efforts to audit both financial compliance and performance. EDMS is rightly considered a flagship of the project and pillar for current and future public administration reforms.
19
Complementary reforms within the component also aimed to strengthen the basis for policy formulation and implementation through improvements in public communications. With project support the government has created a better functioning press operation at the center of government charged with overseeing a modern and more sophisticated effort at communications with citizens. The press center (with its expanded communications capacity, regular briefings, various information outlets available to the press, etc.) in the view of counterparts is increasing the flow and quality of information from government, enhancing the transparency of its operations, improving the quality of public debate on national issues, and promoting increased citizen awareness of, and feedback on, government operations (which in turn can improve policy formulation and implementation). Along with the processes supported by EDMS, it is judged (at least by government officials and local Bank staff) as continuing a trend toward more participatory government and rebalancing a legacy of one way communication flow and top down authoritarian public administration inherited from the Soviets. Two related sub-components with the Ministry of Economic Development (subsequently renamed the Ministry of Economy) sought to improve policy making, communications with external partners, as well as the technical capacity in national standardization and certification to better meet the country’s obligations to the World Trade Organization (WTO). Rapidly addressing so called “technical barriers to trade” relating to health and safety procedures and testing, etc. was considered a top priority for the country and its strategy to better integrate into the global economy. Technical assistance supported initial significant steps toward the harmonization of technical regulations and testing methods with international and European standards as well as policy making itself in the area of standardization. Finally, support to the Ministry of Finance was most helpful in strengthening the Ministry’s Training Center and its capacity to train staff government-wide on modern public finance subjects and procedures. Training was provided in specific areas such as internal audit and the project helped establish pilot internal audit units in line agencies. The project built both technical capacity and the material base for the training institution to deliver training on various specific subjects of PFM. Overall, despite the fact that "umbrella" title for the project component was wide and ambitious, due to extra efforts at implementation stage (beyond the original planned activities) the team ensured:
Improved government capacity for communicating policies with the public through modern press center,
More transparent budget execution,
More effective workflow through reengineered business processes and timely communication of government decision via e-government system covering central government and territorial administration, and
Institutionalizing and automating work planning in-line with government strategies through the work planning and performance management system.
The latter is the most critical and valuable change, since it ensures not only planning individual performance targets aligned to organizational and higher government targets,
20
but also involves civil servants in consultative process of policy formulation to achieve this. Further, the performance management system motivates achievement of the relevant targets against which individual and organizational performance is assessed. Developing Local Government Capacity. A major initiative in Armenia has been to devolve responsibilities and taxing authority to local governments which are closer to citizens and presumably better able in the long run to manage local service delivery. Near the end of project preparation and in response to the client’s request, this component was included to help support the decentralization program. Progress under several areas has been notable. For example, data on property and land taxes have been transferred to nearly all local communities, pilot programs have been initiated to improve human resource management at the local administrative level (to in part better manage the increased responsibilities for tax collection), and an integrated, computer based management information system has been designed and installed in over 200 municipalities, including the largest 60. Consequently the long term process of institutional capacity building in municipalities is well launched with the support of the project. One sub-component has not advanced as parliament has not approved legislation mandating involuntary Inter-Community Units (designed to compel consolidation of services in neighboring small municipalities) so work on the necessary regulatory framework could not commence. Nevertheless there appears to be a voluntary process of consolidation underway, which is benefiting from the other investments within this component.
3.3 Efficiency As an institutional capacity building project, quantitative projections of rates of return are not applicable and were not developed during project preparation.
3.4 Justification of Overall Outcome Rating Rating: Satisfactory The substantial advances highlighted above have laid the basis for more efficient public sector management. This has been achieved through: (i) an improved material base (hardware, software, and communications infrastructure) for core management information systems underlying better policy formulation; (ii) more transparent and two way communications with citizens and other stakeholders; and (iii) targeted improvements in the operations of several key agencies or functions impacting government performance. Whatever the specific challenges of poorly designed performance indicators the project can fairly be judged to have met its goals warranting a rating of Satisfactory. Indeed in some instances, such as HRM and EDMS, it exceeded expectations. Several of the modernization efforts will be deepened under the follow on PSMP II operation.
21
The rating takes into account some shortfalls. These include the inability of the project to complete the regulatory framework for the Inter-Community Union legislation as it has not yet passed the National Assembly and an apparent decision not to vigorously pursue training and other investments to enhance coordination between the internal revenue and Custom Departments. These, based on their nature (small relative funding size as well as modest policy importance) do not warrant downgrading the project outcome to Marginally Satisfactory (although on the other hand they do suggest that a Highly Satisfactory rating would be too generous). Similarly the progress toward creating the material basis for an E-procurement system has met the expectations of the PAD, even though the roll out of the core E-tendering system is only occurring now (a pace which is normal for global practice). Had the project been formally restructured and performance indicators and some components refined accordingly the project may have earned a Highly Satisfactory rating. Nevertheless, judged against other public sector projects this operation is very solid and undoubtedly one of the most successful in the portfolio.
3.5 Overarching Themes, Other Outcomes and Impacts
(a) Poverty Impacts, Gender Aspects, and Social Development There is no direct poverty, gender or social impact on the population. Indirectly, however, the project can be judged as contributing to more efficient management of public resources, improved policy formulation and implementation, more transparent government operations, and improved two way communications between government and the private sector—all of which should benefit citizens. (b) Institutional Change/Strengthening Institutional strengthening is the core focus of the project. As indicated above and in Annex 2 below, it has supported significant improvements in government ICT systems, training, and specific agency activities all fostering more efficient and transparent public sector management. Most notably the civil service system has been strengthened by the HRMIS facilitating a more merit based system as a whole starting from competitively filling vacancies, through personnel evaluation, linked staff training, better oversight of agency personnel actions, etc. EDMS is the backbone of Armenia’s advance toward E-government, with all the benefits implied in terms of speed, accuracy, efficiency, and more effective public administration. The investments in communications (press centers and public relations units, improved web sites for the flow of information and citizen requests for services, and public monitoring of government operations) will strengthen institutions’ ability to refine and implement policies.
3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops
In line with Sub-component 6B a comprehensive “Monitoring and Evaluation of the Public Sector Modernization Project” was contracted and covered the period 2005- 2009/2010 and a final report delivered in April 2010. The report focused on intermediate results indicators then being tracked in ISRs. The data was gathered, analyzed, and presented on 27 topics using surveys, focus group discussions, review of government
22
documents, interviews, etc. The data helped inform preparation of this ICR as well as implementation of the project in its final months. Efforts were made by the report team (a private company) to capture data from earlier years (through time specific questions and data assessment) for few additional indicators for local government added after MTR of the project. Its usefulness would have been greater had earlier similar studies been conducted as indicated in the PAD.
4. Assessment of Risk to Development Outcome
Rating: Moderate The sustainability of project achievements and thus the Development Outcome seems likely. The investments in ICT and resulting improvements in HRM, the core features of e-government including EDMS and E-tendering, as well as improvements in training and specific capacities in several areas seem to be largely uncontroversial, representing non-partisan building blocks for any public administration. Save a severe national crisis it is unlikely that any government would give up the cost efficiency and effectiveness of most of the measures supported by the project. Of course any future government could theoretically cut citizen access to government information, curtail the Chamber of Control’s efforts to increase performance auditing, or limit the transparency and merit principles in public hiring or procurement, but that is a risk that seems low given the trajectory of Armenia’s progress over the last two decades and deep commitment to the global economy and association with the European Union. A more practical complication might stem from an unwillingness or incapacity to maintain/upgrade the modern systems, perhaps due to budget constraints. But with time government and public comfort with, and indeed reliance on, the new systems would likely mitigate against any sustained rollback or degradation of those systems.
5. Assessment of Bank and Borrower Performance
5.1 Bank Performance
(a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Unsatisfactory Rating the Bank’s performance at entry is a challenge and ultimately a close judgment call. The Bank drew on a long list of relevant ESW, extensive preparation and dialogue, a strong team of public sector specialists, and close collaboration with counterparts and other donors to design the project. The primary focus on ICT investments laying the basis for fundamental improvements in government capacities was the underlying thrust of the project at conception and in implementation as indicated by the amount of resources dedicated to these components. The successful results of the project in the core areas of EDMS, HRM, and fully anticipated in E-procurement validate the underlying quality of the project at preparation.
23
However, because the scope was too wide, some targets too ambitious, the articulation and consistency of results indicators inadequate, and some preparatory activities required supplementation during implementation a moderately unsatisfactory rating seems appropriate. The inclusion of activities somewhat tangential to state modernization burdened supervision. This may have reflected a close collaboration with government and willingness to accommodate a partner with a proven track record of performance. Additionally a limited number of operations available under tight IDA funding constraints may have inclined Bank management to accept additions to the project in order to provide flexibility to tackle discreet, financially small, targets of opportunity. Whatever the reason the broad scope deepened the challenges of execution and supervision. While it would be a stretch to claim that the project reined in a disparate state modernization effort by giving it a holistic vision, the project over time did become a backbone of the state modernization effort which, according to counterparts, improved the internal coherence of public administration reform and provided a basis for better prioritization of some donor activities. Nevertheless there may have been a tendency to oversell the possible benefits of the operation in order to secure a place in the lending queue. Another issue is the combination of a quite general PDO with a longish list of indicators of varying quality essentially trying to make the objective more concrete. This appears to have been a problem for many public sector operations prepared at the time where the development of performance indicators seemed to follow rather than lead project conception, and where “doing something” to improve governance exceeded the Bank’s experience in measuring such improvement. Unsurprisingly the performance indicators had to be supplemented during supervision1. (b) Quality of Supervision Rating: Satisfactory The Bank mounted a solid mixture of headquarters and field based staff for supervision. The project was supported from the early stages by a full-time field based Public Sector Specialist and a strong headquarters based team including specialists in information technology, public administration, civil service reform, and other specialists as required. The team established fluid and frequent contact with counterparts throughout supervision. It sought and obtained alternative sources of funding for the project’s special supervision requirements and found the needed expert assistance in several areas. The Country Director during the bulk of implementation, drawing on extensive experience with public sector modernization projects within the Bank, provided timely and deft policy support,
1 The coding of sector themes also seems somewhat arbitrary and in places at some variance with the actual composition of funding or expressed project goals.
24
as well as additional budget resources reflecting her appreciation of supervision requirements. The team was proactive in identifying and attempting to find solutions to the challenges which arose during implementation, most notably in ICT. Problems of duplicating investments with little attention to questions of inter operability as well as unanticipated problems surrounding communications infrastructure for the HRMIS and EDMS systems (and which slowed the pace of procurement and disbursements) were identified within two years of effectiveness (and one year after real project launch due to lagging performance by the then counterpart project director). The supervision team brought in high quality ICT expertise to join the team and assess ICT technical and policy requirements. The resulting recommendations were followed and the government and Bank made a concerted effort to better integrate the disparate ICT programs under the Public Sector Modernization office in collaboration with technical units within government. As a consequence the current architecture for E-government covering E-procurement, HRMIS, and EDMS among others is in place with good connectivity to all levels of government and geographic regions. The willingness of the Bank to stay the course in the project, despite the downgrades in ISR ratings and slow disbursements, was not based on inertia or lack of management attention, but rather an informed view of the challenges and risks and confidence in the teams (by Country and Sector management) to resolve the issues. As indicated above, the 2007 QAG review of the project praised the dedication and professionalism of the supervision team. Supervision was Satisfactory despite a decision not to formally restructure the operation and adjust the PDOs and/or performance indicators to evolving circumstances. There is a fairly extensive paper trail of contacts with ECA Operational Services and Quality unit as well as Country and Sector Management, but the consistent response was to update the performance indicators and note for the record those no longer considered valid (e.g. developing the regulatory framework for Inter-Community Units). The needed adjustments were judged insufficient to warrant, under then practices, a full restructuring. Moreover, country counterparts apparently opposed a restructuring as potentially opening a variety of issues with the multitudinous beneficiary agencies with their own wish lists of additional activities plus an unwelcome diversion of energy and focus. The government insisted that restructuring the Local Government component by dropping the focus on the Inter-community legislation (stalled in the parliament) would have sent an unwanted political signal. Thus in total there were numerous reasons for not restructuring the project although with hindsight and under current guidelines such a restructuring should have been pursued. That said, a redesign of performance indicators and tweaking of some sub-components would not likely have impacted much of the project’s success in meeting or exceeding most targets and improving public sector management efficiency.
25
(c) Justification of Rating for Overall Bank Performance Rating: Satisfactory The Bank provided strong support in terms of preparation and supervision. The vision of improved efficiency in public sector management was achieved in substance and exceeded in many areas. The overly broad scope of the project, shortfalls in preparation, and decision not to restructure the project do not undermine recognition of the project’s Satisfactory outcome and the Bank’s critical contribution to that result. The Bank has been a good and steady partner to Armenia in this area, as highlighted by the government’s decision to obtain a follow up technical assistance operation (even in a time of extremely strong balance of payments pressures which might have led to decisions to seek faster disbursing resources). The project demonstrated the strengths of the Bank—technical and country expertise, commitment to project goals and the partner, and willingness to go extra lengths to ensure a successful project.
5.2 Borrower Performance
Rating: Satisfactory (a) Government Performance
The government performed as expected—with diligence, commitment, and adaptability. After a slow start it moved to install strong project leadership which has remained in place. Internal communication and coordination appears to have been fully adequate. Procurement and financial management of the project were also satisfactory. The counterparts interacted well with the Bank team and evolved into a partnership which helped identify and correct emerging problems, refine project activities as required, and facilitate overall public administration policy management within government. The government made timely decisions to utilize the positive results of investments in HRMIS and EDMS to expand beyond pilot phases and install them much more widely within the broader public administration. Commitment to the reforms continued and even intensified with changes in senior government leadership. The decision to drop the development of a public landline connection to the regions and instead to rent the capacity on an existing private sector WAN freed the project resources for other purposes. It also demonstrated a pragmatic approach to technical issues as well as openness to public/private collaboration. The government’s efforts to facilitate local government capacity building through voluntary inter-community associations showed a commitment to the goal of improved service delivery even if blocked on one means for promoting it. (b) Implementing Agency or Agencies Performance Rating: Satisfactory Despite the broad array of beneficiary agencies and activities, implementation of the project components and sub-components was rather uniformly satisfactory. The Chamber of Control met all performance targets. The Civil Service Council and Armenia
26
Academy for Public Administration met, and in the former’s case, exceeded project targets. The Ministry of Economic Development successfully executed or oversaw several sub-components. The unit charged with the project’s procurement and financial management performed well. The Ministry of Finance’s Training Center was significantly improved and internal auditing is spreading within the public administration. (c) Justification of Rating for Overall Borrower Performance Rating: Satisfactory The borrower implemented a multi-faceted and technically challenging institutional strengthening program. Targets were largely met or exceeded and the substantive impact in terms of modern information systems, improved human resource management, more accurate and transparent data, as well as substantive improvements in several discrete areas of government operations are real and sustainable. A broad inside the black box capacity building operation has indeed improved the basis for Armenia’s policy formulation and implementation capacity. Better HRM and document processes, improved information inputs, and more transparency in and of themselves do not guarantee better governance outcomes. But they do improve the tools for achieving them.
6. Lessons Learned More narrowly focused operations limit the complexity of implementation for the borrower and supervision breadth for the Bank. In this operation the broad extent of the operation stretched the capacity of government to fully coordinate and execute. It required a substantially increased supervision effort (and funding) by the Bank. While largely successful in this case because of the commitment and capacities on both sides, the question remains whether opportunities for deeper reforms clustered around fewer components were missed. Plugging financing gaps or responding to client requests are frequent factors in Bank operations, but sometimes at the cost of coherence and dispersed efforts. Greater candor in identifying the heavy ICT focus of the operation might have alerted management and government to the need for enhanced project preparation. Basic investments in ICT for HRMIS, EDMS, E-procurement are prosaic, technically complex if not mysterious, and somewhat boring for most people (to be frank). They are also unavoidable for institutional capacity building for any organization and government. When done well and leveraged to promote changes in business practices, interactions with customers (citizens), accountability, etc. the investments can serve as the basis for revolutionary improvements in government operations and service delivery. However there appeared to be a bias against reporting in detail the extent of planned ICT investments in this operation. This in turn can obscure for management and government any gaps in the quality of project preparation, the ICT strength of task teams, potential interactions with other investments and projects, and serious issues of technology options, cost trends, or risks. Undue focus on the expected benefits or output indicators or high flown goals of integrated state modernization can mislead stakeholders away from a
27
focus on critical decisions and investments in ICT. The decisions are ultimately made, but often with delays and the need to educate officials on the true nature of the project. In this project ICT preparation was less than adequate with real impacts on the speed, scope, and cost of project implementation. While project objectives were met, they may have been exceeded had additional time and money not been required during implementation to retrofit parts of the ICT program. Ideally this work would have been fully completed in preparation had management been fully cognizant of the predominant ICT character of the project. A dearth of task team experts knowledgeable on ICT technical, policy, and market evolution issues and their integration may be a Bank-wide problem. Vague and generally worded Project Development Objectives can hide underlying conceptual problems, lack of clarity, and lead to monitoring and evaluation difficulties during and after implementation. Improving the efficiency, effectiveness, and transparency of government are worthy aspirations but cannot serve as the basis for judging value for money or provide much of an operational guidepost for counterparts, teams, or the Board. When accompanied by sloppily conceived and/or poorly written outcome and intermediate results indicators the burden on client and Bank supervision team is exponentially increased. This is especially ironic in operations which genuinely seek to improve administrative processes, transparency, and better accountability to citizens. Restructuring should be the default position for mid-term reviews of complex institutional capacity building operations. It appears in this case, as in many similar projects, that mid-course corrections are inevitable. Hence there should be a presumption on the part of teams and management that restructuring is required. This might help overcome counterpart resistance by lowering any perceived negative public fallout of restructuring, focus the attention of teams earlier on to the specific priorities for adjustment, permit more flexibility in project design if restructuring is viewed as a more acceptable and normal development, help client countries adapt projects to their evolving needs rather than the reverse, and provide more transparency in operations to the Board. Mid-term reviews themselves should be treated by sector and country management as important events (rather than somewhat expanded supervision missions) requiring more active participation of managers or outside donor funded evaluators if feasible. In the case of this operation the mid-term review did lead to important course corrections, but this reflected the strength and commitment of the task teams rather than an institutionalized presumption of the need for change.
28
7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners
(a) Borrower/implementing agencies The Borrower’s Completion report was submitted to the Bank on June 21, 201. There were no differences in views of the ICR task team and the counterparts as confirmed by Borrower comments received on September 13, 2011. (b) Cofinanciers Not Applicable. (c) Other partners and stakeholders (e.g. NGOs/private sector/civil society) Not Applicable.
29
Annex 1. Project Costs and Financing
(a) Project Cost by Component (in USD Million equivalent)
Components Appraisal Estimate (USD millions)
Actual/Latest Estimate (USD
millions)
Percentage of Appraisal
STRENGTHENING CIVIL SERVICE MANAGEMENT 2.45 2.47 101
E-PROCUREMENT 0.85 0.41 48 STRENGTHENING CHAMBER OF CONTROL 0.71 0.32 45
IMPROVING POLICY FORMULATION AND IMPLEMENTATION
3.13 4.99 159
DEVELOPING CAPACITY IN LOCAL GOVERNMENT 2.55 1.87 73
PROJECT MANAGEMENT 0.54 0.96 177
Total Baseline Cost 10.23 11.02 108
Physical Contingencies 0.00
0.00
0.00
Price Contingencies 1.03
0.00
0.00
Total Project Costs 11.28 11.02 98 Front-end fee PPF 0.00 0.00 .00 Front-end fee IBRD 0.00 0.00 .00
Total Financing Required 11.28 11.02 98
(b) Financing
Source of Funds Type of Cofinancing
Appraisal Estimate
(USD millions)
Actual/Latest Estimate
(USD millions)
Percentage of Appraisal
Borrower 0.45 0.71 158 International Development Association (IDA) 10.15 10.32 102
Bilateral Agencies (unidentified) 0.68 0.00 0
30
Annex 2. Outputs by Component Strengthening Civil Service Management: Highly Satisfactory
Intermediate Results Indicators, Targets, and Accomplishments of Component 1
Indicator--Standard tests for recruitment processes produced and published Target--Tests designed and approved by the CSC. New tests applied by all institutions subject to the Civil Service Law (CSL).
Achieved Under CSC leadership, capacity was built in all 45 civil service agencies to develop, apply, and periodically update sector specific tests. Tests, approved by the CSC, are applied during recruitment and attestation for all institutions under the CSL.
Indicator--Standard procedures for performance evaluation produced and published Target-- Performance evaluation system evaluated and finalized, linked with HRMIS.
Achieved Performance Evaluation procedures were developed (adopted by Government Protocol Decree Number 180, April 30, 2009), applied, linked to staff work plans, automated through EDMS, and rolled-out to the whole CS system (the reason EDMS was used for automation instead of HRMIS is that it allowed full integration with workflow).
Indicator-- Information on vacancies in all institutions made available through the CSC web-site and information center Target--All vacancies available at the CSC information center.
Achieved All vacancies are announced through the CSC website and Information Center. All new recruitment is conducted under HRMIS. The CSC is presently testing implementation of attestation through HRMIS. www.csc.am/index.html (in Armenian) www.csc.am/eng/index.html (in English, sample)
Indicator-- Recruitment and attestation centers equipped to allow better oversight over recruitment and attestation Target--Assessment centers fully functional in all institutions under the CSL and linked to the CSC.
Achieved Assessment centers are functional in all institutions under the CSL and linked to the CSC.
31
Indicator— HRMIS system installed and effectively providing data to the CSC on recruitment and career planning Target--HRMIS functioning for all institutions including a career tracking system
Achieved HRMIS functioning for all institutions including a career tracking system
Indicator—CS training curricula reformed and training programs integrated in career development. Target--Training records integrated in HRMIS.
Achieved Training plans based on performance evaluation are linked to HRMIS and EDMS.
Achievements under this component met, or exceeded expectations. An electronic based HRM system has been created, leading to a comprehensive system of electronic personnel files, annual performance evaluation, and linked staff training planning. Recruitment of staff is more competitive and transparent based on improved and more agency relevant testing and attestation programs and training (both on site and long distance) has been significantly upgraded. Where the project anticipated only piloting, in several cases the advances have been rolled out to all agencies in the civil service system which are largely civilian and excepting health workers and teachers. (As the various sub-components below complemented each other, it can be in some instances somewhat arbitrary to attribute specific contributions.) 1A: Draft Civil Service Development Strategy and complete Regulatory Framework: Satisfactory The stated objectives of this subcomponent were achieved. The 2010-2015 civil service Development Strategy was submitted to senior government authorities for review, and approved by Public Sector Reform Commission on August 4, 2010. It outlined a series of measures, based in part on a careful review of global best practices, to enhance transparency of the system, refine recruitment of qualified professionals, apply more effective performance management, motivation and training programs, and establish a flexible system for staffing also balanced from the gender and social justice perspective. The strategy is the basis for reforms being funded in part by resources from the follow up Public Sector Modernization II project. Moreover, the regulatory framework for the Civil Service Law approved in 2001 was advanced with support from project funds and materialized in form of the draft “umbrella” Law on Public Service, currently before the National Assembly. 1B: Strengthen the Civil Service Council: Satisfactory The subcomponent successfully strengthened the Civil Service Council which oversees the HRM system. Specific project support to the Council included technical assistance for the design of the integrated HRM software, input on revisions to training and
32
attestation systems, competitive recruitment guidelines, and personnel evaluation and rating methodologies. The project also funded ICT equipment including computer hardware for the Council, communication links to the 45 ministries and other agencies in the Civil Service System, as well as furniture to upgrade the working conditions within Council premises. Another critical aspect of HRM reform supported by the subcomponent was funding for a communications program to explain the ongoing reforms to public employees within government and other external stakeholders with the aim reducing misinformation and the inevitable resistance to change. Information centers were established or upgraded in most CSS agencies, supporting, among other things, efforts to explain the nature of the civil service reforms to staff, promote more citizen awareness of their rights, and provide information on the new EDMS system which had a significant impact on public access to data and interaction with government agencies. The sub-component also helped increase CSC oversight of the public information centers to increase the uniformity (and accuracy) of information provided through agencies and enhance the government’s broader transparency goals. The government believes the improved ratings for public sector performance measured by citizen surveys reflects in part efforts to improve communications and public understanding of government operations. 1C: Human Resource Management and Assessment/Attestation systems established: Highly Satisfactory With regard to the design and establishment of modern HRM and attestation systems, achievements under the subcomponent were highly satisfactory. Under the terms of Government Decree Number 180 (April 30, 2009) work plans and performance appraisals for all civil servants are automated through the EDMS and applied to each civil servant. Guidelines were issued on the operation of the program detailing in part the roles of immediate supervisors and civil servant in discussing performance effectiveness and decisions on possible remedial actions including training, e-education, professional literature to review, etc. Supervisors are enjoined to meet individually with staff to review performance and design a business related training program. These merit based human resource management principles and systems have been rolled out to parallel public services. All will be regulated under umbrella legislation on public service currently before the legislature. Indeed the civil service administration is considered to be more advanced than the policy planning system and procedures within the government as a whole. (An IDF grant is currently supporting efforts to design an enhanced system of Government results monitoring linked to agency business processes, work plans, and staff performance assessments developed under the project.) Assessment centers (with space and equipment) have been established or upgraded in 45 agency centers to conduct pre selection assessments as well as periodic tests of existing staff capabilities. Officials note that the project helped develop transparent and consistent methodologies to help guide individual agencies in designing and applying their agency specific tests, facilitating Council oversight of merit hiring practices
33
1D: Improve Academy of Public Administration (AAPA) training program: Satisfactory
This sub-component successfully strengthened the Academy of Public Administration. Credit resources funded a wide variety of technical assistance and goods. Support was provided for curricula refinements, development of long-distance learning and its modules, modernizing course presentations and methods, test booklets, training manuals for teachers, classroom equipment including computers and furniture, and an upgraded physical and electronic library. Officials underline the importance of intensive HRM training provided to 200 plus personnel staff in line agencies (including the respective chiefs of staff) as critical to improving the units’ ability to implement the new HRM policies Introducing E-procurement: Moderately Satisfactory
Intermediate Results Indicators, Targets, and Accomplishments of Component 2
Indicator— The E-procurement system is introduced and operational Target--Basic elements of an E-procurement system are operational.
Partially Achieved E-tendering system developed, tested, transferred to government in February 2011 and loaded onto servers by end May 2011. It will be publicly launched in October 2011, and its use will be mandatory for all central government purchasing units, ten regional offices, and potentially major local governments as of January, 2012. It represents a critical step toward full E-procurement system as it underlies the core of E-procurement in terms of electronic solicitation and receipt of bids in a transparent auction for public contracts. In the future additional modules relating to contract negotiation and execution will be added to the system
Indicator-- The number of suppliers bidding for government contracts increases by 30 percent from the baseline by the end of the project. Target--30% Increase (527 firms or more)
Achieved The number of suppliers has increased by over 400 percent to 2192 firms in 2010. Increase, however, is unrelated to project intervention.
Indicator-- Time taken for average procurement exercise is reduced by 25 percent from the baseline by the end of the project. Target--25% Reduction (38 days or less)
Partially Achieved Time taken for average procurement in 2010 estimated at 45 days, a reduction of 12 percent. Decrease, however, is unrelated to project intervention.
Indicator— The number of sole source Achieved
34
contracts is reduced by 25 percent from the baseline by the end of the project. Target--25% Reduction (1081 contracts or less)
Only 380 sole source contracts in 2010 (74 percent reduction). Decrease, however, is unrelated to project intervention.
The component aimed to create the material base within government for an E-procurement system as well as the procuring and installation of such a system. The component’s aims were largely achieved, and represented a major effort within the project, including the last extension of project closing date in order to complete an E-procurement sub-component process. By the end of the project the core elements of an E-procurement system had been designed, tested, and transferred to government. The E-tendering system is currently under phased implementation and its use will be mandatory within the government by January, 2012.2 The E-tendering system represents the heart of an E-procurement system, involving in electronic gathering and posting of government tenders for goods and services, transmission of information to registering suppliers, and submission of bids (specifications and costs) all accessible on a government web-site. The PAD and ISRs included other intermediate performance indicators relating to improvements in public procurement (reduction in the average time for a procurement exercise, reduction in the number of sole source contracts, and an increase in the number of bidders). The last two targets were exceeded and the first partially achieved. Achievement of the targets, however, was unrelated to project expenditures as the sub-components were focused exclusively on E-procurement. Project management (both in government and the Bank) exclusively targeted E-procurement activities so the improvements in these procurement metrics are serendipitous to the project (although to be fair the steady improvements in the indicators may have precluded any pro-activity by government and Bank teams). 2A: Create material base for E-procurement: Satisfactory Progress under this subcomponent is judged satisfactory in the sense that hardware and a functioning electronic network to conduct a full E-procurement system have been
2 The last ISR of February 28, 2011 indicates that there were no issues related to the counterpart funding even during the economic recession (section titled Ratings), but because of the economic crisis the Ministry of Finance was unable to provide the previously agreed co-financing (section titled E-procurement). While these two statements may appear to be contradictory, they are not for the following reason. The ISR reference is related to some extra co-financing that the government was willing to provide on top of the already agreed-upon funding for the project. Extra funding was thought necessary at one stage during implementation when the project allocation seemed insufficient for planned work. This situation stemmed from the preparation, by a local firm, of TORs for a full e-procurement system, rather than the narrower e-tendering component anticipated in the Government’s strategy. Not surprisingly the resulting proposals exceeded anticipated costs. With the assistance of the Bank team and experienced international consultants the TORs were reformulated and the subsequent rebidding led to lower prices, eliminating the need for the additional co-financing from the Government. Thus, in terms of formally agreed upon counterpart funding, there were no issues and the above-statement refers to this fact.
35
purchased and/or established. Central government, all regional governments, and most local governments have been connected to the government’s ICT system (which also carries HRM and EDMS data) which will serve as the material base for E-procurement. ICT hardware has been secured for all procurement offices in those agencies and the software for E-tendering designed, tested, and installed. Both government and the Bank team expended considerable time, energy, and resources to achieve this progress, which had to overcome issues of inadequate connectivity, duplicative initiatives, and lack of full knowledge of technical issues which were not fully identified during project preparation. Despite these challenges (which required the use of an external specialist hired by the Bank supervision team to conduct a mid-term assessment and propose and action plan) the material basis largely has been established for the entire E-procurement system. 2B: E-procurement software: Marginally Satisfactory Progress under this subcomponent is marginally satisfactory, not because of lack of effort and commitment but largely an over ambitious target. The core of an E-procurement system is in place and E-tendering is coming online, although the goal of a fully functional E-procurement system was not met by project closure. Such a system includes functions beyond E-tendering, including regulated communications between the buyer and suppliers, notification of bid decisions, publishing of bid award, contracting, oversight of contract execution, etc. However, with E-tendering the country has a process of transparent public auctioning for government procurement, representing the core of E-procurement benefits. According to government officials the E-tendering system was loaded onto servers by end May 2011, will be publicly launched in October 2011, and its use will be mandatory for all central government purchasing units, ten regional offices, and potentially major local governments as of January, 2012. It is a judgment call whether the project can claim a success in E-tendering since it was not functioning at project closure—however the fact that in the intervening months important steps have been taken to activate the system gives assurance (against the background of a well-performing and serious public administration) that E-tendering should fairly be judged a success. (The Bank could well have set a later project closing date to better capture the E-tendering advances which were indeed anticipated when the final date was chosen, but for other reasons chose not to do so). The fact that Armenia is carefully rolling out this system in line with normal international practice, and initiatives are continuing under the follow up PSMP II operation to complete the remaining, if less critical, elements of the E-procurement system, give further weight to a basically positive rating for this sub-component.
36
Strengthening Chamber of Control: Moderately Satisfactory
Intermediate Results Indicators, Targets, and Accomplishments of Component 3
Indicator-- Audit standards on 10 subject areas, one of which should be ethical standards, are developed and approved by the Chairman and applied. Target--At least 10 principal audit standards are written and approved by the Chairman.
Achieved Ten audit standards have been developed for specific areas, approved by the Chairman, and provide guidance to the staff when auditing specific areas. They will be updated to reflect international standards adopted during the recent meeting of INCOSAI in South Africa. The CoC website is operational and contains audit reports.
Indicator-- Audit manual is developed and approved by the Chairman Target--COC staff uses new methodology and standards. Audit manual chapters covering standards developed to date are written and approved.
Achieved Audit manual was approved and COC staff applies new methodology and standards. The manual will also be updated to reflect revised INCOSAI standards.
Indicator-- Action is taken on the Recommendations of the Consultancy funded under the Dutch Grant managed by the Association. Target--Government approves the strategy of external audit reforms based on the recommendations that come out of the Consultant’s report produced under the Dutch Grant Managed by the Association.
Achieved A Chamber of Control law, in line with recommendations from the consultant’s report recommending adoption of international standards for external and internal auditing, was passed on December 25, 2006 and became effective on June 17, 2007.
The stated objectives of this financially small component were largely achieved. The project provided support to the CoC for the development and internal review of modern performance audit standards covering 10 areas, as well as a code of ethics for CoC staff. All have been approved by the Board of the CoC including its Chairman and are in effect. An audit manual for financial compliance was similarly developed, approved and applied. The audit standards and methodologies being applied by the CoC are in line with international practices. The CoC is a member of international audit bodies, including INTOSAI and EUROSAI, and Armenia participates in IIA. The anticipated equipment and furniture purchases were made and the web site of CoC was upgraded and is now available to the public at www.coc.am.
37
3A: Audit standards and Audit Manual. Moderately Satisfactory. Ten performance audit standards were drafted and approved by the Chairman of the CoC covering Agriculture, Community, Education, Environment, Health Care, Judicial, Public Debt, Road Construction, Social Security, and Tax and Customs. A new Audit Manual for financial compliance was also drafted and approved by the Chairman. An internal code of ethics for CoC staff was issued. CoC staff indicate that an evaluation of budgetary programs utilized by the Ministry of Finance was undertaken and that it has helped inform CoC operations. 3B: ICT and Office Equipment. Moderately Satisfactory CoC and Bank records confirm that workstations, printers, peripherals, as well as local-area and wide-area network technologies were funded for the CoC, along with some office furnishing. 3C: Enhanced CoC web site. Satisfactory The web site was significantly enhanced in terms of information provided, reader friendliness, and ease of navigation. Additionally it provides information on the CoCs operations, including its annual reports, and contact points for citizens to report potential issues warranting CoC investigation. The web site—www.coc.am—is in Armenian and English.
38
Improving Policy Formulation and Implementation: Satisfactory
Intermediate Results Indicators, Targets, and Accomplishments of Component 4
Indicator--Electronic office and document tracking system established for pilot ministries. Target--System running in three pilot institutions.
Achieved EDMS piloted, rolled-out, linked to entire CS system including 15 ministries and 10 regional centers, enabling automated work planning streamlined with national strategies and staff performance evaluation and on-line tracing correspondence between the citizens and government agencies among other attributes. The introduced work planning and performance management system put in place the base for improved policy formulation and implementation. Policy implementation gained additional transparency through introduced online system, named “Interactive Budget”, allowing real time tracking of budget expenditures.
Indicator-- Government Press Center and public information outlet developed and utilized Target—No separate target
Achieved The Government Press Center is fully functioning enabling more efficient communication of policy with the public. Web site http://www.gov.am/en/press-conferences
Indicator-- Information availability on product standards easily accessible and testing instruments for products available. Target--Web based system for standards and technical regulations established and running.
Achieved {A relevant website: http://www.sarm.am/en/about_sarm/history} Consultancy delivered on certification and conformity assessment of various products. Laboratory equipment is procured.
Indicator-- Training courses and study tours on internal audit procedures and techniques are organized. Target--Training on internal audit procedures and practices is delivered.
Achieved A study tour to UK was conducted and pilot internal audit units have received training. In addition at least three training workshops were conducted in 2009 for chief auditors of central agencies and municipal administrations.
Indicator— Two pilot internal audit units are established in a line ministry and community administration.
Achieved Internal audit units are currently operational in three agencies.
39
Performance in this component is judged satisfactory. The rating reflects a weighting of achievements according to resources spent (and allocated in the PAD) and in some cases inevitably subjective assessments due to the broad character of the component and some less than clear objectives, indicators, and activities proposed in the PAD. The formal title of the sub-component, for example, is “Improving Policy Formulation and Implementation”, which might suggest substantive and measureable improvements in specific government policies. In fact the component largely aimed at improving ICT, training, and informational inputs which could lay the basis for improved policy and implementation outputs and outcomes. The successful creation of an Electronic Data Management System (sub-component 4A), by digitizing government operations and moving toward a paperless administration, has had (and is still having) a significant impact on government operations by improving its speed, accuracy of information, and internal consistency, permitting widespread business process reengineering and functional reforms in agencies, freeing government to focus more on better informed policy deliberations and less on routine transaction processing and monitoring, and fundamentally transforming citizen/government interaction (electronic application for government services, tracking of government responses to those requests, public oversight of budget expenditures, etc.). The EDMS has become the backbone of Armenia’s e-government effort, underlying the reforms in human resource management, E-procurement, and local government capacity building supporting other components of the project, as well as other public administration modernization efforts fully funded by the government and/or other donors. EDMS is considered a flagship of the project and pillar for current and future public administration reforms. Complementary reforms within the component also aimed to strengthen the basis for policy formulation and implementation through improvements in public communications (sub-component 4D). With project support the government has created a well functioning press operation at the center of government charged with overseeing a modern and more sophisticated effort at communications with citizens. An interactive web site (www.e-gov.am/interactive-budget) provides up to date information to the public on government spending in the various agencies. The press center (with its expanded communications capacity, regular briefings, information outlets, etc.) is in the government’s view increasing the flow and quality of information from government, enhancing the transparency of its operations, improving the quality of public debate on national issues, and promoting increased citizen awareness of, and feedback on, government operations. This is turn is judged (at least by government officials and local Bank staff) as continuing a trend toward more transparent government and rebalancing a legacy of one way communication flow and top down authoritarian public administration inherited from the Soviets. Two related sub-components (sub-components 4C and 4E) sought to improve policy making and communications with external partners, as well as the technical capacity in national standardization and certification to better meet the country’s obligations to the
40
World Trade Organization (WTO). Rapidly addressing so called “technical barriers to trade” relating to health and safety procedures and testing, etc. were considered a top priority for the country and its strategy to better integrate into the global economy. Admittedly somewhat far afield from a standard Bank public sector modernization effort in building core financial and HRM systems and center of government capacities, the two sub-components aimed at a substantive improvement in scientific capability to better ensure food safety, adequate product standards especially for exports, the validity of government certifications of technical standards, mobile laboratories to permit field testing, as well as communicating such activities. According to government officials, the investments in equipment, technical assistance to improve internal methods and capacity in the laboratories, observational visits to advanced labs in other countries, etc. have led to improved quality standards within the Ministry of Economic Development testing program, which inevitably has led to better policy formulation. The improved technical capability and standards of the laboratories has been confirmed by Bank funded consultant specialists. Regarding the MEDT’s ability to communicate and coordinate with external interlocutors a comprehensive web site has been developed and is operational at http://www.sarm.am/en/about_sarm/history. Performance in the sub-component directly related to the Ministry of Finance and Economy (4B) is more problematic. There were significant achievements in the sub-sub component related to the Ministry’s Training Center (building technical and material capacity, development of training curriculum and programs) as well as in training provided in specific areas such as internal audit and establishment of pilot internal audit units in line agencies. Improved coordination between the autonomous internal revenue and Customs organizations nominally under the Ministry is proving to require a longer term effort which is being supported by Bank expertise outside the project. 4A: Introduce Document Management System: Highly Satisfactory The stated objectives of this subcomponent were fully achieved. A sophisticated Electronic Document Management System (EDMS) is now functioning throughout the government, including in at 15 ministries and 10 regional administrations, and is being installed in regulatory commissions. EDMS has significantly cut the time for internal government communication (particularly with the regions) and is contributing to improved efficiency and transparency. The impact of EDMS in terms of speeding processes highlighted bottlenecks in other business procedures, prompting a series of agency functional reviews to streamline processes further. The government has produced a video demonstration of EDMS successes for use with citizens, donors, and others. It notes the impact on government efficiency, citizen tracking of requests, reduction of paperwork and corruption opportunities, increased use of electronic signatures, and reduction of transmission from “hours and days to seconds” especially for regional governments.
41
4B: Strengthen MOF Operations in Selected Areas: Marginally Satisfactory Performance in this sub-component has been mixed leading to the marginally satisfactory rating as essentially two of three sub-sub-components were implemented. The Ministry of Finance’s internal audit capacity has been strengthened through a series of seminars and observational travel to observe internal audit operations in the UK. Pilot internal audit units established in three agencies of different characteristics (Ministry of Education and Science, Avan community of the city of Yeravan, and Melikyan Music College-rather than the anticipated two as called for in the PAD). In the event, the effort at strengthening the Ministry’s internal audit mechanisms appears to have succeeded, as least as far as the anticipated training, foreign observational visits, and creation of internal audit units in three agencies is concerned. Bank staff confirm that the three supervision services within the Ministry have been consolidated into one, as indicated in the PAD. The entire process will take time to complete as internal audit, with its emphasis on analyzing and refining control systems, risks assessments, and support to agency heads to improve performance is a qualitatively different approach to the police mind set and internal investigation activities associated with the legacy Soviet internal control system. Armenia has been an active participant in regional networks (including the Public Expenditure Management Peer Assisted Learning—PEMPAL—effort supported by the Bank) seeking to foster a modern internal audit function. The sub-sub-component aiming to strengthen the Ministry’s Budget and Financial Management Training Center to provide training to financial units across government agencies has been a success. Investments in ICT equipment, furniture, pedagogic material, and technical assistance on training methodology and exposure to training programs in other countries’ Finance Ministries has been considered a major success by MOF officials. Project officials have indicated that the government has merged the State Tax Service and Customs Committee into the State Revenue Committee leading to the sharing of many systems and integration of departments, improving collaboration between the two units. However the extent of project support for training to strengthen revenue policy development and coordination between internal revenue and Customs is unclear. 4C: Improved Policy Making in Certification and Standardization by MEDT: Satisfactory The efforts under this sub-component should be considered in parallel with the related investments under sub-component 4E. An improved quality of scientific and technical data, better analysis capacity, increased confidence by government officials in the quality of these assessments led not only to more informed decision making but also improved compliance given greater domestic and foreign stakeholder recognition of the scientific base of the various policy announcements or adjustments. Officials note, for example, that the Ministry’s efforts to improve food testing, alcohol standards, and assessments of the character of tobacco products sold in the country have benefited from the project funded investments, leading to better policies in terms of more timely reaction to potential threats, better ability to policy established standards, and met export quality
42
expectation. Regarding the MEDT’s ability to communicate and coordinate with external interlocutors a web site has been developed and is functioning at http://www.sarm.am/en/about_sarm/history. 4D: Improved Communications Outreach: Satisfactory Performance under this subcomponent is satisfactory. With regard to this Press Center, the project objectives have been fully met. The Government Press Center, which existed before the project, has been significantly upgraded in terms of equipment (computers, communication links internally within the government and to media outlets, physical capacity to conduct briefings and information sessions, ability to broadcast cabinet meetings as warranted, etc.). The government’s website has been upgraded. The Center is the central locale for press briefings, the main source of written and electronic government information, and the heart of the public communications network also supported by the project in line agencies. Government officials have reported that the existence of the Press Center and increased citizen/press expectations regarding transparency in public activities has impacted policy decision making positively in terms of underscoring the need for clearer explanations of initiatives, marshalling of evidence, anticipation of press questions and alternative viewpoints, etc. The Center facilitates the role of the media as well as citizens in providing feedback (and oversight) to government. 4E: Improved Certification and Standardization Functions in MEDT: Satisfactory The project’s funding of upgraded laboratories, equipment, operating procedures, technical assistance in a wide variety of substantive areas, benchmarking against similar offices in more developed countries provided under 4E has led to improved basis for certification and standardization by MEDT and its policy making in the view of Ministry officials. A Bank funded specialist visited Armenia and reported in 2007 that, among other things, technical proposals and equipment offered from successful bidders were sound and acceptable, new and remodeled laboratories visited were acceptable for their intended uses (clean, well lighted, and maintained), staff appears to have basic technical competence, answers from senior officials were satisfactory in general, etc. The report noted areas for further work by counterparts. No additional independent reporting was available beyond this date.
43
Developing Capacity in Local Governments: Moderately Satisfactory
Intermediate Results Indicators, Targets, and Accomplishments of Component 5
Indicator— Legislation on the establishment and operation of ICUs approved, so that administrative fragmentation can be overcome.
Not Achieved Although pilot ICUs were created and certain legislative scope was created for voluntary establishment of ICUs, the Government did not obtain legislative approval of for involuntary establishment of ICUs.
Indicator-- Necessary legal amendments/complementary regulation, and an action plan for the transfer of property and land tax database adopted. Targets--Transfer of the land tax database to local authorities is completed.
Achieved Legal amendments and complementary regulations to enable local tax administration put in place. Nearly 100% of property and land tax databases have been transferred to communities.
Indicator-- Municipal service law approved and implemented in congruency with the national civil service law. Target--The implementation of the municipal service law has been completed in the whole country.
Achieved Municipal Services Law was approved in Dec 14, 2004 and technical assistance delivered to support its implementation in municipalities.
Indicator— Municipal governance information system developed and operational in 60 largest municipalities.
Achieved MGIS introduced in over 200 municipalities (including the target 60 largest municipalities).
Indicator-- Improved access to information on public services provided by local communities (15 percent increase over the baseline).
Achieved 16 percent increase achieved by 2010
A major initiative in Armenia has been to devolve responsibilities and taxing authority to local governments which are closer to citizens and better able in the long run to manage local service delivery. At the time of appraisal, and in response to the client’s request, this component was included to help support the decentralization program. Progress under several areas has been notable. For example, data on property and land taxes have been transferred to nearly all local communities, pilot programs have been initiated to improve human resource management at the local administrative level (to in part better manage the increased responsibilities for tax collection), and an integrated, computer based management information system has been designed and installed in over 200 municipalities, including the largest 60. In short, the long term process of institutional capacity building in municipalities is well launched, with support of the project. One
44
sub-component has not advanced as parliament has not approved legislation mandating involuntary Inter-Community Units (designed to compel consolidated services in neighboring small municipalities) so work on the necessary regulatory framework could not commence. Nevertheless there appears to be a voluntary process of consolidation underway, which is benefiting from the other investments within this component. 5A: Legal Framework for Inter-Community Unions: Unsatisfactory The objectives of this subcomponent were not achieved as the National Assembly has not approved draft legislation permitting the involuntary grouping of municipalities into Inter-Community Units (ICUs). Therefore there was no scope for assisting in the drafting of subordinate legal and regulatory frameworks. No funds from the project have been expended on this subcomponent, and its inclusion in the project prior to approval of legislation obviously was premature. The subcomponent and related intermediate results indicator were not formally canceled due to the government’s hope that it would secure legislative approval. In the event the government is awaiting completion of an evaluation of international experiences by the Ministry of Territorial Administration (MoTA) to adjust the legislation as appropriate for another attempt at legislative action. It is important to note that the non-performance under this sub-component is due to circumstances largely outside the control of the government and was noted in ISRs. 5B: Building local capacity for property and land tax collection: Satisfactory By the end of the project 100% of property and land tax data (cadastre) had been transferred to local authorities and local governments had been supplied with a budget module to their management information system (see below) to incorporate this data. 5C: Strengthening HR management at the local level: Satisfactory A major step toward improving HR management at the local level was the deployment of an HRM sub-system within the new Municipal Management Information System to the bulk of Armenia’s local governments. The sub-system provides processing of various kinds of data on municipalities’ personnel structure, individual employees information, their trainings, certifications, awards, disciplinary action, annual and sick leaves, business trips, hours worked, etc. The project funded a consultancy to assist local authorities in the implementation of the municipal services law and related training of officials. 5D: Management information system at the local level: Satisfactory The objectives of this component were fully met with the deployment of a municipal management information system (MMIS) to 225 municipalities, including the largest 60. It is managed at the national level, with technical support, help desk, and training to local staff. The web based system complements an Information System for Civil Registry Offices. Training has been provided to over 1300 municipal staff on the operation of the systems. The MMIS itself covers the majority of the functions of municipal governments, improving efficiency, as well as transparency and effectiveness. The functions integrated
45
within MMIS include managements of municipal budgets, assets, human resources, licensing, local taxes, document flow, registries of businesses and citizens, catalogue of service procedures, calendars of senior government officials, events calendar, demographic statistics, citizen opinion polling, citizen application and complaint hotlines, general municipal contact and rules information, and internet forums.
Annex 3. Economic and Financial Analysis (including assumptions in the analysis) Not Applicable. As a technical assistance operation no quantitative economic and financial analyses were prepared at appraisal. Rather it was asserted that improvements in ICT systems, E-procurement, and electronic data management would lead to savings offsetting at least in part the costs of the project. The anticipated improvement in civil service performance could be offset by higher salaries. In the event it would appear reasonable to assert that these assumptions were broadly correct and that the efficiency and effectiveness of public management has improved with positive implications as a whole for government private sector performance.
46
Annex 4. Bank Lending and Implementation Support/Supervision Processes
(a) Task Team Members
Names Title Unit Responsibility/
Specialty Lending
Supervision/ICR Alexander Astvatsatryan Procurement Officer ECSO2 Kathy Lalazarian Senior Public Sector Specialist LCSPS Davit Melikyan Public Sector Mgmt. Spec. ECSP4 Craig R. Neal Consultant ECSPE Joao C. Oliveira Consultant AFTHE Armen Shahverdyan Consultant ECCAR Gurcharan Singh Senior Procurement Specialist TWICT Jiro Tominaga Special Assistant IEGDG Arman Vatyan Sr Financial Management Specia ECSO3
(b) Staff Time and Cost
Stage of Project Cycle Staff Time and Cost (Bank Budget Only)
No. of staff weeks USD Thousands (including travel and consultant costs)
Lending FY00 1 2.37 FY01 10 23.96 FY02 20 90.10 FY03 29 126.28 FY04 69 276.99 FY05 0.00 FY06 0.00 FY07 0.00 FY08 0.00
Total: 129 519.70 Supervision/ICR
FY00 0.00 FY01 0.08 FY02 0.00 FY03 0.00 FY04 0.00 FY05 44 151.15 FY06 50 176.51 FY07 45 106.51 FY08 45 183.08 FY09 9 0.00
Total: 193 617.33
47
Annex 5. Beneficiary Survey Results Outsourced beneficiary surveys covered the period 2005- 2009/2010 and a final report delivered in April 2010. The report focused on intermediate results indicators then being tracked in ISRs. The data was gathered, analyzed, and presented on 27 topics using surveys, focus group discussions, review of government documents, interviews, etc. More detailed findings of beneficiary surveys are reflected in the Annexes 2 and 7 of this ICR.
Annex 6. Stakeholder Workshop Report and Results Not Applicable.
Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR Component 1. Strengthening Civil Service Management The activities under Component 1 supported the development of a merit-based civil service system. The main emphasis was on the development of systems and capacities to ensure a consistent application of merit-based recruitment and promotion procedures across all institutions under the civil service law. Under Component 1A support was to be provided for drafting of a Civil Service Development Strategy and the completion of remaining elements of the Regulatory Framework for civil service management. The Civil Service Development Strategy drafted under this subcomponent was endorsed and approved by the Public Sector Reform Commission. The Strategy sets out a number of actions that will make the system more transparent and enable using an effective performance management, justification and training system. Subcomponent 1B aimed to strengthen the Civil Service Council’s ability to improve its testing and attestation procedures for furthering the introduction of a merit-based recruitment and promotion system, equipping the Council’s offices as well as establishing a Public Information Center. The following activities were implemented under this Subcomponent:
1. Competition and attestation tests and exam paper questions were developed for the Civil Service Council and all relevant civil service bodies enabling a more effective specialist selection and evaluation. This is found by reviewing the ratings of effectiveness of competition and attestation test exams in civil service presented below:
48
Figure 1
As illustrated in Figure 1, the number of respondents having expressed a positive opinion, i.e. considering the test exam of the competition as an effective tool for selecting qualified candidates for a particular position, increased by 10 percent during the last two years.
Figure 2
49.7%
34.5%
15.8%
51.1%
33.0%
15.9%
64.7%
27.9%
7.4%
0%
20%
40%
60%
80%
100%
2007 2008 2009-2010
Is the test exam of attestation a useful tool for establishing the aptness for acivil service job?
Don’t know
Negative
Positive
65.1%
27.4%
7.5%
62.9%
30.0%
7.1%
72.8%
19.3%
7.9%
0%
20%
40%
60%
80%
100%
2007 2008 2009-2010
Is the civil service test exam an effective tool for merit-based appointment ina civil service position?
Don’t know
Negative
Positive
49
The ratings for the effectiveness of attestation tests in the Figure 2 show a 13.6 % increase in the number of people having expressed a positive opinion during the last two years.
2. Computer hardware, printing house equipment and accessories, software, air conditioners, water pipes, electrical and technical appliances and furniture were purchased for and installed at the Civil Service Council to ensure a proper work environment.
Component 1C was to support the introduction of cross-ministerial human resource management and assessment/attestation systems. The following activities were implemented under this subcomponent:
1. A human resources management information system (HRMIS) was developed enabling the CSC to monitor the regular use of merit-based procedures and contributed to rewarding professional performance through promotion. HRMIS has been installed in all relevant civil service bodies, and training has been provided to system users. Information on civil service competitions, attestations, trainings and their results is entered into the system and administrative relations with the CSC are regulated through it.
2. IT equipment (servers, computers, scanners, uninterrupted power supply devices, printers, network switches and cables) were purchased for RA government bodies for operating the HRMIS.
Component 1D was to assist the Public Administration Academy as the key institution providing civil service training in improving the quality of training programs, as the main provider of civil service; training at the central, as well as regional levels was a pressing need of the civil service. The ability of the Academy to develop and implement training programs had to be strengthened, and the Academy’s training facilities upgraded. The following activities were implemented under this subcomponent:
1. Training programs and 24 training modules were developed for the RA Public
Administration Academy and training of trainers was conducted aimed at developing wider professional development opportunities for civil servants.
2. Computer and miscellaneous electrical equipment, furniture, building materials and around 700 books were purchased for the Academy.
3. A manual guideline on the Public Service was developed for the Public Administration Academy enabling both the students of the Academy and all employees and the population to become familiar with the key provisions of the law in a more accessible manner and check their knowledge in the area of public administration.
50
4. A questionnaire was prepared for the Public Administration Academy enabling to check student knowledge during admission to the Academy, moving to the next year and final exams.
Component 2. Introducing E-procurement
Component 2 was to help to establish IT infrastructure for e-procurement and establish the basis for the development of a full-fledged e-procurement system. Under this component it was planned to reduce the use of non-competitive procedures, ensure satisfaction of the business community with the public procurement process and increased fairness of the process, and make updated information available to all participants of public procurement on offered and awarded contracts. Off-the-shelf software was developed/ introduced for an electronic public procurement system (e-tendering), the use of which would enable improving the management of a significant share of annual public expenditure and make the procurement process more transparent. As part of the assignment training was provided to final users and technical, operational and user’s manuals were prepared.
Component 3. Strengthening the Chamber of Control Component 3 provided focused support to the Chamber of Control (CoC) as part of which external audit manuals and standards were developed in line with INTOSAI standards and EUROSAI guidelines to enhance its ability to provide professional audit reports to the National Assembly and related training was provided. Under this Component computer hardware and office equipment were purchased for the Chamber of Control, the website was upgraded; the annual report of the National Assembly was printed, published (in Armenian and English), disseminated, etc. Component 4. Improving Policy Formulation and Implementation through Capacity Building Support provided under Component 4 was to assist the Government in addressing weaknesses in policy formulation and implementation process in the central state administration, in particular through the development of electronic document management systems and systems for tracking policy implementation. In addition, assistance was to be provided for other model activities in certain state administration institutions aimed at increasing accountability and transparency and meeting priority capacity building needs in selected institutions. Subcomponent 4A was to support the improvement of policy process management in core state administration, in particular the document flow.
51
The following activities were implemented under this subcomponent:
1. The policy implementation and monitoring process was diagnosed aiming to develop and strengthen policy formulation, implementation and monitoring capacities of the RA Government in three pilot bodies (the Office of the Government, the Ministries of Finance and Economy).
2. Based on initial analysis of policy implementation the subcomponent was to support development of a wider governance system and increased implementation efficiency. As a result, the Mulberry electronic document management system was developed and piloted in the Government Office, ministries of Finance and Economy. After being tested and getting successful results from them pilot, the system was rolled out to other government bodies reaching the current number of 30. This system enables tracking decision making processes, implemented policies in these bodies, timely and efficient implementation of Prime Minister’s and Cabinet decrees by these bodies and making administrative documents and information available to the public.
3. To ensure the efficient operation of the Mulberry electronic document management system a large number of IT equipment were purchased and installed and training was provided to the users of the system.
4. A security audit of the Government electronic management system was conducted to verify the conformity of the Mulberry electronic document management system to international security standards.
5. An electronic system for accepting license applications to the RA government was introduced to enable natural and legal persons to send electronic applications for being issued any license existing in RA.
The activities under Subcomponent 4B were to enhance the capacity of the Ministry of Finance in the areas of revenue policy, internal audit and the Budget and Financial Management Training Center through training and upgrading equipment. The following activities were implemented as part of this sub-component:
1. Equipment for heating, anti-fire, ventilation and cooling systems were supplied to
the Training Center state non-commercial organization, IT equipment and office furniture were purchased and simultaneous translation system was installed and tested. Strengthening of the Training Center’s capacities was aimed at providing efficient training to all budget department staff in the public sector.
2. Training courses on program budgeting, budget formulation, accounting standards and internal audit were organized.
52
Subcomponent 4C was to support the Ministry of Economy in enhancing its policy making capacity in the areas of standardization, certification and metrology and strengthen the capacity to communicate with external interlocutors, domestic and international, public and non-governmental. The following activities were implemented under this Component:
1. In the area of RA standardization and conformity assessment, technical legislation
was diagnosed. Accordingly, a strategy was developed for harmonization of Armenia’s standardization system with international and European systems, where national standardization priorities were reviewed and substantiated.
2. Under the support planned for the Training Center of the National Institute of Standards, training was provided on standardization and certification of conformity of quality management systems for fuels and lubricants, perfumes and cosmetic products, household electrical appliances, means of telecommunications, building items and structures and food stuffs.
3. Guidelines-manuals on standards and certification were developed and published.
4. Laboratory equipment was purchased for the National Institutes of Standards and Metrology reducing work-time and human interference. These devices are used to test physical and chemical indicators of foodstuffs for detecting the content and quantity of toxic metals.
The purchased mobile laboratory enables checking the quality of petroleum products and detecting impermissible deviations.
The surveys conducted showed that 63.1% of businesses were well aware of the required laboratory tests for certification of goods produced/imported/exported by them and 62.3% were satisfied with the time the certification process took for goods. Progress was also made in the area of standardization; in particular 56.3% of businesses having applied for registration of a standard in 2009-2010 and 57.4 % of respondents indicated that were satisfied with the length of time of the standardization process.
1. The Website of the Ministry of Economy was designed and presented providing access to the Ministry related information, increasing the level of public awareness and ensuring the linkage between the society and other bodies.
2. Consultancy was provided to the Ministry of Economy for introduction and implementation of the performance appraisal system where the international experience was reviewed and recommendations were made on the introduction of the system.
53
Subcomponent 4D focused on strengthening the capacity of the Office of the Government in external communications and outreach. The following activities were implemented under this subcomponent:
1. Armenia’s Government Press Center was renovated and equipped with modern equipment resulting in a more effective cooperation between the Government and reporters. Office furniture, satellite connection, acoustic and simultaneous translation equipment, speakers, microphones, cameras, digital cameras and IT and other video editing equipment were purchased.
2. Electronic signature devices and network equipment were purchased for the Government information maintenance center.
3. The Government website was redesigned posting information on state budget aimed at making the budget expenditure as transparent and as monitorable as possible for the public.
Component 5. Developing Capacity in Local Governments The project activities under Component 5 aimed at improving local self-government capacity to deliver public services. Component 5A was to support the development of the legal and policy framework for intercommunity unions (ICUs) aimed at eliminating the negative impact of administrative fragmentation. The Government sees the voluntary formation of ICUs as the best way to address administrative fragmentation in view of the Communities’ weak capacity and resource base. Given that the Parliament delayed the passage of legislation on formation of Intercommunity Unions, it was agreed to substitute the consulting services category under this subcomponent for goods and buy equipment for the already existing ICUs.
Under this subcomponent IT equipment (servers, computers, printers and network equipment) were supplied for ICUs. Subcomponent 5B was to finance local and international assistance and trainers to advise the Ministry of Territorial Administration on strengthening local tax administration capacity. Local capacity building for efficient revenue collection is a priority in improving local governments. The law also empowered the local governments to collect additional revenue including the right to collect the property tax and the land tax. The property and land tax collection is quite problematic, in particular in rural areas. Under this subcomponent the necessary legal changes were made, additional regulations passed and an action plan was introduced for transferring the property tax and land tax database; the database was fully transferred to communities.
54
Subcomponent 5C was to help design and implement a human resources management system for the municipal service. The following activities were implemented under this subcomponent.
1. Armenia’s legislation on Municipal Service was improved through amendments and supplements to the RA Law on Municipal Service and secondary legislation implied by the latter. As a result, the municipal service system was brought into conformity with the situation driven by legal processes underway in the Republic of Armenia.
2. The secondary legislation on civil status registry of the Republic of Armenia was drafted; namely birth, marriage, adoption, determining paternity, divorce, death as well as new draft legal acts on changing, amending and correcting civil status records.
Subcomponent 5D focused on strengthening municipal data management to improve local governance. It was to be carried out through the development and deployment of an integrated computer-based management information system and by the strengthening of the national center for technical support and systems development in Charentsavan. The following activities were implemented under this subcomponent:
1. Government information system was developed and informed the activities carried out in local governments. They have become transparent and efficient. Introduction of the system enabled recording and managing registries of land, buildings, means of transport, natural and legal entities.
The surveys conducted revealed that community offices had sufficient ability to manage the population registry (80.6%) and carry out administrative work in the community (100%).
2. IT equipment (servers, computers, printers, scanners, network equipment), licenses, antivirus software were purchased for local government civil status registry offices intended for ensuring the efficient operation of the local government management information system.
3. Computer equipment (servers, computers, printers and network equipment), software and office furniture were purchased for and installed in the National Center in Charentsavan.
Due to the activities implemented under Component 5 information on the services provided by the community became more accessible and the population’s satisfaction with delivered services increased as illustrated in the tables below:
55
Table 1: Accessibility of Information on Services Delivered by Communities
Services provided by Civil Status Registry’s regional
center
Issuing references (for Paros, on the place of residence and family
composition)
Provision of information on property tax and land tax
Provision of information on vacancies in the
municipality/village mayor’s office
2005 2007 2009-2010
2005 2007 2009-2010
2005 2007 2009-2010
2005 2007 2009-2010
Yes 39.2% 41.4% 44.8% 63.2% 66.0% 70.6% 78.8% 81.2% 87.8% 27.4% 29.0% 31.4%
Rather yes 1.8% 2.8% 3.2% 1.8% 3.0% 4.2% 2.0% 4.0% 6.4% 2.0% 2.0% 3.0%
Rather no 2.6% 2.6% 2.6% 2.2% 2.2% 2.2% 0.8% 0.8% 0.8% 2.0% 2.0% 2.0%
No 11.8% 11.0% 10.0% 7.8% 7.6% 6.8% 5.6% 4.6% 2.2% 33.0% 33.0% 32.2%
Don’t know 44.6% 42.2% 40.4% 25.0% 21.2% 16.2% 12.8% 9.4% 3.8% 35.6% 34.0% 31.4% Total 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100%
Table 2: Population’s Satisfaction with Services Provided by the Community
Services provided by
Civil Status Registry’s regional center
Issuing references (for Paros, on the place of residence and family
composition)
Provision of information on property tax and land tax
Provision of information on vacancies in the
municipality/village mayor’s office
2005 2007 2009-2010
2005 2007 2009-2010
2005 2007 2009-2010
2005 2007 2009-2010
Satisfied 31.2% 32.2% 35.0% 53.8% 56.0% 60.0% 54.4% 56.8% 60.6% 19.8% 20.2% 23.0% More likely satisfied
5.2% 6.6% 7.0% 4.8% 5.2% 8.4% 12.2% 13.2% 17.4% 3.8% 4.8% 5.2%
More likely not satisfied
1.8% 1.6% 1.8% 3.4% 3.4% 3.6% 8.0% 8.0% 7.0% 2.2% 2.2% 1.6%
Not satisfied 1.8% 1.8% 1.8% 5.2% 5.6% 5.4% 5.8% 5.0% 4.8% 11.4% 10.6% 10.2%
Don’t know 60.0% 57.8% 54.4% 32.8% 29.8% 22.6% 19.6% 17.0% 10.2% 62.8% 62.2% 60.0%
Total 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100.0% 100.0%
Component 6. Project Management, Surveys and Communication Component 6 was to support project management, surveys and outreach activities since monitoring all achieved results and impact was an essential element to project management. Funds provided under subcomponent 6A were aimed at funding the Project Manager and support staff and covering the FFPMC expenses, which took care of the financial management and procurement functions for the project. Surveys of the population, the business community and relevant government bodies were carried out under Subcomponent 6B aimed at collecting data on the key project indicators and receiving feedback from beneficiaries on implemented reforms. Public awareness campaign of the project was conducted to raise the awareness on the project among the public and beneficiaries.
56
Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Not Applicable.
Annex 9. List of Supporting Documents World Bank Documents
Project Appraisal Report—Public Sector Modernization Project, April 8, 2004 Development Credit Agreement—Public Sector Modernization Project Implementation Supervision and Results Reports (ISR) 2005-2011 Aide Memoires Country Assistance Strategy Country Partnership Strategy, May 12, 2009, Report Number: 48222-AM Institutional and Governance Review Improving Policy Formulation and Implementation through Capacity Building Assessment of the Laboratory Equipment Program, Ed Nemeroff, March, 2007
Government Documents Public Sector Modernization Project, Mid Term Report, Public Sector Reform Commission, May 26, 2008 Final Report: Monitoring and Evaluation of Public Sector Modernization Project, April, 2010 Municipal Management Information System Report, Ministry of Territorial Administration, 2009 Laboratory Assessment Report, LSU AgCenter and Colorado State University for the US Department of Agriculture and Government of Armenia. December 2006 Mulberry (EDMS System) DVD Public Sector Modernization Project, Borrower ICR (Brief Report), June, 2011
57
Individuals Contacted World Bank
Tony Verheijen (TTL at Board Presentation) Kathy Lalazarian (TTL in Supervision) K. Migara O. de Silva (TTL in Supervision) Davit Melikyan (Team member, Yerevan) Yoko Kagawa (Team member, Washington) Knut Leipold (Team member, Washington) Arman Vatyan (Team Member, Yerevan) Alexander Astvatsatryan (Team Member, Yerevan Government of Armenia
Aharon Mkrtchyan (Project Manager) Edgar Avetyan (PIU Director) Vigen Ktikyan (Government Office) Edgar Gevorgyan (Government Office) Emil Tarasyan (Ministry of Economy) Grisha Khachatryan (Ministry of Territorial Administration) Artur Zakaryan (Ministry of Finance and Economy)
LL ee ss ss ee rr CC aa uu cc aa ss uu ss MM oo uu nn tt aa ii nn ss
ZZaa
nngg
ee zzuu
rrRR
aann
ggee
VV aa rr dd ee nn ii ss RR aa nn gg ee
L O R R IL O R R I
A R A G AA R A G AT S O T NT S O T N
A R A R AA R A R ATT
KO
TK
OTAA
YK
YK
’’
YEREVYEREVANANA R M AA R M AV I RV I R
S H I R A KS H I R A K
TTAAV U S HV U S H
G E G H A R KG E G H A R K ’’ U N I KU N I K ’’
S Y U N I KS Y U N I K ’’
VVAAY O T SY O T S ’’
D Z O RD Z O R
AlaverdiAlaverdi
DilijanDilijan
SevanSevan
AraratArarat
MegriMegri
GorisGoris
AngekhakotAngekhakot
VVaikaik
JerJermukmuk
VVardenisardenis
MarMartunituni
ArArtsvashentsvashenArArtiktik
TTashirashir
KarKarmirmir
StepanavanStepanavan
GyumriGyumri
IjevanIjevan
VVanadzoranadzor
ArArmavirmavir
HrazdanHrazdan
AshtarakAshtarakGavarGavar
ArArtashattashat
YYegegnadzoregegnadzor
KapanKapan
YEREVYEREVANAN
HHrraa
zzddaann
L O R R I
A R A G AT S O T N
A R A R AT
KO
TAY
K’
YEREVANA R M AV I R
S H I R A K
TAV U S H
G E G H A R K ’ U N I K ’
S Y U N I K ’
VAY O T S ’
D Z O R
Alaverdi
Dilijan
Sevan
Ararat
Megri
Goris
Angekhakot
Vaik
Jermuk
Vardenis
Martuni
ArtsvashenArtik
Tashir
Karmir
Stepanavan
Gyumri
Ijevan
Vanadzor
Armavir
Hrazdan
AshtarakGavar
Artashat
Yegegnadzor
Kapan
YEREVAN
AZERBAIJAN
GEORGIA GEORGIA
AZERBAIJANISLAMIC REPUBLIC
OF IRAN
TURKEY
Kura
Arpa
Arpa
Hra
zdan
Aras
Vor otan
Aras
Aras
Aras
LakeSevan
MingechevirReservoir
L e s s e r C a u c a s u s M o u n t a i n s
Za
ng
e zu
rR
an
ge
V a r d e n i s R a n g e
Aragats(4090 m)
ToNaxçivan
To Naxçivan
To Ordubad
To Füzili
ToQubadli
ToQubadli
ToGäncä
To Gäncä
To T’bilisiTo T’bilisi
To Borjomi
To Gäncä
ToKars
43°E
44°E
44°E
45°E
46°E 47°E
39°N
40°N 40°N
41°N 41°N
43°E
47°E
ARMENIA
This map was produced by the Map Design Unit of The World Bank. The boundaries, colors, denominations and any other informationshown on this map do not imply, on the part of The World BankGroup, any judgment on the legal status of any territory, or anyendorsement or acceptance of such boundaries.
0 10 20 30 40
0 10 20 30 Miles
50 Kilometers
IBRD 33364
SEPTEMBER 2004
ARMENIASELECTED CITIES AND TOWNS
PROVINCE (MARZ) CAPITALS
NATIONAL CAPITAL
RIVERS
MAIN ROADS
RAILROADS
PROVINCE (MARZ) BOUNDARIES
INTERNATIONAL BOUNDARIES
![QAG Version 2.2 May 16, 2014 - King Schools, Inc. · QAG Version 2.2 May 16, 2014 [2] Section 1: List of Configurations Currently Approved: 1. Configuration: TD2 BASIC-Standard -](https://img.dokumen.tips/doc/110x75/605aa7998ed29f5c5d69c04d/qag-version-22-may-16-2014-king-schools-inc-qag-version-22-may-16-2014-2.jpg)
![QAG Version 2.2 May 16, 2014 - kingschools.com · QAG Version 2.2 May 16, 2014 [3] Statements of Compatibility of Software and Hardware AC61-136, Appendix 2, 4. b.: The device and](https://img.dokumen.tips/doc/110x75/5ae70c4f7f8b9aee078da9b7/qag-version-22-may-16-2014-version-22-may-16-2014-3-statements-of-compatibility.jpg)
