Upload
dinhliem
View
214
Download
0
Embed Size (px)
Citation preview
Joint Evaluation of the International Trade Centre
Version Final pg 1/142
Joint Evaluation of the
International Trade Centre
File No. 104.A.1.e37
Draft Final Inception Report
August 20, 2004
DMI ASSOCIATES
in association with
ACE-Global, Ticon/DCA, Copenhagen DC
21, rue Longue
Lyon, France
Joint Evaluation of the International Trade Centre
Version Final pg 2/142
Table of contents
1 INTRODUCTION..............................................................................................................................6
1.1 PROJECT SYNOPSIS: JOINT EVALUATION OF THE INTERNATIONAL TRADE CENTRE..................6
1.2 OUTLINE.......................................................................................................................................6
2 BACKROUND AND RATIONALE ...............................................................................................7
2.1 OVERVIEW OF THE ITC ................................................................................................................7
2.1.1 Mission Statement...................................................................................................................7
2.1.2 Objectives of ITC.....................................................................................................................7
2.1.3 Origins and governance ..........................................................................................................8
2.1.4 The last Decade........................................................................................................................8
2.1.5 Strategic Issues......................................................................................................................13
2.2 OBJECTIVES OF THE EVALUATION .............................................................................................17
2.3 KEY EVALUATION QUESTIONS FROM THE TORS ......................................................................18
3 METHODOLOGY ...........................................................................................................................19
3.1 INTRODUCTION ..........................................................................................................................19
3.2 DESCRIPTION AND CHOICE OF METHODOLOGIES ....................................................................19
3.2.1 Uniqueness of ITC.................................................................................................................19
3.2.2 Stakeholders...........................................................................................................................20
3.2.3 Approach to the evaluation....................................................................................................22
3.3 OVERVIEW OF THE METHODOLOGY ..........................................................................................24
3.4 THE EVALUATION MATRIX.......................................................................................................26
3.4.1 TRTA Results-Based - reference framework.........................................................................26
3.4.2 ITC - Results-Based logic of intervention .............................................................................32
3.4.3 The key indicators..................................................................................................................33
3.4.4 The key evaluation questions.................................................................................................37
3.5 IMPLEMENTATION MODALITIES................................................................................................38
3.5.1 Assessment of ITC relative position in TRTA and of Developed Country Donor/TSI
Perspectives..........................................................................................................................................39
3.5.2 Assessment of ITC performance and comparative advantage at the Country or Regional
Level 40
3.5.3 Assessment of ITC Products..................................................................................................40
3.5.4 Assessment of Corporate Organisation, Financing and Management..................................42
3.6 DATA GATHERING TOOLS ..........................................................................................................43
3.6.1 Sampling................................................................................................................................46
3.6.2 Evaluation steps ....................................................................................................................47
3.7 APPROACH TO THE EVALUATION .............................................................................................48
3.8 COMMUNICATIONS STRATEGIES ...............................................................................................49
3.8.1 Communication with Stakeholders........................................................................................49
3.8.2 Public Communications and Website....................................................................................49
3.9 QUALITY ASSURANCE ...............................................................................................................56
3.10 RISKS, MITIGATING STRATEGIES AND ASSUMPTIONS...............................................................56
3.10.1 Risks:.................................................................................................................................56
3.10.2 Mitigation strategies.........................................................................................................57
Joint Evaluation of the International Trade Centre
Version Final pg 3/142
3.10.3 Assumptions .....................................................................................................................58
3.11 ISSUES/CONCLUSIONS................................................................................................................58
4 PROJECT PLANNING....................................................................................................................59
4.1 WORKPLAN ................................................................................................................................59
4.1.1 Methodology, Timeframe and outputs...................................................................................59
4.1.2 Tasks, Milestones and Reporting ..........................................................................................60
5 RECOMMENDATIONS.................................................................................................................62
5.1 TIME FRAME...............................................................................................................................62
5.2 BUDGET ......................................................................................................................................62
5.3 COMMUNICATIONS....................................................................................................................62
5.4 MOBILISATION OF PERSONNEL..................................................................................................62
6 ANNEXES..........................................................................................................................................64
6.1 ANNEX 1: EVALUATION QUESTIONS.........................................................................................64
6.2 ANNEX 2: INDICATORS MATRIX.........................................................................................82
6.2.1 Project Universe for Evaluation ..........................................................................................104
6.3 NOTE ON COUNTRY SELECTION..............................................................................................113
6.4 ANNEX 4: QA PLAN ................................................................................................................122
6.4.1 Introduction to the Quality Assurance Management System ............................................122
6.4.2 Administrative procedures ..................................................................................................122
6.4.3 QA Annexes ........................................................................................................................129
6.4.4 QA Annex 1 ........................................................................................................................131
6.4.5 QA Annex 2 QA of Reports ................................................................................................133
6.4.6 QA Annex 3 Template for Reports......................................................................................136
6.4.7 QA Annex 4 File Structure.................................................................................................137
6.4.8 Structure of the WEB Server...............................................................................................140
Joint Evaluation of the International Trade Centre
Version Final pg 4/142
Document control
Name Function Date Signature
Author MGS, AF, AL,
NJG, ME,
SVD
TL,
Evaluators,
Approved by
(if relevant)
ACRONYMS AND ABBREVIATIONS
ACP African, Caribbean & Pacific States
APEC Asia-Pacific Economic Co-operation
CCA Common Country Assessment
CIS Commonwealth of Independent States
CO Country Office
CSO Civil Society Organization
EOI Expression of Interest
EU European Union
FDI Foreign Direct Investment
FTA Free Trade Area or Free Trade Agreement
GATS General Agreement on Trade in Services
GATT General Agreement on Tariffs & Trade
GSP Generalised System of Preferences
IF Integrated Framework
IFIS International Financial Institutions
ITC International Trade Centre
JITAP Joint Integrated Technical Assistance Program
LDC Least Developed Countries according to UN designation
NAFTA North American Free Trade Area
NHDR National Human Development Report
NPO National Programme Officer
NGO Non-Governmental Organization
OECD Organization for Economic Co-operation and Development
PRSP Poverty Reduction Strategy Paper
RBM Results-Based Management
ROAR Results-Oriented Annual Report
SHD Sustainable Human Development
SME Small or Medium-sized Enterprise
Joint Evaluation of the International Trade Centre
Version Final pg 5/142
ACRONYMS AND ABBREVIATIONS
SPS Sanitary and Phyto-Sanitary
SRF Strategic Results Framework
SSE Small-Scale Enterprises
TBT Technical Barriers to Trade
TRA Trade Related Assistance
TRIPS Trade Related Intellectual Property Rights
TORS Terms of Reference
TRTA/CB Trade Related Technical Assistance & Capacity Building
UN United Nations
UN/CEFACT United Nations Centre for Trade Facilitation & Electronic Business
UNCTAD United Nations Conference on Trade & Development
UNDAF United Nations Development Assistance Framework
UNECE United Nations Economic Commission for Europe
UNDP United Nations Development Programme
UNIDO United Nations Industrial Development Organisation
UNISTE United Nations International Symposium on Trade Efficiency
WCO World Customs Organization
WTO World Trade Organization
GLOSSARY1
Indicator: Signal that reveals progress (or lack thereof) towards objectives; means of
measuring what actually happens against what has been planned in terms of
quantity, quality and timeliness. It is a quantitative or qualitative variable
that provides a simple and reliable basis for assessing achievement, change
or performance.
1 Note that we are using the OECD Glossary of Key Terms in Evaluation and Results Based Management,
2002.
Joint Evaluation of the International Trade Centre
Version Final pg 6/142
1 INTRODUCTION
1.1 Project Synopsis: Joint Evaluation of the
International Trade Centre
Project number: File No. 104.A.1.e37
Country : Global
This project is the Joint Evaluation of the International Trade Centre (ITC). It is
supported by a group of seven donors led by Danida. The purpose is to assess the
comparative advantages of the ITC, to evaluate the performance of the ITC’s
interventions, and to assess the capacity of the ITC and Donor-Funding Modalities. The
project is organized in four phases. The first phase, Inception and Desk Study, is from 5
January until 31 March 2004. The second phase, Field Studies and Technical Studies, is
from 15 April 2004 until September 30. The third phase, which overlaps with the second
phase, is the preparation of the validation of the technical reports and is from 15
September through November. The fourth and final phase, the presentation of the initial
draft synthesis report and the preparation of the final report is to be completed by
January 2005.
The project involves a team of international evaluators and a group of local and regional
evaluators will be added to the team. The evaluation is an external activity-based
evaluation, which examines the international context and the experience and
performance of ITC over the last decade and which will make recommendations for the
future.
1.2 Outline
The next section of the Draft Final Inception Report, Chapter 2, examines the
Background and Rationale for the Evaluation. The background analysis in the Draft
Final Inception Report is supplemented by, and draws upon, the Desk Study, which
examines the context for, and the key activities of, the ITC in more depth.
Chapter 3 presents the evaluation methodology. Chapter 3 together with Annexes 1, 2,
and 3, presents the evaluation matrix.
Chapter 4 summarises the planning for the evaluation and Chapter 5 summarises
recommendations.
Joint Evaluation of the International Trade Centre
Version Final pg 7/142
2 BACKROUND AND RATIONALE
2.1 Overview of the ITC
This section provides a brief overview of the Mission, Objectives and Activities of the
International Trade Centre (ITC). The ITC is jointly sponsored by the UNCTAD and the
WTO.
2.1.1 Mission Statement
“ITC is a technical cooperation organization whose mission is to
support developing and transition economies, and particularly their
business sectors, in their efforts to realize their full potential for
developing exports and improving imports operations with the
ultimate goal of achieving sustainable development. ITC deals
specifically with the operational aspects of trade promotion and
export development.”
ITC mission statement / Mandate, www.intracen.org.
2.1.2 Objectives of ITC
The latest ITC’S strategic and specific objectives have been gathered from 2 sources: The
Business Plan 2003-2005, and the Budget program 2004-2005. The strategic level of
objective defines the broader context in which ITC, as an organisation, operates. This
broader context is mostly framed through the Millennium Development Goals (UN) and
the Doha Development Agenda (WTO).
The corporate level of objective for ITC defines the added value of ITC, the specific goals
it focuses on. At the present time the ITC has specified five long-term corporate goals,
i.e.:
Goal 1: To facilitate the integration of developing and transition economy
enterprises into the MTS;
Goal 2: To support national efforts to design and implement trade development
strategies;
Goal 3: To strengthen key trade support services, both public and private;
Goal 4: To improve sector export performance in sectors of critical importance
and opportunity;
Goal 5: To foster international competitiveness within the business community
as a whole and the SME sector in particular.
Joint Evaluation of the International Trade Centre
Version Final pg 8/142
2.1.3 Origins and governance
The International Trade Centre (ITC) was created in 1964 as an affiliate to the GATT and
since 1968 has been operated jointly by GATT (replaced by the WTO in January 1995)
and the United Nations through UNCTAD. ITC is responsible in the UN system for
implementing technical assistance projects in developing countries and economies in
transition related to trade promotion with a view to expanding their exports and
improving their import operations. ITC mandate gives priority to supporting trade
development efforts of the least developed countries with, more recently, a focus on
transition economies. ITC receives its core funding from the UN and WTO, and is also
funded by donors and by UNDP through ITC’s trust funds.
2.1.4 The last Decade
Following a protracted period in the early 1990s, during which the ITC suffered from a
lack of direction, and which led to a significant drop in funding, a major priority–setting
exercise was undertaken in 1994. The outcome of this was the publication of the initial
strategic orientation document entitled ITC at the Crossroads, an Agenda for the Future,
which was presented to the Joint Advisory Group (JAG) in November 1994.
Subsequently, the new priorities were approved by the JAG in May 1995 in the Priority
Setting Review.
1994-95 Strategic Review
The 1994-95 review acknowledged the fact that developing countries generally were
being left behind in terms of their share of, and participation in, world trade. The
economic reform programmes implemented at the time were not in and of themselves
sufficient – they needed to be based on effective trade development strategies and
reinforced with trade support initiatives. At the same time, the approach to technical
assistance had changed substantially since the formation of ITC 30 years previously, and
many new providers of TA had emerged.
ITC’s technical cooperation programme was identified as comprising six core
services or areas of intervention, specifically:
• Trade development strategy and design;
• Trade Information;
• Export product and market development;
• Development of trade support services;
• Import management;
• Human resource development.
Joint Evaluation of the International Trade Centre
Version Final pg 9/142
Within each of these six core areas of intervention, various priority activities were then
identified.
The main elements for revitalising ITC were summarised as follows:
• Becoming more demand driven and enhancing ITC’s orientation towards
recipients;
• Strengthening ITC’s core services;
• Fully mobilising ITC’s human resources (e.g. through interdivisional
collaboration);
• Restructuring and streamlining ITC’s operations, by modifying the
organisational structure and related measures;
• Enhanced mobilisation of financial resources through measures such as the
proposal for a Global Trust Fund, a review of donors priorities, and other
initiatives to reduce the dependence on UNDP.
As part of the restructuring, and in order to increase the resources available for technical
cooperation with developing countries and countries in transition, the Global Trust
Fund was established in 1995. This is the primary channel through which donor country
funding is channelled, and it forms the primary source/pool of financing for ITC field
operations.
Clearly the immediate impetus to the 1994 Strategic review was the internal
management challenges and the sharp decline in funding reflecting both issues specific
to ITC and broader issues in the UN system. However, the external context for the ITC
was changing in the 1990s. It is worth noting that the Priority Setting Review took note of
the external context and explicitly raised the issue of the comparative advantage of the
ITC.
The Priority Setting Review adopted by the JAG stated:
“ITC will devote resources only to areas in which it has
comparative advantage. Among those identified in the Agenda (for
the Future) paper are: its substantive expertise in trade promotion
and export development for developing countries; its impartiality
and multilateral approach; its access to policy- and decision-makers
in developed and developing countries; its thorough
understanding, born of long and extensive experience, of the
interlinkages between the global, macro- and microeconomic
aspects of international trade; its proven ability to formulate and
implement trade promotion and export development programmes;
Joint Evaluation of the International Trade Centre
Version Final pg 10/142
and its wide-ranging contacts. Further prioritization of ITC’s future
areas of specialization was made on the basis of the following
criteria: relevance to the most pressing needs of recipient countries;
the likelihood of making a visible contribution to trade
development efforts; the probability of attaining a critical mass of
related technical expertise in ITC; the possibility of achieving an
acceptable expenditure/impact ratio (cost-effectiveness); and the
potential for generating extrabudgetary resources for activities at
the country level.”
Priority Setting Review, pp. 1-2, paragraph 4
If we are to consider what the comparative advantage of ITC is today and will be in the
future, we need to consider how the external context for the ITC has evolved and is
likely to evolve in the future.
The External Context in the 1990s
In the mid-1990s the role of the ITC evolved due to external factors including
developments in the global trading system and broader economic and political
developments. Key developments include the following:
• the fall of the Berlin Wall and the collapse of the Soviet Union created a new set of
challenges to support the integration of transition economies into the global system;
• new regional trading initiatives, including the North American Free Trade
Agreement and the enlargement of the European Union, created new competitive
pressures for trade and investment;
• new concerns about environmental, technical and sanitary and phytosanitary
standards in the markets of developed countries;
• the development and widespread application of new information and
communications technologies; and
• the completion of the Uruguay Round of multilateral negotiations and the creation of
the World Trade Organization (WTO) on January 1, 1995, with a more comprehensive
set of multilateral trading rules.
Other important developments during the 1990s, but which may have received less
attention at the time, included:
• the large queue of about 30 countries including China wishing to accede to the
WTO;
Joint Evaluation of the International Trade Centre
Version Final pg 11/142
• the implications, for governance and for the business communities in developing
and transition economies, of the developments of the Multilateral Trading System
(MTS);
• the links between trade enabling policy environment and trade development; and
• the broader social and development impacts of trade development
Within this overall context, the developments in the MTS were of greatest direct
relevance to the evolution of the role of the ITC. After the creation of the WTO in 1995,
the role of the ITC was adapted to the new configuration of international organisations.
In particular, there were discussions among the WTO, UNCTAD and the ITC about their
respective roles, and new approaches were developed for co-operation among these
agencies.
The Joint Integrated Technical Assistance Program, (JITAP), which was a specific
response to a proposal at the meeting of African Trade Ministers in Tunis in October
1994, reflected a different approach to supporting African LDCs to meet the challenges
of participation in the trading system with the creation of the WTO. The JITAP, which
involved collaborative identification activities among the organisations, was launched as
a joint activity of the WTO, UNCTAD and the ITC in 1998. The WTO and UNCTAD are
partners with ITC in JITAP, which in phase I operated in eight African countries, and an
additional eight countries were added in phase II launched in 2003. A broadly positive
evaluation of JITAP was carried out in 2002, although a number of issues were raised in
the evaluation.
The launch of the Integrated Framework (IF) represented a similar shift of approach, in
recognition of the shortcomings of a piecemeal approach to trade related development.
The political impetus for the IF came from the 1996 Singapore WTO Ministerial
Declaration, which called for a high level meeting on LDCs involving the WTO,
UNCTAD and the ITC. The IF was formally launched at the high-level meeting on
LDCs at the WTO in late 1997. The IF recognises the interdependencies involved and the
need for an integrated approach, with the ITC’s emphasis on trade promotion combined
with the respective strengths of the World Bank, the IMF, WTO, UNCTAD and UNDP.
In turn the IF is also intended to be a catalyst for bilateral TRTA projects in the
participating countries.
Recent Developments and the Present Context
The context within which the ITC operates today is different from that of 10 years ago in
some important respects. Some trends have intensified while other developments have
declined in significance. The overall strategic context is considered more extensively in
the desk study and this section only summarises the key points.
Joint Evaluation of the International Trade Centre
Version Final pg 12/142
The most directly relevant development to the technical assistance activities of ITC is the
debate about the links between trade and development and the expansion of TRTA.
There is now within the mainstream international development community a much
greater acceptance of the importance of trade enhancement and of business sector
support as development tools. At the same time, there has been a substantial expansion
in trade related technical assistance (TRTA), which now runs to several billions of
dollars annually (and of which the ITC expenditure constitutes a very minor share).
In terms of the specific funding of ITC, these changes are manifest in shifts in the sources
of funding. In contrast to ten years ago (or fifteen years ago when the budget was
larger), the ITC operations are now much more dependent upon bilateral donors and the
funding from UNDP has declined significantly.
A notable recent development has been an increase in the variety and number of donor
countries coming forward, following many years in which the Scandinavian countries
and the Netherlands comprised the core-funding base. For example, though on a small
scale, the USA funding commitment in 2003 could be seen as a breakthrough. Also, after
many years of expectancy, the EU has committed €5 million over 3-4 years under an
Asia Trust Fund. Switzerland, the UK, Canada, France, and Germany are now much
more to the fore than in earlier years.
According to the most recent documentation available, the regional expenditure
breakdown of operations funds is as follows overleaf:
Joint Evaluation of the International Trade Centre
Version Final pg 13/142
Regional Distribution of Activities 2001 2002
Africa - 43% Africa - 43%
Asia/Pacific - 13%
Asia/Pacific - 15%
Latin America and the Caribbean -
13%
Latin America and the Caribbean -
13%
CEE and CIS - 19%
CEE and CIS - 14%
Arab States - 5% Arab States - 7%
Global - 7% Global - 8%
Types of services provided
According to the ITC Business Plan 2003-2005, the ITC currently delivers its services
under seven programme headings, as follows:
• Strategic and Operational Market Research: It is designed to support priority
setting by trade strategists and managers of trade support institutions.
• Business Advisory Services: This is aimed at supporting participation by
developing countries in the MTS, with the support to participation in the MTS
being coordinated under the World Trade Net system, and at supporting
competitiveness at the enterprise level.
• Trade Information Management: This programme concentrates on capacity
building and networking activities within LDCs in the trade information area.
• Export Training Capacity Development: This programme focuses on
developing the competencies of enterprise managers to ‘nurture’ their export
business.
• Sector Specific Product and Market Development: This programme focuses on
supply-side constraints (e.g. high costs of doing business) to improving a
sector’s export performance.
• Trade In Services: The focus here is increasing awareness for service sector
opportunities and enhancing capabilities of services industries to participate in
international commerce.
• International Purchasing and Supply Chain Management: This programme
focuses on improving purchase and supply operations with the aim of
enhancing the reliability and competitiveness of export delivery.
2.1.5 Strategic Issues
Joint Evaluation of the International Trade Centre
Version Final pg 14/142
The ITC faces a continuous challenge to appropriately position itself within a dynamic
trade policy and trade promotion environment. (These challenges are addressed in more
detail in the desk study and only the key points are summarised here.) Its programmes
and activities need to be continuously monitored to ensure ongoing relevance in this
changing environment. This evaluation will incorporate into its approach an assessment
of the key strategic considerations that the future evolution of ITC should
‘acknowledge’. Foremost among these are the following:
1) Developments in the Multilateral Trading System
The changing role of the ITC is linked to broader changes in the Multilateral Trading
System, which include the following developments:
• The mid-1990s saw the rapid evolution of the new multilateral trading system (MTS)
including the creation of the World Trade Organization (WTO) in 1995. The trading
rules were changed and expanded substantially under the WTO and developing
countries were subject to a fuller range of obligations under the trade rules, but
many of these obligations only came into effect after 2000 or even later, due to
transition provisions negotiated in the Uruguay Round. The accession of many
countries to the WTO has brought many countries fully into the MTS for the first
time.
• Related to the wider application of trading rules under the WTO there was
considerable apprehension in developing countries and more particularly in the
least-developed and low-income countries and in civil society in developed and
developing countries, regarding the implications of the World Trade Organisation
and the rules which would be negotiated and applied. These concerns still dominate
the landscape of International trade and are important factors influencing the
negotiations on the Doha Development Agenda.
• These concerns and negative perceptions of some developing countries have
manifested themselves in different ways at the Seattle, Doha and Cancun Ministerial
meetings of the WTO. At the Seattle Ministerial Meeting the concerns of developing
countries and anti-globalisation activists resulted in a blockage of the negotiations. In
contrast, at the Doha Ministerial Conference the result was a complex negotiating
agenda, -- the Doha Development Agenda (DDA) which attempted to respond to the
concerns of developing countries about WTO implementation issues associated with
the implementation of the Uruguay Round commitments and to negotiate new
trading rules for developing countries. The inconclusive outcome of the Cancun
Ministerial Meeting reflects the complex challenges of integrating developing
countries into the trading system and differences of view on how to adapt the
trading rules resulting from the Uruguay Round and on how to deal with difficult
challenges such as negotiations on agricultural trade.
The ITC’s mission includes assisting with the integration of developing countries into
the MTS, and supporting the business communities to understand the significance of,
Joint Evaluation of the International Trade Centre
Version Final pg 15/142
and participate in, the development of trade policies and to prepare for international
negotiations. This has been reflected in new activities of the ITC such as regional
business fora on trade negotiating issues.
2) Developments in regional integration
Delays in the multilateral negotiations under Doha, do not block, and may also act to
stimulate, regional trading initiatives with important implications for developing
countries. Examples of such initiatives include:
• The process of enlargement of the European Union to Central and Eastern
Europe;
• Implementation of existing regional arrangements such as NAFTA and AFTA.
• The prospective negotiation of regional trading arrangements between China
and AFTA;
• Negotiation of bilateral FTAs by the EU with trading partners such as Mexico as
well as negotiations with MEDA partners;
• The negotiation of bilateral FTAs by the United States with various countries or
groups of countries including Central American countries, Singapore and
Thailand respectively;
• The prospective negotiation of the Economic Partnership Agreements (EPAs)
under the Cotonou Agreement between the EU and the ACP countries;
• Prospective regional arrangements in APEC and the Americas;
• Various regional initiatives in Africa; and
• Negotiations within SAARC in South Asia.
3) The recognition of linkages in the role of trade development and the broader process
of economic development.
• The expansion of TRTA globally and by multiple agencies reflects the recognition
of the link between trade and broader sustainable development objectives.
• A re-examination of the link between trade and human development is
underway, both within and outside the UN. The widening human development
mandate in terms of poverty alleviation has potential implications for the process
of trade development.
• While the process of globalisation has given added urgency to the need to
address issues of trade and poverty in LDC countries, the MTS has also
strengthened the global consensus, in the developed world and among the
donors and the development assistance community, that trade, poverty,
development and capacity-building issues need to be addressed together, in a
holistic manner, thus promoting the integrated approach. The Integrated
Joint Evaluation of the International Trade Centre
Version Final pg 16/142
Framework, for instance, is a novel and innovative approach to development
assistance that reflects this new approach2.
• Related to changing approaches to development are the changing approaches of
donors to support the multilateral technical assistance organisations and
international financial institutions and the changing approaches of donors to the
programming bilateral technical assistance activities.
4) The widespread diffusion of information and communications technologies
The development and application of information technologies has important
implications for developing countries and direct relevance to the core competencies of
ITC. Among relevant factors are:
• The disparities in telecommunications infrastructure and Internet connectivity
between developed and developing countries and the associated technology
deficit and digital divide between the developed and developing world, and the
impact that has on ITC’s mission.
• The impact of e-commerce in reshaping global trade patterns and possibilities
(e.g. in reversing long standing situational disadvantages, c.f. outsourcing of IT
services).
• The changes in the relative costs of, and access to, business and trade
information.
5) Broader changes in business strategies and practices in response to changes in the
trade and business environment and in response to the competitive opportunities
and challenges associated with the new technologies.
Some of the most prominent factors are:
• The growing interconnectedness of investment and trade flows. Foreign Direct
Investment can be a powerful tool for trade development and FDI has been very
important in supporting the expansion of exports in regions as diverse as China,
Mexico and the Central European economies acceding to the European Union.
Over the last decade Foreign Direct investment has grown much more rapidly
than international trade, which in turn has grown more rapidly than national
economies.
• The development of global supply chains, which have important implications for
both marketing and business organisation and quality standards.
2 The integrated Framework aims at linking trade to development and poverty reduction by coordinated
interventions of all the major actors of development assistance, namely: the World Bank, the IMF, the
WTO, the UNDP, UNCTAD and ITC. One of the innovative aspects of the project is the participatory
approach, where beneficiaries are fully engaged in the development process by linking technical assistance
to their national strategies of poverty reduction, and where bi and multi lateral donors are mentors to the
process.
Joint Evaluation of the International Trade Centre
Version Final pg 17/142
• The appropriateness of ITC’s focus - in particular on SMEs and TPOs in light of
the changing technologies and business practices.
All of these factors are relevant to the consideration of the present and future role of the
ITC.
If, for example we focus on the last point mentioned above, the role of national TPOs is
constantly evolving in the new, e-commerce and internet-based, trading environment.
National TPOs have been the main organisational counterpart to the ITC in beneficiary
countries. ITC’s relationship with these organisations must be examined, and it must be
determined whether they remain the optimal vehicle for the achievement of ITC goals.
For example, changes in information and communications technology greatly facilitate
self-generated business trade ventures and research, independently of the support
network of TPOs.
Furthermore, given the increasing interconnectedness of investment and trade, should
national investment agencies be brought more to the fore? In many countries trade
promotion and investment promotion functions have been merged, yet ITC does not
address investment promotion. In many countries also, the client firm base of TPOs may
not be fully representative of the wider prospective export community.
These factors impinging on the strategic context for the ITC issues are explored in more
detail in the Desk Study, but our results clearly are preliminary at this stage. A scan of
the international context for the ITC is a major focus of this evaluation and will be a
major focus of the field and technical studies.
2.2 Objectives of the Evaluation
The objectives of the Evaluation are to learn from experience and to contribute to the
enhanced effectiveness of the ITC’s programs and activities.
The Terms of Reference state the following objectives:
• The first main objective of the evaluation is to determine ITC’s results,
and success in fulfilling its objectives (see … above) by measuring output,
outcome and impact to the extent possible.
• The second main objective is to determine the comparative advantage of
the organization within the international trade and development community,
which has changed significantly, especially during the past decade.
Joint Evaluation of the International Trade Centre
Version Final pg 18/142
• The evaluation is further expected to offer recommendations on priority
setting and on steps ITC should take to improve its future performance.
2.3 Key Evaluation Questions from the ToRs
There are three broad sets of evaluative questions under the Terms of Reference
Q1: What is the Comparative Advantage of the ITC?
Q2: To assess the performance of ITC interventions and whether
they are achieving their objectives?
Q3: How effective is the management capacity of ITC and what are
the implications of donor-funding modalities?
Particular attention needs to be given to the latest developments of ITC interventions:
1. Old and new geographical coverage (issues of level from national to cross-border to
regional, evolution of transition economies, Doha Development Agenda)
2. Guiding principles and coordination with emerging common concerns (of the UN, of
the international donor community, Millennium Development Goals)
3. Impact of new information technologies on trade promotion activities
4. Old and new partnerships (exploring long term partnerships, new structures and
modalities). Evaluation of the comparative advantage of ITC in the post Doha
environment. Synergies and risks.
5. Enhancement of performance through state-of-the art management practices, use of
best practices as per lessons learned, use of evaluation as basis for enhanced
management.
Joint Evaluation of the International Trade Centre
Version Final pg 19/142
3 Methodology
3.1 Introduction
We propose to use a mix of evaluation methods and tools. The primary evaluation
model will be the activity-based Results-Based Management Model; an Evaluation
Matrix is developed below to organize the evaluation of inputs, activities, outcomes and
impacts. Special attention will be paid to sampling in order to minimize biased
reporting. Where possible we will use triangulation to verify facts and observations by
obtaining different sources of information. Complementary tools and approaches
including the SWOT analysis and the Budgetary Analysis, will supplement the RBM
approach.
3.2 Description and Choice of Methodologies
3.2.1 Uniqueness of ITC
ITC appears to be unique in a number of respects, which we will strive to reflect in the
evaluation approach. These distinctive aspects are outlined below.
Development/business hybrid? ITC is a UN technical cooperation organisation whose
mission is to support developing and transition economies, and particularly their
business sectors, in their efforts to increase exports and to improve import operations.
As such, ITC combines a Development and Business mind set-, and terminology, tools,
evaluation criteria and management processes.
ITC in TRTA? ITC specialises in Trade Related Technical Assistance (TRTA), a field
undergoing rapid change since the commitment of Governments to the Doha
Development Agenda and the Millennium Development Goals (MDG).
No universally accepted and comprehensive common framework of what is TRTA
exists. Some indicators are provided with the recent development of the OECD database
on TRTA, which is one approach to benchmarking ITC interventions. This particular
framework however needs to be validated with the empirical realities of ITC
interventions and the limitations of the OECD data need to be taken into account.
With about 185 staff, and an annual core budget of around 33 million USD, ITC is a
relatively small UN organisation. However, it is not necessarily small when compared
to other TRTA partners and similar service providers. Yet it is working in many
countries and in different areas of competence.
Joint Evaluation of the International Trade Centre
Version Final pg 20/142
A changing organisation! ITC has also been challenged in a number of different ways
and in different areas in recent years. Examples include the requirement to:
• adopt results-based management,
• respond to an increase in the number and diversity of requests from the field,
• rebuild credibility after the organisational crisis of the mid 1990s
• raise funds to increase its capacity,
• be recognized as a centre of excellence for trade development, with highest
standards of efficiency, competence, integrity and neutrality, and
• respond to broader issues in the evolution of the trading system and global
development concerns.
The organisation’s response to these pressures is reflected in on-going changes to
corporate documents, in the creation of new units, which reshape existing divisions, and
in the addition of new programmes to the seven primary ones. These developments
have had both positive and negative consequences: ITC appears to be very dynamic,
adaptive, and innovative, but this adaptability can also be confusing to newcomers and
donors, leading to such questions as: what has really changed, what is the real added
value of ITC, and where is ITC headed over the longer term?
These questions are very specific to ITC and have affected the methodology adopted for
the evaluation. We have adopted a TRTA result-based reference framework broken
down to the key common denominators of TRTA. This is presented as an open results-
based reference framework (not a blueprint), to permit elements captured during the
field study to enrich the reference framework of TRTA and better assess ITC’s position.
The overall strategy is to build upon the most recent conceptual models and tools of
development and business evaluations.
3.2.2 Stakeholders
ITC Stakeholders are those agencies and organisations that have a direct or indirect
interest in the ITC intervention or its evaluation3. Six types of stakeholders are identified:
• Parent body (WTO, UNCTAD)
• Donors to ITC, or to beneficiary countries (bilateral, regional development banks)
• Executive (Management and staff)
• Cooperation partners, peers, similar service providers
• Direct beneficiaries or clients of services provided by ITC (e.g. trade promotion
organisations or trade support institutions, business associations, or partner
3 OECD Glossary of evaluation terms
Joint Evaluation of the International Trade Centre
Version Final pg 21/142
networks). Typically ITC works with intermediaries located at regional or
country levels to increase its outreach capacity.
• End-users (e.g. firms, enterprises, and particularly SMEs).
Stakeholders’ perspective is important to the evaluation of ITC as they all reflect
differently on the success or otherwise of ITC’s activities. Our approach is to integrate
the participation of stakeholders at each phase: design of the evaluation matrix, data
generation, analysis, and feedback.
Stakeholders and their Expectations.
Type of stakeholder Lists Expectations towards the
evaluation
Parent body - WTO,
- UNCTAD
- Comparative advantage of
ITC
Donors - Bilateral donors
- Regional
development banks
- Comparative advantage
- Performance
- Modalities of funding
- Transparency
Executive and staff - SMC,
- Individual ITC
Sections
- ITC’s position in TRTA,
Impact measurement,
funding stability
- Organisation structure
Partners, peers,
similar service
providers
- Multilateral agencies,
- Bilateral agencies,
- NGOs,
- Universities,
- private and public
sector in beneficiary
countries
To be identified during field
study
Direct beneficiaries - Public and private
Trade Support
Institutions (TSIs),
- business associations,
- chambers of
commerce,
- training firms,
- visible networks,
- trade and industry
associations,
To be identified during field
study
Joint Evaluation of the International Trade Centre
Version Final pg 22/142
- ITC liaison, etc.
(1500 registered in ITC’S
TSI database)
End-users Enterprises, particularly
SMEs
To be identified during field
study
3.2.3 Approach to the evaluation
The approach to the evaluation will be (1) independent, (2) participatory, (3) output and
results oriented,(4) comparative and (5) forward oriented.
(1) Independent. The evaluation will be independent, carried out by entities and
persons free of the control of those responsible for the design and implementation of
ITC’s intervention. Independence implies freedom from political influence and
organizational pressures and incentives. It is characterized by full access to information
and by full autonomy in carrying out investigations and reporting findings4.
(2) Participatory. A participatory process will be facilitated and led by the Evaluation
Team. Key types of stakeholders at each level of implementation will be part of the
evaluation process at each phase: building the evaluation matrix, gathering data,
analysis and feedback. This participatory approach has been chosen to better learn and
capitalize on existing knowledge and to provide a greater base for the ownership of
recommendations.
(3) Output and Results Oriented: “Results” are defined as “outcomes, effects and
impacts” or all “changes induced by ITC’s activities”. Changes can be positive or
negative, primary or secondary, short or long term, resulting from a development
intervention5. Impact indicators are identified in the evaluation matrix. Measurements
refer to changes in beneficiaries’ capacities, which is the first level of impact, and
changes in end-users capacities, and in export performance and other development
concerns at the enterprise level, which are second level impacts. Assessment of results
will be done with the beneficiaries and end-users.
(4) Evaluation Criteria: The evaluation methodology will permit the assessment of ITC,
both in terms of its comparative advantage and its performance with regards to five of
the criteria generally adopted by all major donor agencies6:
4 OECD’s 2002 glossary
5 OECD / DAC glossary.
6 Definitions given here were extracted from: EU – Evaluation guidelines, UNDP HandBook on
Monitoring and Evaluating for results, DANIDA Evaluation guidelines. The criteria of impact won’t be
Joint Evaluation of the International Trade Centre
Version Final pg 23/142
• Relevance: relates to a project/activity design and the extent to which its
stated objectives correctly address real needs and remain valid owing under
changing circumstances;
• Efficiency: relates to how well the various activities transform resources
(inputs) into outputs. The Budget Analysis will seek to benchmark cost per
type of deliverable and assess efficiency of ITC.
• Effectiveness: relates to the degree to which a project achieves its planned
results (goals, purposes and outputs) and how far were the project’s results
used by the beneficiaries (outreach). Cost-effectiveness is the relation
between the costs (inputs) and results produced by a project. A project is
more cost-effective when it achieves its results at the lowest possible cost
compared with alternative projects with the same intended results.
• Impact: relates to the longer term effects produced by the intervention,
whether positive or negative, directly or indirectly and intended or
unintended.
• Sustainability: Will the positive results of the project continue once the
project ends and what is needed to ensure continuity, and the multiplier
effect (replicability or scaling-up) to achieve wider impacts for the
region/nation as a whole.
In addition since ITC is working with or supporting networks both among countries and
within countries, we will also assess the outreach effects of ITC activities and networks.
This is an important aspect of output as noted above, but it is also linked to
sustainability.
(5) Forward Oriented: The analysis will provide a basis for identifying a range of
options for strengthening the comparative advantage of ITC, enhancing the results and
impact of its activities and developing the capacity of ITC and donor funding modalities.
Focus will be placed on matching the strengths of ITC and future opportunities.
used as stated in these sources, but is related to “effectiveness” in the proposed methodology for ITC’S
evaluation.
Joint Evaluation of the International Trade Centre
Version Final pg 24/142
3.3 Overview of the Methodology
Evaluation Methodology at a Glance
External
Environment
of ITC
ExternalExternal
EnvironmentEnvironment
of ITCof ITCITCITCITC
Matching Strengths of ITC with
Opportunities in its environment
Matching Strengths of ITC with
Opportunities in its environment
RECOMMENDATIONSRECOMMENDATIONS
Information
collection
Information
collection
FACTUAL FINDINGSFACTUAL FINDINGS
Analysis and AssessmentsEfficiency
Effectiveness
Relevance
Impact
Sustainability
Analysis and AssessmentsEfficiency
Effectiveness
Relevance
Impact
Sustainability
Mapping
information
Mapping
information
Structured Reference
Framework (TRTA and ITC)
Structured Reference
Framework (TRTA and ITC)
The starting point is a structured
reference framework using the
Result-Based (RB) logical
framework applied to ITC as it
stands today and to Trade Related
Technical Assistance (TRTA) as a
benchmark. This reference
framework will be used to identify
indicators measuring each level of
the causality chain: resources,
outputs, and expected changes at
specific, corporate and ultimate
goal. It will also be used to
aggregate findings7, mapping key
information gathered through the
evaluation process.
A structured set of questions will be
used to collect information and
assess the external environment, and
ITC itself, first in the field then in
headquarters. This set of key
questions, and the RB reference
framework and linked indicators
together form the Evaluation
matrix.
The analysis of the performance of interventions of the ITC and of the external context
will be conducted according to the five criteria of evaluation and will use various
evaluative tools including interviews of beneficiaries and end-users, donors and other
international organisations, various survey techniques, and will be supplemented by the
budgetary analysis, the SWOT analysis and benchmarking.
The following diagram links the results chain and the evaluation criteria.
7 Factual statements about the project. (UNDP definition)
Joint Evaluation of the International Trade Centre
Final Inception Report v final pg 25/142 15/04/2005
Logical Framework Analysis Model
Evaluation Criteria LogFrame Levels
Means (inputs) (Materiel, personnel, financial resources)
Activities (process of converting inputs into results)
Results (confirmed planned deliverables
Purpose (benefits actually received)
Overall Objectives (overall lasting change, both at project / program
level and beyond
Efficiency links: from means through activities to results
Effectiveness links: (depend on assumptions made and on outside conditions)
Impact & Sustainability links: (depend on wide assumptions made and on outside conditions)
Relevance to the identified problems or real needs to be addressed
Design & Preparation
Joint Evaluation of the International Trade Centre
Version Final
26
Recommendations will build on matching strengths of ITC with opportunities in
its environment and lessons learned.
3.4 The Evaluation Matrix
3.4.1 TRTA Results-Based - reference framework
The results – based (RB) logical framework is used to build a structured reference
framework upon which ITC will be evaluated. See the diagram overleaf.
Joint Evaluation of the International Trade Centre
Version Final
27
TRTA corporate
Goal (impact)
Ultimate goal
MDG 8 ( impact)
TA/
Capacity
building
specific
Objectives
(outcomes)
Technical
assistance
Outputs
Inputs
Types of deliverables
1. Tools
2. Information sources
3. Advisory
4. Training services
5. Networking
6. Combination of above:
Programmes,initiatives
and projects
Funding:
- Level and sources of funds
- Beneficiary cost-sharing
- Rate of disbursements
Human resource
- Staff and consultants
- Skills and competences
- Business processes
TRTA’s results based – reference framework
Management process of outputs
- Requests
- Needs assessment
- Project design and fund raising
- Partnerships
- Implementation and procurement
- M&E
- Termination
Geographic focus (scale):
Global
Regional – inter, sub
Country
Target groups
(Outreach)
-Intermediaries
-End-users
Develop a global partnership for sustainable development and poverty alleviation,
through an open trading system that is rule-based and non-discriminatory
Enhance enterprise competitiveness
and national / sector export performance
Areas of technical competences transferred to
Trade support services (network and/or
private/ public institutions) and/ or end-users
1. Export strategy
2. Multilateral trading system (MTS) knowledge
and business implications
3. Enterprise management and competitiveness
4. Product development by sector
5. Market analysis and strategic market research
6. Public-private partnership and networking
7. Trade in services
8. Trade information management
9. E-competence
10. Trade finance
11. Standards and quality management ( including
TBT and SPS)
12. Supply chain management ( including public
procurement)
13. Export packaging
14. Legal aspects
15. Institutional strengthening of TSI
16. …
Channel of delivery: (3 tracks)
1. Global delivery of generic tools and programs
2. Integrated in multiagency, multi objectives programs
3. Regional/country specific pipeline of projects and tools
Technical competences available:
1-15 Areas of TRTA
a) – e) Areas of TA in development
Areas of technical
competence
linked to development
concerns:
a) Environmental
sustainability
b) Women in
development
c) Poverty eradication
d) Economic and
technical cooperation
among developing
countries
e) Digital divide
TRTA corporate
Goal (impact)
Ultimate goal
MDG 8 ( impact)
TA/
Capacity
building
specific
Objectives
(outcomes)
Technical
assistance
Outputs
Inputs
Types of deliverables
1. Tools
2. Information sources
3. Advisory
4. Training services
5. Networking
6. Combination of above:
Programmes,initiatives
and projects
Funding:
- Level and sources of funds
- Beneficiary cost-sharing
- Rate of disbursements
Human resource
- Staff and consultants
- Skills and competences
- Business processes
TRTA’s results based – reference framework
Management process of outputs
- Requests
- Needs assessment
- Project design and fund raising
- Partnerships
- Implementation and procurement
- M&E
- Termination
Geographic focus (scale):
Global
Regional – inter, sub
Country
Target groups
(Outreach)
-Intermediaries
-End-users
Develop a global partnership for sustainable development and poverty alleviation,
through an open trading system that is rule-based and non-discriminatory
Enhance enterprise competitiveness
and national / sector export performance
Areas of technical competences transferred to
Trade support services (network and/or
private/ public institutions) and/ or end-users
1. Export strategy
2. Multilateral trading system (MTS) knowledge
and business implications
3. Enterprise management and competitiveness
4. Product development by sector
5. Market analysis and strategic market research
6. Public-private partnership and networking
7. Trade in services
8. Trade information management
9. E-competence
10. Trade finance
11. Standards and quality management ( including
TBT and SPS)
12. Supply chain management ( including public
procurement)
13. Export packaging
14. Legal aspects
15. Institutional strengthening of TSI
16. …
Channel of delivery: (3 tracks)
1. Global delivery of generic tools and programs
2. Integrated in multiagency, multi objectives programs
3. Regional/country specific pipeline of projects and tools
Technical competences available:
1-15 Areas of TRTA
a) – e) Areas of TA in development
Areas of technical
competence
linked to development
concerns:
a) Environmental
sustainability
b) Women in
development
c) Poverty eradication
d) Economic and
technical cooperation
among developing
countries
e) Digital divide
Joint Evaluation of the International Trade Centre
Version Final
28
During the desk study, corporate documents such as the Business Plan 2003-
2005, the Proposed Programme Budget for the biennium 2004 – 2005, the intranet
web sites, the 2003 Annual Report and the Organisation Chart have been
reviewed to get a clear view of what ITC is doing and why. Interviews with staff8
helped gain a clearer perspective of their activities. The OECD database on Trade
Related Technical Assistance (TRTA) initiated in 2001 has also been reviewed to
identify a common framework upon which ITC’s performance would be
compared to other similar service providers. A correspondence table9 of the
different structures shows the nuances and differences between these
perspectives. In light of the intricacies of the resulting table, it was decided to
break down ITC deliverables to their smallest component (6 types of
deliverables and 20 areas of competence, of which 15 related to trade and 5
related to development concerns). This places the focus on capacity building as
the key specific objective of any technical assistance program. It also helps
identify a corporate driving force to all trade related technical assistance (a
corporate goal of TRTA) and to build a link with the Millennium Development
Goals (MDG), particularly number 8, relating trade to sustainable development
and poverty alleviation, as the strategic or ultimate goal.
This RB reference framework will be used to benchmark ITC performance and
comparative advantage with other services providers. During the field study,
this reference framework will be used as an open tool to validate its structure
with other TRTA leaders, to better position ITC in this field of technical
assistance.
8 March 11 and 12th, with Division Directors and sections chiefs from the Divisions of Product
and Market Development (DPMD), Division of Trade Support Services (DTSS), and the Division
of Technical Co-operation Co-ordination (DTCC). Separate meetings were also held with the
Deputy Executive Director and the Director of the Division on Programme Support on budgetary
and organisational activities, with the Co-ordinator of the Executive Forum, and with the Senior
Officer for Evaluation. On March 16, a draft of the Evaluation Matrix was discussed in a
workshop with almost all the Management and Senior Staff of the ITC. Several meetings were
held with the Senior Management Committee of the ITC. 9 Annex. Correspondence table for key technical areas of competences of ITC, showing the
various ways to articulate what ITC is doing. This tool was used to help clarify and simplify the
logic of intervention of TRTA and ITC in TRTA. During the field study, the TRTA RB reference
framework will be confronted to other service providers to assess its relevance.
Joint Evaluation of the International Trade Centre
Version Final
29
The following five distinct elements within this RB reference framework have
been agreed upon:
(1) Deliverable: A deliverable is an ITC output, a service delivered by ITC. A
deliverable can be any combination of types of deliverable + areas of competence.
(2) Types of deliverables:
Defined as different types of support used to transfer competence and build
capacity:
• Tools: to support and to sustain the delivery of TRTA through partner
institutions; they include practical guides, either generic or adapted to local
requirements, methodologies and approaches, benchmarking & assessment
tools.
• Information sources: to give access to information on international trade and
business development through internet, CDROM or paper form: include web
sites, newsletters, bulletins, press abstracts, online information systems,
Trade MAP, country market analysis profile (the country MAP), Market
access map (MacMap), Product market analysis portal (Product Map),
databases, e-shop.
• Advisory: to provide technical expertise to build awareness, to carry out
needs assessments, to solve a problem, to study an issue, etc.
• Training services and modules: as referred to here, will be limited to
training modules and formal training, since training, as generally
understood, underlies any activity carried out in the technical assistance –
capacity building sector
• Networks: defined as visible networks, and includes conferences, exhibitions
like the tools of trade fairs, the Africa meet, etc.
• Programmes, initiatives and projects: incorporates a certain number of
proven tools and technical competencies under a convention, with a
perceptible time-proven impact. These programmes can be at preparation or
implementing phase
Source of information on deliverables:
In the Project Portal, the main source of information on ITC’s deliverables in a
country (2001-2004), each entry is either a project/programme or an activity
carried by an ITC division / section targeting that country. In the name of the
entry or in the brief description attached to it, we identify the types of deliverable
and areas of technical competencies. Each entry corresponds to a product
delivered at the country level. The list of projects in the sample of countries will
be completed for 5 years.
Joint Evaluation of the International Trade Centre
Version Final
30
The total amount identified in the Project’s portal database corresponds to the
amount of the project, not the amount dedicated specifically to the country.
Small amounts could then be an indication of an amount specifically invested in
one country. Negligible amount - 0 – can be found and should be assessed from
a country’s perspective.
The e-shop provides a list of all material (tools, information sources, training
material) published either on paper or e- format.
Advisory services and visible networks will have to be identified through other
means.
(3) Areas of technical competence:
Defined as a technical aspect of trade, for which a certain level of knowledge is to
be reached, and a certain number of skills10 is to be adopted depending of the
targeted audience. It is an open list for the reference framework to include areas
identified with other similar service providers during the field study.
There are 15 areas of technical competence in ITC.
1. Export strategy
2. Multilateral trading system (MTS) knowledge
3. Enterprise management and competitiveness
4. Product development by sector
5. Market analysis and strategic market research
6. Public – private partnership and networking
7. Trade in services
8. Trade information management
9. E-competence
10. Trade finance
11. Standards and quality management (including Technical Barriers to
Trade (TBT) and Sanitary and Phytosanitary measures (SPS))
12. Supply chain management (including public procurement)
13. Export packaging
14. Legal aspects
15. Institutional strengthening of TSI
A sub-group of 5 is to be added related to issues of common concern and the
Millennium Development Goals. These are cross cutting technical areas linked to
development concerns. Any trade technical competence can be linked with one
or more development area.
a) Environmental sustainability
10 See indicators of capacity building for more details.
Joint Evaluation of the International Trade Centre
Version Final
31
b) Women in development
c) Poverty eradication
d) Economic and technical cooperation among developing countries
e) Digital divide
The areas of technical competence will be assessed in two ways: as an input,
when looked at their availability in ITC staff and consultants, and as an outcome,
or a change induced by ITC’s activity, on targeted groups.
(4) Target groups: Trade Support Institutions (TSIs) are the directly targeted
group of ITC’s activity. These TSIs can be public or private service providers to
businesses. The businesses are the end-users of ITC’s services. The outreach of
ITC will be measured in terms of number of TSIs for the first level of impact, the
capacity building specific objective. The multiplier-effect will be measured in the
number of end-users reached by TSIs benefiting from ITC’s support.
TSIs can be characterised by the following criteria:
- Private or public,
- Sector-specific or general
- Main area of competency,
- Centralised, decentralised or both.
- Number of years in TRTA business
End-user beneficiaries of TSI services can be distinguished as follows:
- SME or large ( threshold for SMEs to be agreed upon)
- sector of activity
- Prospective or actual exporters
- Number of employees – women ratio and gender sensitivity
- Centralised or decentralised location in country
(5) Channel of delivery (or Tracks): there are 3 channels for delivering ITC’s
activities, also referred to as the 3 tracks:
• Track 1: Global delivery of generic tools or programs developed through
headquarters – initiated research and under its product-network approach.
(This track will include for example: development of a new programme like
Trade-in Services )
• Track 2: ITC participation in multi-agency program, through which ITC
provides specialised inputs related to Trade to a more broad based
cooperative initiative. (Importance will continue to be given to increasing
field activity through this track.)
• Track 3: Pipeline of national and sub-regional projects that concentrates on
the two-fold objectives of identifying export opportunities for particular
Joint Evaluation of the International Trade Centre
Version Final
32
product sectors and strengthening the national trade support network upon
which exploitation of these opportunities by the business sector relies. (ITC’s
first priority).
The distinction between track 1 and 3 is not always clear-cut. Track 1 might on
occasion be said to be the development stage including the test in the field, using
ITC’s own funds and global trust fund, whereas track 3 could be viewed as more
country or region adapted and funded by donors bilateral flagged funds.
Geographic focus (scale): There are 3 scales for geographic coverage of the
product delivered to the country. These are:
• Global: an access of the country to a generic product;
• Regional: an access of the country to a product that has been designed
for the region, sub-region, or inter-region specificities; and
• Country: an access of the country to a product that as been adapted to
its specific local requirements.
3.4.2 ITC - Results-Based logic of intervention
The ITC results-based logic of intervention is built around ITC’s mission, both at
the level of strategic goals, and at the level of the five (5) corporate goals, which
specify the types of changes ITC aims to achieve by delivering TRTA. The
evaluation will assess the relevance of these statements and their targets.
The ITC deliverables are structured in “programmes”, expanding from the 7
initial programmes to 17, with new ones being designed to answer evolving
requests. These programmes can be assimilated to the 15 areas of technical
competence.
Joint Evaluation of the International Trade Centre
Version Final
33
ITC’s
corporate
Goals
Mission
Technical
assistance
Outputs
Inputs Funding
Humanresource
ITC’sResultsBased– logicof intervention
Management process of outputs
To support developing and transition economies, and particularly
their business sector, in their efforts to realise their full potential for
developing exports and improving import operations
with the ultimate goal of achieving sustainable development
1. To facilitate the
integrationof
enterprises
into theMTS
2. To support national
efforts to design
and implement
trade development
strategies
3. To strenghten
key trade support
services, both
public and private
4. To improve
export performance
in sectors of critical
importance and
opportunity
5. To foster international
competitivenesswithin
thebusiness communnity
as a whole and the small
andmedium-sized
enterprise (SME) sector
in particular7 Programmes + extras:
1. Strategic and operational market research (MAS)
2. Business advisory services (BAS)
3. Trade information management
4. Export training capacity (EMDS)
5. Sector specific product development (MDS)
6. Trade in services
7. International purchasingandsupply chainmanagement (IPSMS)
8. Export LedPovertyReduction (EPRP)
9. E-TradeBridge Program
10. SouthSouthTradePromotion (SSTP)
11. Joint IntegratedTechnical Assistance Program(JITAP)
12. IntegratedFramework (IF)
13. World TradeNet (andbusiness for development) (WTN)
14. ExecutiveForum
15. CapacityBuilding andNetworkingBusiness Information Services (CAPNET/BIS)
16. Programfor Building AfricanCapacity for Trade (PACT)
17. Programfor Competitiveness Improvement of SMEs (PROCIP)
18. …
Types of deliverables:
Advisory and training services
Information sources
Tools andproducts
Global programs
Channel of
delivery
(3 tracks)
3.4.3 The key indicators
Based on this analysis of the ITC Results – Based logic of intervention, the TRTA
Results-Based reference framework clarifies what TRTA is about and what
objectives it is aiming at, with ITC’s specificities. These structured reference
frameworks are used to identify indicators for each level of the results chain:
strategic goal, corporate goals, technical assistance specific objectives, outputs
and inputs.
Joint Evaluation of the International Trade Centre
Version Final
34
Types of indicators:
Inputs:
Input Indicators are standard among many types of organisations. They are
related to financial resources, human resources, management processes and time
frames.
Outputs:
Output Indicators are common among many organisations but they are not
always measured as carefully and comprehensively as inputs. They are
measured in terms of volume, quality and outreach.
Impacts:
Technical assistance11 specific objectives are the first level of ITC’s impact, with
the measurement of changes directly linked to ITC’s activities. These can be
measured and assessed with the direct beneficiaries.
Corporate and strategic goals are the second level of impact, or the rolling effect
induced by the first changes identified. Measurements can be done at national
scale in circumstances where sufficient time has elapsed and multiplier effects
may have occurred. Measurement will be done in this evaluation at the scale of
end-users or enterprises, which directly benefited from the support of a TSI
linked to ITC’s activities. (These measurements are used at this second level of
change to guide decision-making in results-based management). These
measurements can also be compared to trends in national data and analysed
with external factors to build synergy among stakeholders as the achievements of
these goals are outside the capability of any single organisation’s action.
Issues related to indicators:
TRTA impact measurement is a relatively new field of research, which the
development community has only emphasized in recent years. The identification
of indicators is certainly the most difficult part of any evaluation process. The
indicators identified in the matrix in Annex 3, have been gathered through a
review of corporate documents, project documents or during discussion with
11 Capacity development (UNDP definition): The process by which individuals, groups,
organizations and countries develop, enhance and organize their systems, resources and
knowledge—all reflected in their abilities (individually and collectively) to perform
functions, solve problems and set and achieve objectives. Capacity development is also
referred to as capacity building or strengthening.
Joint Evaluation of the International Trade Centre
Version Final
35
ITC’s staff and review of the available literature. It is too early to build a
consensus on these indicators, as no measurement is yet available to assess their
relevance or relative significance.
The following guidelines were relied upon to help sort out the indicators from
many sources of information used by managers.
1. Indicators measuring ITC activities outputs and their outreach are relatively
straightforward to identify as they refer to tangible services delivered to
identifiable targeted audiences.
2. Results and impact Indicators should be:
• Linked to the problem that needed solving or to the success expected (within
a certain time frame in 3-5 years)
• At the scale of the basic unit of information (direct beneficiary and end-user)
for later aggregation
• Available for measurement by field stakeholder(s)
• Designed with a clear methodology of measurement
• Included in a comparative table (either before and after, actual and
normative, etc.), and
• Most important, assessed as to the relative importance and relevance of the
indicators in the decision making/ action process.
The matrix presented in the annex summarises the indicators identified for each
level of the Results-Based structure, with the type of data (quantitative and
qualitative) that will be used. (When targets were available in ITC’s documents,
they are specified and will be used during the evaluation process as one
benchmark). As there are often no baseline data, the actual status of impact
indicators will be measured, and discussed with beneficiaries against their
assessment of the situation 5 years before and 5 years in the future.
Impact measurements of other TRTA services providers will also be reviewed, to
enhance our overall understanding of the current state of the art of TRTA impact
measurement. We have identified priority indicators and we have developed
detailed questionnaires and tools for the country-level field studies in order to
make operational and to set priorities among the large number of potential
indicators. At the end of the evaluation process, these key indicators will be
reviewed to determine if these were most useful for the assessment of the
performance of ITC will be identified.
The key indicators are summarised as follow.
Joint Evaluation of the International Trade Centre
Version Final
36
ITC key indicators to be used for the evaluation:
For the ultimate goal of sustainable development and a global partnership for an
open trading system:
1. Awareness of the relevance of trade performance to overall development
2. Improved employment and region / poor outreach linked to enterprise
export expansion
3. Countries advocacy network’s capacity and effectiveness in building a rules-
based and non discriminatory MTS
For the corporate goal in TRTA, enhancing trade performance and
competitiveness:
1. Trade performance at country, sector and enterprise scale
2. Share of ITC in TRTA at country scale
At the level of capacity building :
1. Multiplier effect of intermediaries on the number of exporting enterprises,
particularly SMEs, and correlated impact on women
2. Changes in trade support services profile at national network scale and
individual TSI scale
3. TSI perception on changes in their capacity in respect to an ideal profile
identified as follows:
- Technical skills (one of the 15 +5 areas of competencies)
- Capacity to transfer knowledge
- Needs assessments / advocacy and diagnostic
- Project management, internal coordination and follow-up
- Promotion to the client and to the donor
- Networking
4. Current state of technical competence in country with respect to the future
foreseen.
5. Development of networks for exchange of information and best practice
techniques.
At the level of ITC’s outputs:
1. ITC’s portfolio of products
2. Uniqueness of ITC deliverable compared to its competitors
3. Demand: TSI needs
4. Demand: End-user needs
5. “Competitors”
Joint Evaluation of the International Trade Centre
Version Final
37
ITC’s outreach to target groups:
1. Access of ITC to intermediaries
2. Intermediaries’ awareness of ITC and access to deliverables
3. Outreach
4. Price of services – willingness of intermediaries to pay for a deliverable
And finally at the level of inputs: indicators will be on
1. financial resources
2. human resources, and
3. management processes
3.4.4 The key evaluation questions
A list of key evaluation questions has been put together to structure and
sequence the process of gathering information. This master questionnaire
is presented in the annex.
It has the following structure.
THEMES OF KEY EVALUATION QUESTIONS
ITC’s external environment
• Global enabling environment
• Enabling environment of beneficiary country
• Changing needs of the beneficiary countries
• Overall availability of services similar to ITC – TRTA
• Emerging issues
ITC’s intervention
• Specification of portfolio of ITC’s deliverables
• Outputs
• ITC target groups
• Geographical focus of ITC
• Impact of ITC interventions
• Performance of ITC deliverables
• Identification and assessment of other providers
Managerial challenges of ITC
• Strategic coherence between interventions, policies and objectives
Joint Evaluation of the International Trade Centre
Version Final
38
• Strategic planning
• Promotion capacity
• Networking and partnership capacity
• Quality issues
Functional issues
• Operations
• Product development
• Human resource management
• Financial resource management
• Monitors and control systems
• Procurement
Organisational capability
• Technical skills and competencies
• Project cycle management including monitoring and evaluation
• Transfer knowledge and capacity building
• E-competence
It takes into account evaluative questions identified in the terms of reference and
other questions raised during the desk review.
It identifies where information can be gathered: documentation review,
interviews with ITC staff, TSI and end-users in the field, other stakeholders at a
global scale and thematic studies.
It will be broken down into specific questionnaires for different types of
stakeholders. Findings will be summarised in tables according to the indicators.
3.5 Implementation Modalities
The following are the four implementation modalities for the evaluation. The
first evaluation modality involves an assessment of the external environment, the
relative role of different international organisations and partners, the perspective
of donors and the current approaches to trade development. The second
evaluation modality is an evaluation of ITC performance and comparative
advantage at the country level. The third evaluation modality is an evaluation of
the main ITC products or competencies. The fourth evaluation modality is an
evaluation of the corporate organisation and management and the budgetary
analysis.
Joint Evaluation of the International Trade Centre
Version Final
39
We use these four modalities in order to evaluate the wide range of ITC
competencies, the full scope of ITC deliverables and the three tracks of delivery,
global, multi-agency and regional/country level. We also propose to use four
modalities in evaluation in order to provide independent sources of data and
information which test, validate and triangulate the information and findings
obtained from the other modalities.
3.5.1 Assessment of ITC relative position in TRTA and of
Developed Country Donor/TSI Perspectives
The first evaluation modality is the assessment of TRTA performance and ITC
added value from the perspective of donors at headquarters level and developed
countries’ TSIs. This analysis focuses on the external context of the ITC and
builds on the sections of the Desk Study related to the external context. The ITC
is compared to other international organisations in terms of its perceived
strengths and weaknesses. Main issues include:
• How is TRTA evolving? Past, today and future? And where does ITC stand
in TRTA?
• How do donors perceive ITC?
• What are the perceived strengths and weaknesses of ITC?
• How do ITC products compare with the products of other international
organisations?
• When and how does ITC partner with other international organizations? Is
there overlap? What is the comparative advantage of ITC? Where and when
are their complementarities?
• What are the current approaches to the management and dissemination of
trade information and to trade development in developed countries?
There will be technical study examining the perspectives of key donors and
potential partner/rival international organizations on the strengths and
weaknesses of the ITC and its perceived comparative advantage. A selected
number of international organisations including WTO, UNCTAD, ISO, ITU,
UNIDO, OAS, Commonwealth Secretariat, WIPO, and IFIS will be interviewed
or surveyed to determine their detailed work programs and potential
complementarities. In addition a selected number of donor countries will be
interviewed and/or receive questionnaires and the views of NGOs will be sought
and their input encouraged.
There will be a technical study examining the perspective of developed country
TPOs on the management of trade information and assessing the current
models for trade development, and the effectiveness of trade development
approaches for developing countries. ITC services are predominantly provided
Joint Evaluation of the International Trade Centre
Version Final
40
through the medium of developing country TSIs, most often the country’s
national TPO. The calibre and capability of these TSIs have a crucial bearing on
the overall effectiveness of ITC interventions. While we cannot evaluate these
institutions comprehensively, we will combine the field interviews of TPOs/TSIs
(under 3.5.2 below) with a selected number of interviews with European TPOs to
assess the suitability and appropriateness of the TPO delivery model in
developing countries.
3.5.2 Assessment of ITC performance and comparative
advantage at the Country or Regional Level
The second modality for implementation of the evaluation is the evaluation from
the individual Country perspective of ITC’s performance, and its comparative
advantage (vis a vis other TRTA suppliers) at the country or regional level. This
country-level evaluation will be carried out in 10 countries, (see discussion of
country selection in the Annex), the results of which will permit comparison:
• Of the ITC portfolio of products (in terms of the frequency of use of different
ones, and their effectiveness); highlighting the limiting factors of human,
institutional and financial constraints
• Of results between LDC, OLIC, LMIC and CEE/CIS countries, considering
the importance of country-specific environmental and structural factors;
• Between regions and countries: importance of external factors. Some
perspectives will also be available through this study:
• On the channel of delivery integrated in multi-agencies and multi-objectives
programmes. (track 2) ( for example in Burkina Faso, Tanzania, Kenya); and
• Of the performance at the country level of the horizontal product areas in the
ITC evaluation as discussed in the next section.
The country level evaluations will be conducted on the basis of the Country
Evaluation tools developed detailed questionnaires and data sheets for the Core
Evaluation Team Member and the Local/Regional Consultant. The CE tools and
data sheets are discussed below.
3.5.3 Assessment of ITC Products
The third implementation modality is an assessment of the ITC performance
based on horizontal or functional analysis of the technical capacity and
competence of ITC focusing on selected services and products of ITC. Since
much of ITC services are provided through Track 1, the global track, it is
necessary to evaluate these products or services at headquarters and to assess the
products and their dissemination and use also for Track 2, through working with
Joint Evaluation of the International Trade Centre
Version Final
41
other agencies. For this analysis the products and services of ITC will be grouped
into the following clusters:
� Product and Market Development, Strategic Market Analysis, Trade
Information Management and Trade in Services comprise four of the seven
programmes of the ITC. For these programmes or services a series of specific
e-questionnaires will be utilised in order to assess their utilisation by
Beneficiaries and End-Users. The products and services will be compared to,
and bench marked against, other comparable products and services through
various techniques including expert peer review. Main questions are related
to: who are the users, what is the quality and added value of these products
for the beneficiaries, compared to similar providers.
• Participation in the Trading System, Business Framework, National Export
Strategies and Networks comprise another important cluster of products. The
ITC has two complementary roles of supporting business dialogue and
networks about the trading regime and of supporting business enterprises in
developing countries to be able to function effectively in international
commerce. The evaluation of these products will examine the quality and
utility of the ITC tools and advisory services in these areas, the effectiveness
of ITC in developing networks and the effectiveness of dialogue and
information exchange within these networks. The business advocacy
capacity and effective integration is evaluated based on the Business for
Development Forum held in Rio de Janeiro in advance of the UNCTAD 11
meeting in Brazil in June. Specific ITC competences to be evaluated include
competences (1) export strategy, (2) Multilateral Trading System, (11)
standards and quality, (12) import purchasing and supply chain
management, (13) export packaging, and (14) legal framework. The
complementarities with, and effectiveness relative to, other services
providers and international organizations will be examined.
• The SME perspective on ITC performance and comparative advantage, in the
E-trade bridge program; assessing the effectiveness in supporting enterprise
competitiveness and capacity building. In assessing Enterprise
competitiveness, the main focus is on: role, capacity and interest of TSI in
transferring knowledge to SME, and capacity building of TSI brought by ITC.
• The integration of development concerns in TRTA: lessons of success in
Export – led poverty reduction and in developing South-South Trade, with
key issues being views on links between trade and development, and ITC
competences in those fields. These projects will be evaluated based on field
work in South Africa and in China. Other development concerns will be
looked at in the country studies.
In addition to examination of these products and services at the country level,
these products will be evaluated directly through various techniques including
the evaluation of particular activities such as the Business for Development
Joint Evaluation of the International Trade Centre
Version Final
42
Forum; through evaluation of the utilization of the ITC web site or publications;
through the use of e-questionnaires and surveys to contact participants in
networks and to survey end-users and beneficiaries; and through the use of
expert peer review of particular tools and products.
3.5.4 Assessment of Corporate Organisation, Financing and
Management
The fourth implementation modality will be an assessment of the corporate
organisation, the financial constraints, the cost effectiveness of services and
products, and the management capacity. The main issues include:
• Project management cycle of outputs: the “project” concept in ITC, the
process of project identification, the internal and external coordination for
leverage, the internal quality control process, the effectiveness of monitoring
of project implementation, the internal evaluation process and learning
capacity, the accuracy and validity of the needs assessment, the continuity
and feed-back flows between ITC and field (capacity building is a long term
type of impact, while ITC’s time frame is short term), the degree of result-
based management implementation, and the management information
system (MIS) that supports the PCM and information needs for management.
• Financial resources of ITC and budgetary analysis: Predictability of funding
and project development and selection, implications of funding uncertainty
from one year to the next, donors funding mechanisms and constraints,
adequacy of financing to meet demand, cash-flow and disbursement
considerations, relative costs of services and products, allocation of
overheads, comparative costs for comparable organisations, impact of scale
of the organisation on costs, and pricing of, and willingness to pay for ITC
products. Financial allocation ratio for development of products and
technical assistance (Total budget – total costs project portal per section),
funding ratio per donor, funding ratio per section, trends, level and use of
income generating products (“revolving funds”), pricing policy of ITC
products and contribution of beneficiaries (willingness to pay) . The analysis
of financial issues will also consider governance issues for the ITC.
• Organisational structure and effectiveness, particularly the procurement
process, the human resource allocation (the ratio of staff to key outputs
which is linked to the budgetary analysis), the governance of ITC (external
and internal), the e-orientation and e-readiness of the organisation and the
use of the results of evaluation and performance appraisal at the
organisational level will be evaluated.
Joint Evaluation of the International Trade Centre
Version Final
43
3.6 Data gathering tools
The main tools that will be used to gather information are:
a) Documentation review
b) Budget analysis
c) Rapid appraisal tools with stakeholders (semi structured interviews,
country assessment workshop, corporate workshop, focus group, …) and
data sheets to facilitate data processing during and after field studies.
d) Surveys
e) Direct observation
f) Expert and peer review
g) Ranking
h) SWOT tool
a) Documentation review:
The documentation review is very important to identify:
• Objectives and measures or examples of achievements
• Beneficiaries ( intended and effective)
• Activities,
• Geographic coverage
• Services providers
• Cost and sources of funds
• Success and failures
b) Budget analysis:
ITC is a relatively small organisation, but the budgetary process is complex. The
regular budget comes half from the United Nations and half from the World
Trade Organization. In previous decades, operations funds came primarily from
UNDP, but this source of funding has declined and bilateral donors have become
the major sources of operating funding. A careful analysis of the budget
information – to the extent that detailed information is available – will be made
for identifying factual information and analysing strengths and weaknesses of
the financial resource issues in ITC. Specific issues with respect to donor funding
modalities, such as availability of multi-year funding and the implications for
ITC planning and activities will be examined.
The budget analysis will examine the revenue generation aspects of ITC products
and services and consider the pricing strategies. The analysis will also examine
the potential willingness to pay for products and services, not necessarily as a
revenue generation strategy, but as measures of potential benefits and impacts.
Joint Evaluation of the International Trade Centre
Version Final
44
c) Rapid appraisal tools (semi structured interviews, country assessment
workshop, corporate workshop, focus groups)
Semi-structured interviews will be the most common tool used to gather
information. These semi-structured interviews are most useful for developing an
in depth understanding of qualitative issues and to understand why and how
something happens, motivation, behaviour and perspectives of customers and
partners, to see how attitude can help explain successes and shortcomings, to
frame the issues.
Consistency among local consultants and core evaluators will be ensured by the
detailed methodology and tool kit provided to each of them. Regional launching
workshops will be held before starting the field study phase.
Tool kit of information for local consultant to gather:
Country data sheet
Sector data sheet if ITC has a sector programme
Detailed questionnaires
Donor data sheet
Project data sheet
TSI data sheet
Tool kit of information to core evaluators to synthesise findings:
Country evaluation data sheet
Interview data sheet
Workshop data sheet
Direct observation data sheet
Semi-structured interviews: will be carried out either individually or as focus
groups, by the local consultant and cross checked by the evaluator. They will be
carried with
• TSIs, TPO and other associations related to the project
• Other TSIs not involved with the project
• Donors
• End-users
• Other international organisations
A country workshop, facilitated by the core evaluator, and where main
stakeholders will be participating, will assess findings with respects to the 4
criteria of the evaluation, identify strengths and opportunities, and prepare
formulation of recommendations.
Joint Evaluation of the International Trade Centre
Version Final
45
A corporate workshop with the ITC will be facilitated during the last phase of
the evaluation (the final report preparation), by the evaluation team, with the
same purpose.
By combining multiple data sources, methods, analysis, evaluators seek to
overcome the bias that comes from single informant, single method, single
observer or single theory studies. The two workshops are a way to validate
findings through triangulation and transparency.
d) Surveys
Survey techniques and questionnaires will be used at both the country level and
at the horizontal/global level. The website will include an e-questionnaire and
direct surveys will be conducted of selected target groups for selected ITC
services and products. Certainly users of the ITC website will be surveyed, but
other target groups will be surveyed as a control.
The detailed questionnaires for the country studies and the horizontal/global
technical studies will be derived based on the framework of the evaluative
questions outlined in Annex 1.
e) Direct observation
A simple method: observation, made more effective by structuring its use. Much
can be learned by watching what people actually do. It is useful when:
• Performance indicators indicate results are not being accomplished as
planned
• Implementation problems are suspected but not understood
• Details of an activity need to be assessed, such as whether tasks are being
implemented to standards.
During field study phase, direct observation will be useful for certain activities
such as:
• Tools fair
• TSI network meeting
• Business for development in UNCTAD XI.
• TSI providing support to enterprises
f) Expert and peer review
Joint Evaluation of the International Trade Centre
Version Final
46
Selective use will be made of expert and specialist peer review of selected ITC
services and products.
g) Ranking
Ranking is critical when comparing elements of information on the basis of
agreed criteria. Besides the resulting matrix, one other great value of ranking
comes from the discussions that are provoked as participants come to a decision
about the final score of each option as well as on setting the criteria. In the
discussion, the reasons for preferences and rejections of options emerge.
This method will be used with small or large groups and can take from 15
minutes to one hour depending on the size of the group and how much debate
there is over the initial statement. The final outcome is a set of independent
judgments made in a non-threatening private way that will allow people to
generate a group judgment without social pressure to conform.
It will be used to measure qualitative data for impact indicators during field
study and to assess evaluation criteria during country workshop.
h) SWOT tool
The aim of the SWOT analysis tool (Strengths, Weaknesses, Opportunities and
Threats) is to capture a snapshot of the main strengths and weaknesses of an
organisation and the opportunities and threats which may affect the way in
which it works in the future. The SWOT analysis will utilise interviews at the
country level and on both interviews and expert peer review at the global level.
A structured assessment of the fit between what the organisation can (strengths)
and cannot (weaknesses) presently do, and the environmental conditions
working for (opportunities) and against (threats) ITC will be carried out.
3.6.1 Sampling
The detailed sampling approaches for the country studies are outlined in Annex
3 below. The sample selected for country studies is heavily weighted toward
countries where ITC has projects and interventions, both in global activities and
in country specific projects. The assessment of the ITC products will provide a
cross-check on the country level interventions.
Joint Evaluation of the International Trade Centre
Version Final
47
3.6.2 Evaluation steps
There are four (4) phases in the evaluation process, with overlapping between
each phase due to the participatory and adaptive approach used.
Inception phase
The purpose of this phase is to have an overview of ITC and the detailed
methodology that will be used for its evaluation. A draft inception report was
submitted to the Evaluation Management team, providing an overview of ITC,
the evaluation mandate and a draft methodology. The final inception report
finalises the evaluation methodology based on detailed information gathered
through the desk review and a corporate workshop with ITC’s staff to validate
the result-based reference framework and discuss the evaluation matrix. The
outline of the inception report is:
• Background and rationale for the evaluation
• Methodology of the evaluation
• Communication strategies
• Quality assurance methodology
• Risks, mitigating strategies and assumptions
• Project planning.
Desk review phase
The desk review phase consists in documentation review and ITC’s staff
meetings to get a clear view of what ITC is doing and aiming at, and what are the
issues to look at more deeply during the field study. During this phase, country
sampling has been made based on ITC’s portfolio of interventions, background
information has been collected, and evaluation tools have been finalised. A desk
review report has been prepared to present preliminary results on:
• Global context of Trade related technical assistance
• ITC in the international system
• ITC structure, activities and budget
• Summary of previous evaluations
• Preparation of field studies
• Documentation and sources
Field and technical study phase
During the field study phase, documentation review will continue, more
specifically focussed on documents related to the list of field studies. A regional
workshop will launch the field studies in the countries, with the purpose to train
the local consultants on the evaluation methods and tools and to provide
consistency to the work carried by more than 20 people. A typical country study
will consist in data gathering and contacts/ meetings conducted by the local
consultant, followed by a core evaluator visit to cross-check data gathered, carry
Joint Evaluation of the International Trade Centre
Version Final
48
semi-structured interviews, synthesise findings and facilitate a country
workshop to assess with stakeholders, findings with respects to the 4 evaluation
criteria and strengths and weaknesses.
A field study report with the purpose to present aggregated findings will
complete the field phase.
Synthesis phase
During the last phase, aggregated findings will be assessed at the corporate level
with data gathering on corporate issues, focus group and corporate workshop to
assess ITC with respects to the five evaluation criteria and strengths and
weaknesses, and identify recommendations based on the match between
strengths and opportunities found in the external environment.
The final report will present the evaluation results (findings and assessments)
and recommendations.
3.7 Approach to the Evaluation
The evaluation will be independent, carried out by entities and persons free of
the control of those responsible for the design and implementation of the
development intervention. Independence implies freedom from political
influence and organizational pressure. It is characterized by full access to
information and by full autonomy in carrying out investigations and reporting
findings. ( OECD’s 2002 glossary)
The evaluation methodology’s approach is based on four characteristics:
1. It will be participatory. A participatory process will be facilitated and led
by the Evaluation Team, an independent body. All types of stakeholders
at each level of implementation will be part of the evaluation process at
each phase: building the evaluation matrix, gathering data, analysing and
feedback. This participatory approach has been chosen to better learn
and capitalize on existing knowledge and to provide a greater base for
recommendations’ ownership.
2. It will be results oriented. Results include outcomes, effects and impacts.
Impacts are positive and negative, primary and secondary long term
effects produced by a development intervention, directly or indirectly,
Joint Evaluation of the International Trade Centre
Version Final
49
intended or unintended.(OECD’s glossary). Results and impact
measurements will be identified in the evaluation matrix, for the three
levels of objectives: strategic, corporate and specific. They will measure
changes induced by ITC’s outputs. Measurements will be about change in
behaviour and effects on trade and development.
3. It will build on a Bottom – up approach, based on outputs delivered by
ITC, and the responses of direct clients and end-users. Information will be
gathered at this specific scale and then aggregated for analysis at country,
regional and world wide scale.
4. It will also utilise a Top-down approach based on the SWOT Analysis
and the Budgetary Analysis to assess ITC’s objectives in the changing
global context and the impact of funding mechanisms on effective
delivery of programmes.
There might be a need to give consideration to interregional or sub-regional
levels. The desk review will help the evaluation team to determine the level of
attention needed for this specific level.
3.8 Communications Strategies
The overall approach to communications will be to provide information on the
evaluation to all potentially interested parties and to provide for opportunities
for input into the evaluation.
3.8.1 Communication with Stakeholders
The Communication with Stakeholders will occur formally through meetings
with, and reporting to, the Management Group and the Core Group constituted
to supervise and to advise the evaluation. Of course information will be made
available to the Joint Advisory Group JAG and specific reporting to the JAG is
planned. In addition to this formal channel, informal contact will be maintained
with Geneva-based Delegates from developing, transition and donor countries.
Broader contacts will be maintained with business associations and NGOs
involved in trade and development issues.
3.8.2 Public Communications and Website
There are two facets of public communications. One is media activities. Another
is the development of a web site.
Joint Evaluation of the International Trade Centre
Version Final
50
Public Communications – Media Activities
Target Audience
The objective of any communications activity is twofold:
1. first to communicate a specific message from the sender to a particular target
audience, and,
2. second, to raise awareness among secondary target audience that is closely
allied with the target audience, but is not itself a target for the message.
In the case of the ITC Evaluation, there are a number of target audiences to be
addressed. One of the main ones is composed of the various recipients of ITC
assistance over the years, as well as their clients, e.g. Trade Promotion Offices
(TPO) and their clients (exporters, importers, business associations and the like).
A second target audience is made up of the various donors and partners of the
ITC as well as the ITC itself. This includes the organizations funding the ITC
evaluation.
The secondary target audience for this project is made up of other stakeholders
in the development arena such as NGOs, national and regional governments,
potential users of ITC services, etc.
Communications Mechanisms
Given that there is a substantial amount of “noise” in all communications
channels, and given that an activity such as evaluation of the ITC is not
particularly newsworthy for media that are more attuned to disasters (natural
and otherwise), wars, and human tragedies of one kind or another, we propose
to provide information on the project in a very factual and straightforward
manner to the various target audiences.
We propose to use three communications mechanisms – two active and one
passive.
1. The first active mechanism will be direct communications with recipients
of ITC assistance. This will be in the form of e-mail, fax or paper-based
mail describing the evaluation and its progress. Certain recipients will
also be advised that they may be visited by members of the evaluation
team. Recipients will also be told that a web page is available for them to
use to monitor project progress and to allow for their participation in
project activities.
2. The second active form of communication will include circulation of brief
descriptions of activities and results of the evaluation through selected
channels such as the ITC Forum and newsletters of organisations and
NGOS.
Joint Evaluation of the International Trade Centre
Version Final
51
3. The more passive mechanism is the web site itself which is described in
the next section of this report. In addition to providing project
information, the web site will be used for conducting surveys of various
kinds and will be a feedback mechanism whereby the project team can
obtain information from a variety of audiences.
Copies of reports and executive summaries also will be available. The web site
will have both full copies and executive summaries of reports available for
downloading once the evaluation management committee has approved their
release.
Joint Evaluation of the International Trade Centre
Version Final
52
Summary – Communications Activities Information Available Target Audience Mechanism Used
Project Reports All All with website the
preferred mechanism
Project Executive
Summaries
All All with website the
preferred mechanism
Press releases Media (primarily), ITC,
its partners and
stakeholders
Newletters, website
Project progress
(activities)
Beneficiaries, followed by
ITC and its stakeholders
Website
Public Communications – website
A public-access website will be developed as a part of the project’s
communications strategy. The main objective of the website will be to ensure
transparency of the evaluation process and its results. A corollary objective is,
through the application of interactive evaluation tools, to involve a broader
audience of ITC clients, supporters and other stakeholders in the evaluation of
the ITC. A third objective is to allow any individual or organization that wishes
to, to provide feedback and comments on any component of the evaluation or of
ITC’s activities.
The main information the website will contain is as follows:
1. Information about the evaluation project (Box 1):
- Background/key issues/why the evaluation is being
undertaken
- The main objectives of the ITC evaluation/key evaluation
questions;
- The project duration and key milestones;
• Terms of reference for the evaluation:
- Appendix A of the Contract;
• Project Donors:
- Information on who the project donors are;
- Description of their role and involvement in the
evaluation/ITC work;
- project implementation monitoring bodies
• Links:
- to donor websites, such as DANIDA; WTO, UNCTAD,
UNDP, etc.
- To ITC
Joint Evaluation of the International Trade Centre
Version Final
53
- to beneficiary websites and previous project websites
2. Consortium presentation (Box 3)
- Names of the consortium partners
- Presentation of each consortium partner
- Links to websites of the consortium partners
3. Presentation of the Team (Box 2):
- Information on the evaluation team members
4. Project documents (Box 4):
- Information on the expected evaluation
outputs/documents
- Description of the evaluation methodology and main
activities to achieve the project objectives;
- Executive summaries12 and full texts of project reports
- Final Report
- Press releases
5. Feedback (Box 5)
6. Project working papers (Box 6) – available to project team members only
- Interactive tools for evaluation such as questionnaires
- Presentations/training materials for the workshops
- Report drafts
- e-mail address for project administration of various
project team members
- “Question-of-the-week” – a non-statistically valid
question on a current topic of interest to all stakeholders;
results available in real time, i.e. as soon as one has
finished “voting”
7. Site Map (Box 7)
- Quick navigation tool for users who don’t want to scroll
through menus
The website will be developed, maintained and hosted at the project head office.
The domain name www.itcevaluation.org will be registered. Website
information will be available in English. The web-site will be built on a module-
based content management system (CMS) – these kinds of solutions are more
flexible and cost effective for updates and new feature modules.
A tree diagram of the web site is shown in the following section overleaf.
12 All the reports produced during the project will be published only after the consent of the
project beneficiary.
Joint Evaluation of the International Trade Centre
Version Final
54
1. About Project 2. Consortium presentation 4. Project documents Contact us
4.1. Evaluation
methodology and
activities
4.2. Different
project reports
4.2.2. Final
Synthesis Report
4.1.1. Interactive
questionnaire
4.1.2. Links
1.4. Links
1.2. Terms of
Reference
1.3. Donors
2.5. Links
2.4. Ticon
Holdings
2.3. ACE Global
2.1. DMI
Associates
2.2. Copenhagen
Development
Consultants
Site map
4.2.1. Desk study
report
4.4. Press
Releases
3. Evaluation team
members
1.1. About ITCQuestion-of-the-week
VOTE!
Joint Evaluation of the International Trade Centre
Version Final
56
3.9 Quality Assurance
According to the Danida Evaluation Guidelines Chapter 6, at the end of the
project;
the team leader will have to indicate the extent to which:
• Danida guidelines, including the present one, have been used
• the use of methods has been discussed explicitly
• the focus, analysis and results are according to standard
• the language, layout, illustrations, etc are according to
standard
• the professional input of team members is acceptable
A quality assurance plan is attached as an Annex 6.2. The Quality Assurance
Team as proposed in the technical proposal is composed of Denise Colonna
d’Istria from DMI and Annette Munk Sorenson from Copenhagen Development
CA supplemented by the external auditors for validation of paper documents.
3.10 Risks, Mitigating Strategies and
Assumptions
3.10.1 Risks:
The main risks of the project are as follows:
� Lack of access to verifiable indicators in the field related to impact
assessment and result indicators of ITC interventions especially on older
interventions and projects.
� Tight Time frame for carrying out of field studies. The extensive activities
needed for the preparation of the field research as well as the logistics
required to carry out the field research put tremendous pressure on the
resources available during the period.
� Validity of information: there is the risk of collecting biased information
from the stakeholders.
� Reluctance of ITC staff: in cooperating with the evaluators, and
providing the data and information needed for the evaluation
� Difficulties in locating contacts: contacts and individuals who were
involved in completed projects could be impossible to track. Such persons
could provide valuable information, and insight on the impact of projects
in a particular country.
Joint Evaluation of the International Trade Centre
Version Final
57
� Cooperation with Stakeholders: the risk of lack of commitment and
contribution of stakeholders to contribute to the evaluation process
� Variant expectations: Stakeholders to be interviewed may have variant
expectations as to the outcome of the evaluation of ITC, and may be
reluctant to share their view with the Evaluation team
3.10.2 Mitigation strategies
In order to reduce the risks mentioned above, the evaluation team will adopt
the following mitigation strategy:
Participatory process: Stakeholders will be invited to round table/panel
discussions in the evaluation process. This should reduce the risk of the lack
of cooperation and contribution of the stakeholders in the evaluation process,
and create a cooperative atmosphere built on a constructive approach and
openness to the evaluator. This will also provide an opportunity for
validation of facts and discussion of preliminary findings.
Use of control groups: in order to cope with the risk of collecting biased
information, control groups will be selected and included in the evaluation
process.
Use of baseline information and desk study: Maximum use will be made of
other evaluations and published sources of information to establish baseline
information.
Reluctance of ITC staff: Close coordination with SMC has already been
established. The SMC has already issued a memorandum to ITC staff about
the evaluation being carried out and has encouraged staff to provide
requested information to the evaluators. In general the level of co-operation
and support from ITC has been excellent.
Obtaining Verifiable Indicators: Development of detailed Country
Evaluation Tools and Data Sheets and the development of e-questionnaires in
order to generate as much data and information as possible in a coherent
framework, but impact indicators are more difficult to track.
Joint Evaluation of the International Trade Centre
Version Final
58
3.10.3 Assumptions
The following assumptions have been made by the project team:
� Establishing a baseline: the evaluation team assumes that a ten year
historical record for the evaluation process is valid. ITC has
undergone a radical change after a review was carried out in 1994-
1995
� Cooperation and willingness of ITC clients and end users to
participate in the evaluation process.
� Full cooperation from ITC and transparency from management side
� Transparency of beneficiaries during field visit
3.11 Issues/Conclusions
This evaluation project is very ambitious with a global reach. It is important to
adhere to the overall time frame of the project in order that the informal results
can be presented to the informal meeting of the JAG in November. The most
problematic deliverable in terms of the time frame of the study is the completion
of the Field and Technical Study Reports. Preparation of the Field and Technical
Studies Report will require compilation and analysis of the results of Field
Studies by different teams of experts. It will be necessary to verify and cross-
check the results, and this takes time, before they can be considered as validated
findings of Phase II.
The agreed delay in the delivery date for the Field Study, however, will not affect
the timing of other deliverables under the contract. This issue was discussed
extensively and should be treated with a fair amount of flexibility.
An issue related to the budget is that of the costs of the implementation of the
web-site and some of the alternative approaches and tools for evaluation. It was
discussed at the Management Group Meeting in February that the development
of the web site and related communication activities will require additional
funds since these activities were not foreseen in the Terms of Reference.
Joint Evaluation of the International Trade Centre
Version Final
59
4 Project Planning
4.1 Workplan
4.1.1 Methodology, Timeframe and outputs
The following methodology combines the consortium evaluation strategy as
described above with the requirements of the Terms of Reference in terms of
timeframe and output:
I. Inception and Desk Study Phase (January 5- April 15, 2004):
1. Interviews with ITC staff and other organisations
2. Analysis of documentation and reports
3. Preparation of Inception Report
4. Preparation of the Desk Study Report
II. Field Studies (5 months April 15-September 30, 2004):
5. Data collection/stakeholder interviews, including beneficiaries
6. Regional workshops for core evaluators, local consultants and
specialists starting April 13-14, 2004.
7. Production of a briefing note to the annual JAG Meeting, 35th
section, planned for April 2004
8. Production of an initial Report on the Field and Technical Studies,
to include:
• Evaluation of ITC’s intervention in recipient
countries,
• Evaluation of the thematic cross-border programmes
• Evaluation of ITC’s co-operation with other
international organisations
III. Validation Analysis, and Reporting (3 months September 1 –
November 30, 2004):
9. Analysis of the findings presented in the initial comprehensive
report
10. Possible additional fact-finding
11. Production of a Synthesis Report, containing analysis, synthesis
and preliminary conclusions and recommendations.
12. Conduct of a workshop discussing the Synthesis Report
13. Production of notes from the workshop
14. Final conclusions and recommendations elaborated
15. Production of a Draft Final Report.
IV. Finalisation (December 2004 – January 2005)
Joint Evaluation of the International Trade Centre
Version Final
60
16. Reception of comments to draft final report
17. Production of final report
4.1.2 Tasks, Milestones and Reporting
Tasks/Milestones Responsible/Involved Schedule
Documents review TL + Core Team with
iterative consultations
with ST specialists,
management group and
relevant stakeholders
3 months + ongoing
process
Draft Inception report TL 10-02-04
Planning of set-up and
implementation modalities –
team meeting in Geneva
TL + Core team + some
input from ST specialists
From Feb. to March 2004
Planning of field visits and case
studies. Mobilisation of local
consultants
All Key personnel March-May 2004
Team meeting in Geneva All team members 4-5 February 2004
12-20 March 2004
Draft Final Inception Report TL + Core Team + ST
Specialists_
6 April 2004
Desk study report Input of the whole team
Internal QA: TL +
validation with
Management group
7 April 2004
Regional workshops for
preparation of country studies
TL+ Core Team + some
ST specialists
13-14 April 2004
April 21-22, 2004
May 9-10, 2004
June
Field visits implementation:
Pilot visit and full field activities
All key personnel From May to September
2004
Briefing note to Annual JAG
meeting
TL 15.04.04
Preparation of Technical and
Field studies reports
All key personnel August to mid-
September 2004
Joint Evaluation of the International Trade Centre
Version Final
61
Tasks/Milestones Responsible/Involved Schedule
Team Meeting in Geneva
Review of methodologies and
progress of the field and
technical studies.
All team members July 2004
Meetings in Geneva with ITC Team Leader, selected
Team members,
July- September 2004
Submission of Field and
Technical Studies Reports for
validation
Team Leader, Core Team 30 September 2004
Preparation of draft synthesis
Report
TL, Core Team October–mid-November
2004
Presentation and discussion of
field and technical studies with
ITC and with Management
Group and Core Group
TL, Core Team October November 2004
Review of comments TL All team
Submission of draft final
synthesis report
TL, Core Team December 2004
Finalisation of Synthesis Report TL, Core Team January 2005
Joint Evaluation of the International Trade Centre
Version Final
62
5 Recommendations
5.1 Time Frame
The only adjustment to the time frame that is proposed is to delay the
submission date for the Field and Technical Studies Report until September 17,
2004. The other deliverables will be met in the timetable identified in the Terms
of Reference. This has been discussed with the Management/Core Group.
5.2 Budget
We propose a modest addition to the reimbursable expenses budget in order to
pay for the development of the web-site and to implement e-questionnaires and
survey techniques. We propose some reallocation of the budget to accommodate
the translation of some of questionnaires and the materials for field studies. We
also propose some reallocation of the reimbursable funds to provide honoraria
for expert and peer review and to pay for communications costs. Apart from the
website and the administration and tabulation of, e-questionnaires and surveys,
we do not foresee the requirement for additional resources for fees and
reimbursable expenses.
5.3 Communications
In order to provide greater transparency and to promote a broader range of input
and feedback into the Evaluation, we are proposing an active communications
policy including a public website and providing information to the interested
specialized press, to NGOs and to the broader public as well as contact with
Geneva-based delegates, TPOs and other stakeholders in the ITC.
5.4 Mobilisation of Personnel
Within the budget constraints established in the Terms of Reference and the
Contract, the TL will assign missions and tasks to the international evaluators
and the local/regional evaluators for each phase of the evaluation, namely the
desk study phase, the field study phase and the analysis phase. The core team
members and the short term international specialists will be deployed by the TL
as appropriate to undertake specific specialized tasks with specified Terms of
Reference for each phase of the study.
It was proposed that within existing budget restraints one short term
international specialist in Results Based Management and Evaluation should be
Joint Evaluation of the International Trade Centre
Version Final
63
added to the evaluation team to assist in the preparation of the Evaluation
Matrix. Authorization for this additional expert has been granted. Additional
international short term expertise may be required in the course of the
evaluation.
Local and regional experts will be nominated to Danida by the team leader based
on input from the Consortium Members, the team members and other sources.
The local and regional experts will have specific Terms of Reference either
prepared by, or authorized by, the Team Leader.
Joint Evaluation of the International Trade Centre
Version Final
64
6 Annexes
6.1 Annex 1: Evaluation Questions
A ITC’s external environment Desk
research
Research
ITC
Field
Global
Field
Country
1 Global enabling environment
1.1 What are the external factors which may inhibit ITCs ability to achieve its goals X X X X
1.2 What is the development relevance of ITC activities in relation to the overarching
objectives defined in the Doha Development Agenda and the Millennium Declaration?
X X X
1.3 What has been the general performance of multi-agency interventions (IF and JITAP)
within trade and development work? In particular what is ITC's role in these
interventions? How is it evolving?
X X X X
1.4 What was the shift in focus of the WTO, UNCTAD during the past decade? X X X
1.5 How has it influenced the role of ITC? How has it influenced other donor agencies or
international organisations?
X X X
1.6 What is the value-added of ITC’s interventions in relation to the international trade and
development community, particularly in comparison with other international
organizations, such as WTO, UNCTAD and UNIDO?
X X X X
1.7 What is the extent to which it has been possible to achieve concrete results within the
area of trade promotion and the influence of the recent development of the
international trade agenda (trade agreements removal of barriers etc..)
X X X X
1.8 What are the stakeholders views on global policy concerns of trade and development? X X
1.9 What are the constraints due to donor priorities? X X
1.10 How are donor priorities evolving X X
1.11
1.12
How has the Implementation of WTO agreements and market access conditions for
Developing countries developed?
How have broader developments in the trading system such as regional trading
X
X
X
Joint Evaluation of the International Trade Centre
Version Final
65
initiatives affected the role of ITC with respect to the MTS
2 Enabling environment of the host countries ( physical and policy
infrastructure)
2.1 What are the constraints in trade relations in recipient countries? X X X X
2.2 Where have bottlenecks and key potential risks occurred, e.g. in the international,
regional and national enabling environment or in government policies, strategies and
plans as well as national rules and regulations and national institutional capacity?
X X X
2.3 How have national rules and regulations and national institutional capacity affected the
outcome of TRTA interventions and in particular ITC interventions?
X X X
2.4 Does ITC meet the objectives embedded in poverty reduction strategies of recipient
countries?
X X X X
2.5 How has WTO agreement and market access condition influenced developing country
trade policies?
X X X
2.6 What is the capacity of trade support infrastructure - transport, communication and
logistics in LDCs?
X X X
2.7 What is the incentive for developed countries to transfer technical knowledge on Trade
development?
X X
3 Changing needs of the developing countries and countries in transition
3.1 What are the specific changes in needs as perceived by stakeholders in host
countries?
X X X
3.2 What are the needs for diversification of export base – trends in primary commodity
dependence among developing countries, especially least developed ountries?
X
3.3 What are the stakeholders views regarding e-technologies and trade? What is the
preparedness of the country to e-business?
X
3.4 What measures are needed to reduce the digital divide? X X
3.5 How do the needs of countries change according to the level of development with
respect to ITC services?
X X X
Joint Evaluation of the International Trade Centre
Version Final
66
4 Overall availability of Services similar to ITC - TRTA
4.1 What is TRTA? How has is developed in the past and present? X
4.2 What are the subdivisions under trade development where ITC is operating? X
4.3 In what countries and for which income groups? X
4.4 What is the size of the trade development market in volume (number of activities) and
value (USD)? In particular Technical Assistance?
X
4.5 What are the main agencies operating in similar fields? X X X
4.6 How much overlap is there in the target countries/groups? X X
4.7 How to measure TRTA impacts? X X X X
4.8 What are the future trends in TRTA? X X X X
5 Emerging issues
5.1 In any of the above issues, what are the emerging issues relevant to ITC? X X
Joint Evaluation of the International Trade Centre
Version Final
67
B ITC’s intervention for each type of deliverable Desk
research
Research
ITC
Field
Global
Field
Country
1 Specification of portfolio of ITC's deliverables
1.1 For each of the categories listed below, specify the names of deliverables available from
ITC.
1- Tools: include practical guides either generic or local, methodologies and
approaches, benchmarking and assessment tools X X
2- Information sources: include websites, bulletins, databases, online information
systems, e-shop etc..
X X
3- Advisory consists of technical experts who build awareness, conduct a needs
assessment, address issues X X
4- Training: (limited to formal training services and modules provided X X
5- Networks: Defined as visible network, including conferences, exhibitions, e-
networks
X X
6- Programmes, initiatives & projects: (incorporates a number of the above
deliverables, within a time frame and a budget) Programmes can be in preparation
phase, implementation and completed phases?
X X
1.2 What is the relative importance of each deliverable with respect to financial resources
disbursed and target groups over the last 3-5 years?
X X
1.3 How has the product/service been prioritised in view of ITC growth over last 3 years? X X
1.4 For programmes, initiatives and projects, does ITC have an exit strategy for terminating
its activities/programmes in a given country?
X X X
1.5 Has the coverage of deliverables been discontinued? X X X
Joint Evaluation of the International Trade Centre
Version Final
68
2 Outputs
2.1 What are the target groups for the types of products/services offered by ITC? X X X
2.2 How has demand for ITC products and services evolved over the last five years? X X
2.3 How often are the different types of deliverables used? X X
2.4 What are the areas of technical competence where demand has increased for ITC
services over the last 3 years?
X X X
2.5 What is the quality and price of services available? X X X
2.6 What is the outreach of services offered to target groups and countries? X X X X
2.7 How do Global tools adapt to the specific types of developing countries? X X
2.8 What is the uniqueness of the deliverable? X X X
2.9 Is the timeframe for the different interventions appropriate? X X
2.10 Which types of products and services competencies at ITC provide the best synergy
when combined together?
X X
2.11 How important is the element of dissemination of a deliverable? X X
2.12 If a deliverable is fee-based - How much more would the user be willing to pay to use
the deliverable?
X X
2.13 If a deliverable is free - What could be the willingness of users to pay for the
deliverable?
X X
3 ITC target groups
3.1 Who are the main TSIs of ITC?
3.1.1 What is a TSI? What are their main characteristics? (capacity, number of members,
types of services provided, outreach, advocacy)
X X X
3.1.2 What are the new types of TSIs having an increasing role? X X X
3.1.3 How have the skills of TSIs evolved over the last years (Technical skills, capacity to
transfer knowledge, needs assessment/advocacy, project management, promotion, and
networking, e-competence)?
3.1.4 Is there a TSI network: national - international? X X
3.1.5 What is the purpose of the network, how does it operate, who are the members of the
network?
X
3.1.6 What indicators are used to measure the performance of ITC intervention on TSIs? X X X
Joint Evaluation of the International Trade Centre
Version Final
69
How does the TSI access each of the deliverables? X X
What are the strengths/weaknesses/needs of the TSI in its role as an intermediate?
3.2 TSI Relationship with ITC
3.2.1 How close it the relationship of ITC to TSIs? X X X
3.2.2 How much input does the TSI have on development of deliverables? Who identifies
needs? Who identifies sources of finance?
X X
3.2.3 Has the cooperation with TSI ever been discontinued? X X
3.2.4 What is the awareness of TSI of ITCs image, competences and deliverables? X
3.2.5 Who are other providers of support to TSI? X
3.2.6 How important are they compared to ITC?
3.2.7 What are the strengths/weaknesses/needs of ITC activities and deliverables?
3.3 Who are the main end users of ITC?
3.3.1 Define end users: What are their main characteristics (size of business, gender, sector,
in country coverage?)
X X X
3.3.2 What is their influence on need requests? and involvement and contribution to PCM? X X X
3.3.3 How does ITC ensure feedback from end-users? X X X
3.3.4 What indicators are used to measure the performance of ITC interventions at end-user
level?
X X X
3.3.5 How have the skills of end-users evolved over the last years and how has it influenced
the design of ITC product/services (technical skills, capacity to absorb knowledge,
networking, advocacy, benchmarking)?
X X X
3.3.6 How does the end user assess the performance of TSI? X X X
3.3.7 Is the end user aware of ITC? If yes what is the image? X
4 Geographical Focus of ITC
4.1 What are the most important target countries for ITC? X
4.2 What is the geographical coverage of deliverable in question? X X X
4.3 What is the access of ITC to the target groups X X
4.4 What is the outreach of each type of intervention to the direct beneficiaries and end
users?
X X
4.5 What is the experience of ITC in the different countries where it operates? X X
4.6 What are the factors which determine the geographical choice of countries? X X
Joint Evaluation of the International Trade Centre
Version Final
70
4.7 How are priorities made as to the geographical focus of ITC? X X
4.8 Is there continuity in ITC's presence in the countries of operations? how is this ensured? X X
4.9 How are ITC's country activities coordinated? X X
4.10 How are synergies created for ITC activities in a given country? X X
5 Impact of ITC interventions
5.1 How can ITC measure the baseline of a TSI to assess its ability to deliver and ensure
success of ITC intervention?
X X
5.2 What is the actual in-country state of each specific area of competency? X
5.3 What changes has ITC intervention induced on the end users? X
5.4 What changes has ITC intervention induced on the TSI? X
5.5 How has the number of exporters developed during the last 3-5 years? X
5.6 What are the factors which caused this change? What is the contribution of ITC in the
change?
X
5.7 How has the number of sectors in exports evolved over the past 3 years? X
5.8 What has been the export performance (clear evidence of growth, product
diversification, degree of value adding, market diversification) of the sectors of export
during the last 3 years?
X
5.9 What are the factors which influenced this change? What is the contribution of ITC in the
change?
X
5.10 What has been the increase in performance of exporters during the last 3 years
(increase in exports, product diversification, value adding, market diversification)?
X
5.11 What are the factors which influenced this change? What is the contribution of ITC in the
change?
X
5.12 How has the competitiveness of end-users developed over the last 3-5 years? X
5.13 Do end users belong to advocacy networks? X
5.14 How has advocacy network changed over the years? X
5.15 What are the factors which influenced this change? What is the contribution of ITC in the
change?
X
5.16 How does the end user/TSI perceive social impact of support services provided? How
are they measured?
X
Joint Evaluation of the International Trade Centre
Version Final
71
6 Performance of ITC deliverables
6.1 Which type of deliverables are most efficient and effective? In quality and quantity? X X
6.2 What are the key factors of success for each type of deliverable? X X
6.3 What are the strengths and weaknesses of each type of deliverable? X X
6.4 Does the channel of delivery affect the performance of ITC outputs? X X
7 Identification and assessment of other Providers
7.1 Who are they?
7.1.1 Which type of deliverables are most efficient and effective for the service provider? In
Quality and quantity? Who supplies them?
X X
7.1.2 Who are the agencies supplying services similar to ITC's? Their modes of delivery?
What are their key areas of competence?
X X X X
7.1.3 To what extent do private sector providers compete with ITC’ss activities? X X X
7.1.4 How do service providers identify needs and prioritize them? X X
7.1.5 How much overlap in activities/target groups is there in the countries? X X X X
7.1.6 How do other agencies measure the impact of their activities? X X X
7.1.7 What are their key areas of focus? X X X
7.1.8 New comers: Are there potential providers of services similar to ITC? In Which areas? X X X X
7.1.9 How do they differentiate themselves from similar providers? X X X X
7.1.10 What types of bottlenecks do similar agencies encounter in the process of PCM? X X
7.2 Degree of competition
7.2.1 In terms of "price"/quality range how are other service providers "competitive"? X X X X
7.2.2 What is the degree of overlap in interventions by service providers? X X X
7.2.3 What is the competitive advantage of ITC compared to other providers of similar
services?
X X X
7.2.4 What are the strengths and weaknesses of other service providers compared to ITC? X X X
7.2.5 What are the key factors of success for the different types of deliverables provided? X X
Joint Evaluation of the International Trade Centre
Version Final
72
C Managerial Issues of ITC (Managerial Challenges) Desk
research
Research
ITC
Field
Global
Field
Country
1 Strategic coherence between interventions, policies and objectives
1.1 How does ITC define its objectives ( mechanisms, WTO, UNCTAD?) X X
1.2 To what extent are ITC policies adapted to the achievement of the objectives? X X X
1.3 How clear, explicit and measurable are ITC objectives? X X
1.4 Is the ITC management performing efficiently in achieving the objectives and reaching
the goals?
X X
1.5 What it the Management strategic planning capacity? X X
1.6 What is the mgmt. degree of strategic awareness and understanding of environment? X X
2 Strategic Planning
2.1 What is the Capacity of ITC to coordinate and make use of potential synergies with
other actors at HQ and Field level?
X X X X
2.2 How does Mgmt. determine the evaluation and control parameters? X X
2.3 Are control and information systems used for performance management and strategy
development?
X X
2.4 Is the response to the external needs built in the organisation of ITC? X X X
2.5 What is the capacity of mgmt. to integrate lessons learnt responding to evolving and
managing growth?
X X
2.6 How is strategy formulated across divisions? Are there performance indicators for each
division?
X X
2.7 How are needs assessment carried out and priorities made in the approval of projects? X X
2.8 What are the key indicators that management should rely on in its decision making?
(Paretto's 80/20)
X X
2.9 How does ITC management ensure quality and efficiency across the organisation? X X
2.10 How are synergies identified and developed at the different levels of ITC? X X
Joint Evaluation of the International Trade Centre
Version Final
73
2.11 Has management identified areas where efficiency can be improved at ITC? in which
areas? What are the measures used to improve efficiency?
3 Promotion capacity
3.1 How does ITC promote itself to the target groups and countries? X X X
3.2 How does ITC promote its activities/competencies to Donors? X X X X
3.3 How is promotion carried out? At HQ level and at the country level? X X X
3.4 What is the experience of ITC with promoting interventions at country levels X X X
4 Networking and partnerships capacity
4.1 ITC Networking with other agencies: who are they and how important are they? X X X X
4.2 How is ITC's performance in cooperating with different partners at HQ level? X X X X
4.3 How is ITC's performance in cooperating with different partners at country level? X X X
4.4 What kind of networks has ITC established at country, regional and international level? X X X
4.5 What is the experience of ITC with promoting interventions at country levels X
4.6 How does Management ensure the anchoring of partnerships at institutional level? X
4.7 How does Management ensure the anchoring of networks and contacts at institutional
level?
X
5 Quality issues
5.1 Does ITC have a policy for Quality assurance? X X
5.2 How is quality ensured at ITC? Who is responsible for Quality issues? X X
5.3 How is the notion of quality ensured at the different levels of the organisation? X X
5.4 How is the quality of inputs and outputs ensured? X X
5.5 How are corrective actions taken in case situations where quality requirements are not
met?
X X
Joint Evaluation of the International Trade Centre
Version Final
74
D Functional issues Desk
research
Research
ITC
Field
Global
Field
Country
1 General questions related to ITC organisation
1.1 How has the org. structure developed since last review? X X
1.2 What are the main features of the present organisation structure? X X
1.3 How are the dept. Involved at each phase of the Project Cycle Management X X
1.4 Are there bottlenecks in the process of PCM? X X X X
2 Operations
2.1 How are the operations of ITC for providing deliverables organised? X X X
2.2 What are the main steps for ITC operations? From project identification to
implementation
X X
2.3 Which sections and department are involved in the process?
2.4 What is the degree of autonomy of the staff in creating and managing projects? X X
2.5 Does the deliverable fully take advantage of ITC's core competences X
2.6 Are there any areas where there is need to improve efficiency of work of ITC
department staff? Which areas?
X
2.7 What types of improvements could be used to improve the efficiency of operations? X
3 Product development
3.1 The ability of ITC to develop new services and products? X X X
3.2 The degree of market research and development for Product dev. Carried out? X X X
3.3 How ideas are generated across functions at ITC? X
3.4 How close is product development to core competences? X X X
3.5 How are new ideas tested and planned? X X
3.6 How are priorities made in view of increasing demand of ITC products/services? X X
Joint Evaluation of the International Trade Centre
Version Final
75
4 Human Resources Management
4.1 How is the HRD organised internally in ITC? X
4.2 How are ITC HQ competences and skill gaps identified and what measures are taken
to fill the gap?
X
4.3 What measures are undertaken by to build up and sustain capacity at ITC HQ? X
4.4 What is the main profile of ITC technical staff: specialists, generalists, multi disciplinary X X
4.5 How is the institutional memory ensured at ITC? X
4.6 What is the age profile of staff at ITC? X
4.7 What is the motivation level of staff at ITC? X
4.8 How does ITC ensure appropriate training of its staff? X
4.9 How is the distribution of "L" and "P" staff at ITC? What is the influence of the
distribution on the activities and operations of ITC?
X
4.10 How are local consultants identified and selected by ITC? X
4.11 Who ensures the quality and operational compliance of local consultants? X
4.12 How is the experience collected by ITC Consultants anchored at ITC? X
5 Financial Resources Management
5.1 Is the level of financing appropriate to the existing needs of ITC? X X
5.2 What is the predictability in funding and flow of funds from ITC pt of view (multi-annual
vs. annual commitments
X X
5.3 Does the flow of funds function properly (timing, individual regulations, core funds vs.
earmarked funds and other
X
5.4 Does the existing level of financing allow ITC to meet the increasing demand? X
5.5 To what extent does ITC funding modality influence the efficiency, quality and quantity
of ITC activities
X
5.6 To what extent does ITC funding modality influence managerial decisions and priorities
in the selection of countries and development of services?
X
5.7 Are the requests for funding timely seen from the donor's perspective? X X
5.8 What are the key Financial constraints which affect the performance of ITC? X X
Joint Evaluation of the International Trade Centre
Version Final
76
5.9 What is the price structure followed for products/services? How is it set? X X
5.10 Do the financial data and reporting system provide information for the assessment of
the financial outcome of individual projects?
X X
6 Monitoring and Evaluation
6.1 What are the management information tools available at ITC? X X
6.2 Is the information system adequate to the needs of the organisation? X X
6.3 Are the monitoring systems sufficiently developed to allow management to take
appropriate actions timely?
X X
6.4 How do internal systems for follow-up, monitoring, evaluation and feedback work? X X
6.5 How adequate are the monitoring systems to the activities of ITC at the different
levels?
X X
6.6 What are the shortcomings of the existing information system? X X
6.7 Are the ITC result indicators relevant and appropriate for the ITC "result based
Management"?
X X
6.8 How are the evaluations of ITC interventions decided? X X
6.9 What are the mechanisms followed for evaluation of ITC interventions? X X
6.10 How does ITC ensure that the results of the evaluations are being used by the
organisation?
X X
7 Procurement
7.1 How is ITC involved in procurement activities? Who is responsible for project
procurement?
X X
7.2 How important is project procurement in individual projects? X X
7.3 Is there an issue between procurement and time frame in PCM? X X
7.4 How does the individual staff carry out project procurement? X X
Joint Evaluation of the International Trade Centre
Version Final
77
E Organisational capability Desk
research
Research
ITC
Field
Global
Field
Country
1 Technical skills and competencies
1.1 What are the key technical competencies needed at ITC? X
1.2 What are the technical competencies available at ITC X
1.3 What are the competencies which are outsourced by ITC X
1.4 How is the match between the competencies needed and the ones available? X
1.5 How is the capacity building of ITC competencies ensured? X
1.6 How is the institutional memory of experience gathered by local consultants ensured? X
2 Project Cycle management - of a particular type of output
2.1 For a sample of output, what are the different stages of PCM? X X
2.2 Who are the parties involved at the different stages of PCM? X X
2.3 What is the typical time frame of Project cycle? X X
2.4 What is the type of information needed at each stage of PCM?
2.5 Is there consistency of projects and programmes with overarching policy objectives X X
2.6 Is PCM demand driven, expert driven, donor driven? X X
2.7 What is the capacity of staff to evaluate and follow-up, and ensure continuity and
feedback (contrasting capacity building as a long term goal, with ITC's short term
framework)?
X X
2.8 To what degree is there coordination with Partners in the use of PCM? X X
2.9 What are the bottlenecks? X X
2.10 Are the objectives for projects and programmes at ITC clear and realistic? X X
2.11 How are the quality factors identified to enhance project benefits and sustainability? X X
2.12 What is the influence of E-technology on the management cycle of project? X X
2.13 what is the quality of information available especially from beneficiaries and target
groups for PCM development?
X X
2.14 How does ITC ensure continuity after termination of a project? X X
Joint Evaluation of the International Trade Centre
Version Final
78
3 Transfer of knowledge and capacity building
3.1 What are the modes of transfer of knowledge and skills? X X X
3.2 How does capacity building tie in with the overall mission of ITC? X X
3.3 How does ITC ensure that transfer of skills is integrated in its programmes and
projects?
X X X
3.4 How is the transfer of skills organised? X X X
3.5 How are needs and demand reflected in the deliverables? X X X
3.6 Who at ITC is responsible for managing and arranging for the transfer of skills? X X
3.7 Who at ITC is responsible for developing the capacity building element in the
deliverables?
X X
3.8 Who at ITC is responsible for ensuring the quality of the transfer of knowledge? X
3.9 How does ITC select and ensure the relevance of the target group (recipients)? X X
3.10 How relevant in terms of capacity building are the deliverables offered by ITC ? X X X
3.11 How is the quality and quantity of capacity building elements in deliverables as
compared to other providers?
X X X
3.12 What kind of tools is ITC using to assess the capacity building? X X
3.13 How is ITC ensuring the quality of the staff involved in knowledge transfer? X
3.14 How is ITC monitoring the training (and other activities) of the intermediates? X
3.15 How is ITC ensuring that developed capacity of the TSIs is sustained? X X
4 E-competence
4.1 How has the e-technology affected the intervention of ITC and provision of services? X X X
4.2 How does e-technology influence the linkage to end-users and impact on the field? X X X
4.3 How efficient is e-technology as a method of training compared to traditional types of
interventions?
X X
4.4 What is the impact of e-competence technology as compared to other types of
interventions?
X X
4.5 How does ITC cope with e-reluctance? X X
4.6 How is e-competence built in the ITC organisation? X X
Joint Evaluation of the International Trade Centre
Version Final
79
F ITC’s comparative advantage Desk
research
Research
ITC
Field
Global
Field
Country
1 Core competencies of ITC
1.1 What are the core competencies of ITC? List them (15 competences of ITC) X X X
1.2 What is the involvement of ITC in technical areas related to development concerns? X X
1.3 How have these competencies evolved over the last years? X X
1.4 What is the relevance of core competencies to existing and future demand? X X X
2 Identification and assessment of other agencies of TA
2.1 Who are the agencies supplying similar services to ITC? In which areas? X X X X
2.2 How do the competencies of ITC compare to other international organizations, eg
UNCTAD and WTO
X X X X
2.2 To what extent does ITC compete with private sector providers? X X X X
2.3 What is the comparative advantage of ITC on other providers of similar services? X X X X
2.4 How is ITC differentiating its products from similar providers? X X X X
2.5 How much overlap is there in the target countries/groups X X X X
2.6 How do other agencies measure the impact of their activities? X X X X
2.7 Are there potential providers of services similar to ITC? X X X X
2.8 In terms of "price"/quality range how are ITC services "competitive"? X X X X
2.9 What types of bottlenecks do similar agencies encounter in the process of PCM? X X X
2.10 Where do other agencies focus their activities?
2.11 How do other agencies identify needs and prioritize them?
Joint Evaluation of the International Trade Centre
Version Final
80
3 Access to Demand
3.1 How does ITC assess needs from target groups? X
3.2 How does ITC respond to external needs? X X X
3.3 How are TSIs selected? X X
3.4 How does ITC ensure the match between needs and service provision? X X
3.5 How is demand monitored by ITC staff ? X X
3.6 How are opportunities identified by ITC? X X
3.7 How are new types of needs identified on a continuous basis? X X
4 Positioning ITC in the development community
4.1 Which type of activities and interventions are most efficient and effective? X X X X
4.2 What is the share of ITC in TRTA in the areas related to its competencies? X X X X
4.3 Where does ITC have a competitive advantage and comparative advantage? X X X X
4.4 Is ITC a "niche" player? X X
Joint Evaluation of the International Trade Centre
Version Final
82
6.2 Annex 2: INDICATORS MATRIX
TRTA and ITC
RB logic of
intervention
Indicators
(available in
Business Plan
2003-2005)
Suggested revision
( to be improved during field
study phase)
Quantitative
measurement
Qualitative
measurement
ITC target
( when
available)
2003 2005
Ultimate goal
(MDG 8)
Develop a
global
partnership for
sustainable
development
and poverty
alleviation
through an
open-trading
system that is
rule-based and
non
discriminatory
ITC’s statement
1. Awareness of relevance of
trade performance to overall
development
2. Improved employment and
region / poor outreach linked to
enterprise export expansion
3. Countries advocacy
network’s capacity and
effectiveness in building a rule-
based and non discriminatory
MTS
1. In country –
volume of Donor
support (in
financial terms)
per type of
support in TRTA,
per donor (of
which ITC)
2. Total number of
employees in
exporting
enterprise end-
user of ITC’s
services, of which
women and youth
(less then 25 yrs
1. Resource
situation / change
both in public and
in private sector
(chamber of
commerce/
industry
associations )
members of Trade
support network
(growth,
stagnation)
2. List of benefits
perceived by
exporting end-
users in regards to
Joint Evaluation of the International Trade Centre
Version Final
83
for this goal:
Develop a
global
partnership for
sustainable
development
and poverty
alleviation and
help the
developing
countries/transi
tion world
benefit
from a rule
based non
discriminatory
multilateral
trading system
(MTS)"
old); number of
enterprises
located in
decentralised
region of the
country
3. Year of
application to
WTO accession
and actual status
development
concerns and what
is their
measurement unit
(or approximate
quantification
measure they use)
to assess it. List of
type of costs and
incentives for
them.
3. List of topics
brought up by
developing
countries to MTS
(10 years period)
Joint Evaluation of the International Trade Centre
Version Final
84
TRTA and ITC
RB logic of
intervention
Indicators
(available in
Business Plan
2003-2005)
Suggested revision
( to be improved during
field study phase)
Quantitative
measurement
Qualitative measurement ITC target
( when
available)
2003 2005
TRTA corporate
goal
Enhance
enterprise
competitiveness
and national /
sector export
performance
ITC mission
statement:
To support
developing and
transition
economies, and
particularly
their business
sector, in their
efforts to realise
1. Trade performance at
a) country scale (source:
country map)
b) sector scale ( detailed
sector/sub-sector)
c) client scale (enterprise
end-user of TSI
networks) ( source: FIT
tool)
2. Share of ITC in TRTA
at country scale
1. a) Country
trade
performance
index over a 10
years
b) Number of
export sectors
2) Volume and
financial
resources of ITC
compared to
other donors in
TRTA, at
country scale.
1. TSI network’s
assessment of export
performance against 5
criteria:
- Value addition in export
markets (growth,
stagnation)
- Diversification of export
markets (yes / no)
- Diversification of
products (yes / no)
- Growth of export
(growth, stagnation,
reduction)
- Supply response capacity
to potential export growth
(available, not)
a) at country scale
Joint Evaluation of the International Trade Centre
Version Final
85
their full
potential for
developing
exports and
improving
import
operations
b) at sector scale
c) at client (enterprise/end-
user) scale
1. Ranking of trade
performance at country,
sector, client scale, from 1
to 4:
1- 0 or 1 criteria out of 5
are positive
2- 2 criteria out of 5
3- 3 criteria out of 5
4- 4 or 5 criteria our of 5
Joint Evaluation of the International Trade Centre
Version Final
86
TRTA and ITC
RB logic of
intervention
Indicators
(available in
Business Plan
2003-2005)
Suggested revision
( to be improved
during field study
phase)
Quantitative measurement Qualitati
ve
measure
ment
ITC target
( when available)
2003 2005
ITC corporate
goals ( 5)
1. To facilitate
the integration
of enterprises
into the MTS
Establishment of
national core
expertise in MTS
issues:
- good
understanding of
product and
market reality
and potential
- national support
network in
existence and
level of relevant
support
- Business sector
involvement in
the development
of country
positions on MTS
Indicators identified are
mainly related to outputs
directly under the
control of ITC. Impact
indicators related to
these goals are suggested
for Technical assistance
capacity building
Measurements will be
made during evaluation
and compared to past
and future situation as
perceived by the field and
compared to ITC’s target
set based on their
financial capacities.
1. Number of country networks
established or supported under
World Tr@de Net and related ITC
programs
2. Number of network meetings,
information bulletins, responses
and contributions to ITC web
sites
3. Number of institutional
committees established in African
countries participating in the
JITAP.
1. 50
2. 60
3. 8
1. 70
2. 120
3. 16
Joint Evaluation of the International Trade Centre
Version Final
87
issues
2. To support
national efforts
to design and
implement
trade
development
strategies
Trade
development
strategies that
take into account
supply capacity,
international
demand and
commercial
practices
Established and
implemented
with active
cooperation of
public and private
sectors in national
and sectoral
strategies
1. Number of trade development
strategies through ITC ( national
or sectoral) that are under
implementation
2. Number of countries assisted in
incorporating an e-dimension into
export strategies
3. Number of requests for pilot
project under the Export poverty
led program
1. 17
2. 4
3. 8
1. 57
2. 14
3. 16
Joint Evaluation of the International Trade Centre
Version Final
88
TRTA and ITC
RB logic of
intervention
Indicators (available in
Business Plan 2003-2005)
Suggested
revision
( to be
improved
during field
study phase)
Quantitative measurement Qualitati
ve
measure
ment
ITC target
( when available)
2003 2005
3. To strengthen
key trade
support
services, both
private and
public
Reinforcement of national
trade support institutions
which provide business
development and
competitiveness
enhancement services to
the business community
- scope of trade support
services provided by TSIs,
including gender-related
programmes
- expanded customization
of generic tools under the
ITC product-network
approach and network
activation
1. Number of TSI partnering with
ITC
2. Number of institutions in
number of countries applying ITC
support tools
3. Number of demands for trade
related information
1. 400
2. 20
institutio
ns in 20
countries
3.
820,000
visits in
157 links
1. 600
2. 60
institutions
in 40
countries
3. 900,000
visits and
175 links
4. To improve Improvement of trade 1. Number of enterprises 1. 1250 1. 2,750
Joint Evaluation of the International Trade Centre
Version Final
89
export
performance in
sectors of
critical
importance and
opportunity
performance in selected
product and services
sectors
- business opportunity
developed as a result of
supply-demand
matchmaking and other
support activities
- contribution of ITC
research and product or
market studies to sector
strategies, TSIs, SMEs
participating in ITC buyer – seller
meetings and matchmaking
activities
2. Number of countries with
partners using ITC strategic
market analysis tools
3. Number of trade in services
associations established
4. Number of institutions
applying ITC developed
management, export and e-audit
tools
2. 128
3. 2
4. 4
2. 140
3. 7
4. 34
TRTA and ITC
RB logic of
intervention
Indicators (available in
Business Plan 2003-2005)
Revision
during field
study phase)
Quantitative
measurement
Qualitative
measureme
nt
ITC target
( when available)
2003 2005
5. To foster
international
competitiveness
within the
business
community as a
Increase in
entrepreneurship skills for
exports and in
competitiveness at the
enterprise level
- level of demand among
1. Number of partners
institutions delivering
business management
training programs
2. Number of enterprises
1. 45
2. 100
enterpris
es in 70
countries
1. 90
2. 120 enterprises
Joint Evaluation of the International Trade Centre
Version Final
90
whole and the
small and
medium-sized
enterprise (
SME) sector in
particular
export managers for
training, skill development
and advisory services from
local institutions
partnering with ITC
- extent of specialized
competitiveness support
programmes available to
SMEs through local
organisations and
networks supported by
ITC.
associations in number of
countries applying ITC
developed support tools
3. Number of trainers
trained and regional hubs
of networks launched
3. 300
trainers
trained
and 2
regional
hubs or
networks
launched
associations in 90
countries
3. 1000 trainers
trained and 6
regional hubs or
networks
launched
Joint Evaluation of the International Trade Centre
Version Final
92
TRTA and
ITC
RB logic
of
interventi
on
Indicators
(available
in Business
Plan 2003-
2005)
Suggested
revision
( to be improved
during field study
phase)
Quantitative
measurement
Qualitative measurement ITC target
( when
available)
2003 2005
TA / CB
specific
objective:
Capacity
building
of
Intermedi
aries (TSI)
and end-
users
in one or
the other
of 15
areas of
technical
1. Multiplier effect
of intermediaries
on the number of
exporting
enterprises,
particularly SME
and women
2. Changes in trade
support services
profile at national
network scale
3. TSI perception
on changes in their
capacity in
respect to an ideal
1. Number of
sustainable
exporting
business
opportunities
developed
through TSI
support, with an
improved
performance of
the enterprise,
per type of
enterprise (big
or SME) and of
which owned by
women and per
sector of export,
2. List of network activities and stakeholders
ranking of network relevance and
effectiveness in carrying each of them (
current state)
1. Minimal achievement of the
objectives
2. Limited achievement of the
objectives, with limited
understanding of needs
3. Good progress towards
achievement of objectives, and
capacity to move on, with support
4. Thorough understanding of needs,
network demonstrate originality and
insight and produce work of good
quality.
Joint Evaluation of the International Trade Centre
Version Final
93
competenc
es in trade
and 5
areas of
technical
competenc
e linked to
developm
ent
concerns
profile identified
as follow:
- Technical skills
(one of the 15 +5
areas of
competence)
- Capacity to
transfer
knowledge
- Needs
assessments /
advocacy and
diagnostic
- Project
management,
internal
coordination and
follow-up
- Promotion to the
client and to the
donor
- Networking
per TSI
2. Number of
TSI members of
the network
(and
characteristics)
(current, last 5
years and
foreseen in the
next 5 years)
2. Type of
support services
provided and
number of users
per type of
services
2. TSI network ranking with respects to
criteria:
- multilevel trade network (geographic and
sector)
- regular meetings
- recognition and influence with policy
makers
focus covers all competitiveness issues (not
only volume of export)
- management of TSI: leadership,
communication and consultation, capacity to
mobilize additional resources or generate
revenues, M&E mechanisms, first stop – not
one stop.
- International leveraging at regional and
global scale
3. TSI capacities ranking (1 to 4), compared
to 5 years ago and foreseen in 5 years.
(sampling of individual entities depending
on the number of TSI in the country) 1. not
used 2. used with difficulty 3. good
capacity 4. leader in the country
Joint Evaluation of the International Trade Centre
Version Final
94
TRTA and
ITC
RB logic
of
interventi
on
Indicators
(available
in Business
Plan 2003-
2005)
Suggested
revision
( to be improved
during field study
phase)
Quantitativ
e
measureme
nt
Qualitative measurement ITC target
( when
available)
2003 2005
TA / CB
specific
objective:
(continue)
Capacity
building
of
Intermedi
aries (TSI)
and end-
users
4. Current state in
country of
technical
competence with
respect to future
profile foreseen
4. TSI ranking of current actual state in country ( 1 to
4,same criteria)
1. Export strategy: available in country, under
implementation and involving business sector in the
development of country position on MTS
2. MTS: capacity of the TSI to understand the
business implications of the trading system,
negotiate and draft contracts, resort to arbitration /
mediation services
3. Greater capability of enterprises to comply with
market requirements, seek contracts, negotiate and
exploit opportunities, assess its competitive position
and trade potential
4. Full potential of export
5.Market analysis and strategic market research
capacities within the network
6.Public – private partnership and networking
Joint Evaluation of the International Trade Centre
Version Final
95
growing
7.Trade in services
8. Trade information management
9. E-competence
10. Ensure and manage trade finance, access of SME
to trade finance
11. Quality products and meeting standards of
international markets
12. Supply chain management and import
13. Export packaging: label
14. Legal aspects
15. Institutional strengthening of TSI: management
capacities of TSI
16…
a) environmental sustainability
b) women in development
c) poverty eradication
d) economic and technical cooperation among
developing countries
e) digital divide
Joint Evaluation of the International Trade Centre
Version Final
96
TRTA and ITC
RB logic of
intervention
Indicators
(available
in
Business
Plan 2003-
2005)
Suggested
revision
( to be
improved
during field
study phase)
Quantitative
measurement
Qualitative measurement ITC target
( when
available)
2003 2005
Technical
assistance
outputs
6 types of
Deliverables:
• Tools
• Information
sources
• Advisory
• Training
• Network
• Programs,
projects,
initiatives
1. ITC’s
portfolio of
products
2. Uniqueness
of ITC
deliverables
compared to
its competitors
3. Adequacy
1. Percentage of
disbursement and
number of projects at
country level per area
of competence, channel
of delivery, donor, and
compared to requests
3. Number of requests
in ITC per products
5. Percentage of
disbursement of their
TRTA portfolio and
number of projects
5. Numbers of TSI
1. List of competences delivered in the
country / list available at ITC
1. Quality ranking (1 to 4) by the TSIs of
each delivered product in respect to a
previously established of quality criteria of
what would be a good product from the
perspective of the stakeholder. Product
quality ranking:
1. failure, achieve very few number
of criteria
2. achieved half to most of criteria
3. Achieved all criteria
4. achieved more than all stated
criteria
2. List of ITC delivered products and
Joint Evaluation of the International Trade Centre
Version Final
97
with TSI needs
4. Adequacy
with End-user
needs
reached ( and
characteristics);
number of end-users
reached (and
characteristics)
uniqueness factor identified by TSI,
compared to list of competitors most
important deliverables and uniqueness of
their product
3. List of TSIs’ demand (what they asked
for)
3. Open list of area of competence available
and TSIs demand ranking (1 to 4) per area
(criteria for demand ranking to be finalised with
stakeholders)
3. List of type of deliverables, and TSIs
demand ranking
3. List of problems faced in TSIs business
4. List of problems faced by exporting end-
users and by potential exporting end-users
5. List of area of competence / list available
in competitors
5. List of types of deliverables / list of
available in competitors
Joint Evaluation of the International Trade Centre
Version Final
98
TRTA and ITC
RB logic of
intervention
Indicators
(available
in Business
Plan 2003-
2005)
Suggested
revision
( to be
improved
during field
study phase)
Quantitative
measurement
Qualitative measurement ITC target
( when
available)
2003 2005
Target groups
- Trade support
institutions (TSI):
network and
individual entity,
also referred to as
intermediaries in
reaching the
enterprises end-users
of ITC services.
- end-users:
enterprises,
particularly Small
and Medium size
(SME)
See targets
indicated at
the level of
goals
1. Access of
ITC to
intermediaries
2. Visibility of
ITC and access
to deliverables
(see master
document)
3. Outreach
4. Cost and
sustainability(
willingness of
intermediaries
to pay for a
1. at country
scale: Number
of TSI contacts
in the country
and % per types
of TSI
1.at ITC
portfolio scale:
number of
countries
reached by
channel of
delivery and
main
deliverables in
each channel
1. List of TSI contacts of ITC with their
characteristics
• Private or public,
• Which sector,
• Which area of competence,
• Centralised, decentralised or both.
• Number of years in TRTA business
2. List of deliverables of ITC in country and
number of TSI met per level of knowledge
(awareness, know-how, practice)
2. Number of TSI met per means of access
used (internet, partner contact, ITC direct
contact, TSI network contact)
4. Willingness ranking of TSI to pay for ITC
Joint Evaluation of the International Trade Centre
Version Final
99
TSI can be:
Partners institutions
Enterprises
associations
Trainers
Enterprises
Individuals visiting
the e-sites
Countries
deliverable)
3. Number of
end-user direct
beneficiaries of
TSI using ITC
products
services ( ranking 1 to 4 to be determine
with stakeholder in the field)
4. Willingness ranking of end-users to pay
for support service
Joint Evaluation of the International Trade Centre
Version Final
100
TRTA and
ITC
RB logic of
intervention
Indicators
(available
in Business
Plan 2003-
2005)
Suggested revision
( to be improved during
field study phase)
Quantitative
measurement
Qualitative
measurement
ITC target
( when available)
2003 2005
Inputs:
Financial
resources
1. Funding modalities
2. Cash-flow
considerations
3. Beneficiary cost-
sharing and willingness
to pay
4. Pricing of ITC’S
products
5. Cost-effectiveness
1. % of budget per
source of funds
2. Time frame of
financial resources
4. % of saleable services
of total budget
5. % of disbursements
per deliverable, per
channel of delivery
(allocation of financial
resource) ,
5. overhead / operation
ratio
Human Strengths and
Joint Evaluation of the International Trade Centre
Version Final
101
resources weaknesses of ITC’s
human resources
Management
1. Project cycle
management of outputs
result-based oriented,
demand driven
2. Strategic coherence
between interventions,
policies and objectives
3. Promotion capacity
4. Networking and
partnership capacity
5. Quality, monitoring
and control capacity
1. Time frame of
delivery of each phase
of the project cycle
management (PCM) of
output
1. List of phases of
PCM of outputs for a
sample of main
deliverables
2. Service responsible at
each phase of the PCM
of output and human
resource involved
3. Information use at
each phase
4. Level of coordination
with partners
5. Level of coordination
within ITC
6. Strengths and
weaknesses of the
process in respect to
relevance, efficiency,
effectiveness and
sustainability
Joint Evaluation of the International Trade Centre
Version Final
103
Selection of Countries and Projects for Evaluation
The following section has elicited much discussion, but the original proposal has been
retained to describe the evolution of the approach to sampling.
The scope of this Joint Evaluation includes field- assessments of ITC’s activities
and results in selected countries. The presence of ITC interventions in nearly 170
countries, and the large range of ITC products and services that are used by ITC
clients, call for a process of country selection that is highly representative and
balanced in terms of the following criteria:
• Geographical distribution of ITC actions (representing the proportion of
delivery, as stated in ITC annual reports)
• Type of products and services supplied by ITC (selection covering the
flagship products of various ITC divisions, including global and country-
specific services)
• Economic development status of beneficiary countries (a balanced mix of
LDCs as well as OLICs, also large and small countries)
• Coverage of all ITC programmes, including the programmes supporting
the UN common areas of concern, and the multi agency programmes
such as JITAP and IF
• Beneficiary countries under different income profiles
• Trade performance (countries that have recorded consistent, high growth
in GDP and in trade in absolute terms, as well as countries that have not)
The rather limited budgetary and time resources for this Evaluation place
constraints on the number of countries that can be taken up for comprehensive
field evaluations. At the same time, it is expected that the countries chosen for
field evaluations cover all the above groupings, to provide deeper insights into
the results and impact of the interventions, and the challenges that ITC might
face in effective delivery of its services, on account of external factors.
The country level evaluations are expected to provide useful field information to
support answers to the following questions:
a. How far has it been possible to achieve concrete results- outputs and
outcomes-within the area of trade promotion?
• What has been the outcome, efficiency and relevance of interventions
at national, regional and international level targeted towards
i. Trade policies and strategies
ii. National infrastructure for trade promotion, import
management
iii. Support services for trade promotion
Joint Evaluation of the International Trade Centre
Version Final
104
iv. Training institutions
v. Sector and enterprise level
b. Where have bottlenecks and key risks occurred?
i. In the international, regional and national enabling environment or in
government policies, strategies and plans as well as national rules and
regulations and national institutional capacity?
ii. How have they affected the output and outcome of ITC activities?
c. Which type of activities and interventions are most efficient and effective-
i. Horizontal/multilateral efforts or targeted interventions in individual
countries?
ii. Quality versus quantity
iii. Does success of an intervention depend on the region and/or the level
of development?
iv. Is the time frame for different interventions appropriate?
To capture insights and to answer some of these aspects, it is also desirable to
compare experiences and results in countries that have received the same or
similar interventions from ITC.
6.2.1 Project Universe for Evaluation
The project portfolio is discussed in more detail in the Desk Study.
The ITC Projects database (intranet) documents only those projects that were
operational, closed operationally, or closed financially during 2001-2004. In this
database, there are, in all, 539 projects in the period of 2001-04. Year-wise
information is available for years 2003, 2004 and the aggregate for 2001-2004.The
database provides details for these years, classified by beneficiary country,
responsible division/section, donors, and other useful categories. Because
information is not available for projects before 2001, only projects appearing in
the data for 2001-2004 shall be sampled for the purpose of project selection.
Status of Projects
2001-04 2004 2003 2002 2001
All projects 539 231 265 na na
Operational 300 182 167 na na
Closed operationally 131 36 64 na na
Closed financially 37 13 34 na na
Joint Evaluation of the International Trade Centre
Version Final
105
There is a conflict between the selection of projects based on the objectives of
measuring impact (impact manifests only in the longer-term, after an
intervention has been completed), and the importance of feedback and
interaction with project management units and other stakeholders, which will
have moved on soon after the intervention. It becomes difficult to track and
obtain feedback from stakeholder bodies that participated in an intervention
more than three or four years ago. Therefore, to obtain meaningful results from
the evaluation, equal weight would be given to completed and operational
projects, selected from the 2001-04 database.
Average number of projects per country:
On an average, each country has 3 projects from the project universe for 2001-
2004. This would imply coverage of approximately 27 countries, if projects are
selected on a random basis. But, there is a wide dispersion in intensity across
various countries, ranging from 1 project each in 19 countries to 52 projects in
Tanzania alone.
Of the 170 countries in the project universe, 42 countries have 3 or fewer projects,
aggregating to 74 projects, averaging 1.76 projects each. On the other hand, more
than ten countries in Africa have more than 15 projects each going on since 2001.
Sample size:
To adequately cover all types of services and all profiles of countries, we propose
to select around 15 countries and 80 projects from the database, representing
almost 15% of the relevant project universe.
Choice of Sampling Technique:
Given that the Joint Evaluation’s task is to bring an analysis of the effectiveness
and impact of ITC activities. Given the wide variance in ground conditions in the
countries in which ITC operates, and given the technological difficulties in
building indicators for measuring impact of a specific organization/ intervention
in a development environment having several actors and different types of
interventions, the analysis shall depend more on qualitative and experiential
indicators than quantitative interpretations.
Classical Random Sampling by countries:
The random sampling approach by countries–where each member of the
universe has an equal probability of being selected-is not desirable for the
selection of countries for this evaluation. This selection ignores the fundamental
premise that ITC has a prioritization of countries/ regions for its interventions,
and this prioritization needs to be reflected in the samples. Also, because the
Joint Evaluation of the International Trade Centre
Version Final
106
evaluation proposes to gauge the impact of ITC on trade development, there is a
stronger case for taking up countries where there has been considerable ITC
activity represented by a large number of projects, preferably covering several
services and programmes of ITC. A random sample can reduce the proportion of
such countries, given the wide dispersion in number of projects among the
country universe in the database.
Sampling by projects:
On an average, each country has 3 projects from the project universe for 2001-
2004. This would imply coverage of approximately 27 countries, if random
selection were done by projects. Also, there is almost a 25% chance of getting
countries having three of fewer projects. The field studies of countries with such
low volume of ITC involvement (including different types of isolated activities)
would not be useful in determining the impact of ITC, and there is a high chance
that conclusions from such countries would tend to understate the achievements
of ITC. As a result, the inclusion of countries with marginal ITC activity is not
likely to be cost-effective.
Evaluating impacts in countries with greater intensity of ITC activities is likely to
give useful insights into the types of interventions and combinations of
interventions that are likely to demonstrate better results in developing
countries. For these reasons, it is considered important to include in the sample, a
sufficient number of countries where there has been a meaningful and sustained
presence of ITC.
Sampling by service divisions/activities:
ITC delivers its services through specialised divisions, and each project is allotted
a Responsible Section. The allocation of projects by division and sub division,
and a section wise sample, based on a 15% sampling rate from the 539 projects is
as follows:
Division/Department Staff
strength
Number of
projects
Share % Sample size
based on 15%
selection
OED 18 44 8.1% 6 projects
DTCC/OD ? 6 1
DTCC/OD/LDCU 4 4 1
DTCC/OD/SSTU 11 12 2
DTCC/OA 20 170 26
DTCC/OASEC 8 41 6
Joint Evaluation of the International Trade Centre
Version Final
107
DTCC/OAPLAC 11 32 4
DTCC subtotal 54 265 49.2% 40 projects
DPMD ? 1
DPMD/TSU 6 14 2
DPMD/TIS 17 18 3
DPMD/MAS 35 18 3
DPMD/MDS 17 52 7
DPMD sub total 75 103 19.1% 15 projects
DTSS/OD ? 10 1
DTSS/FASS ? 8 1
DTSS/EMDS 15 20 3
DTSS/IPMS 15 47 7
DTSS/BAS 30 42 7
DTSS subtotal 60? 127 23.5% 19 projects
Total 207? 539 100% 80
A sampling based on the service divisions addresses the geographical dispersion
of activities, but differently than the value distribution of ITC services. According
to the above sampling, there should be 26 projects in Africa, which is a
proportionally higher weight than Africa’s share in terms of value of ITC
projects, which is around 40%; and 10 countries together in the Arab, East
Europe, CIS and Asia Pacific, which is proportionally lower than their value
share (45%).
There is another dimension to the sampling of projects based on responsible
sections, given that two of the divisions- DTSS and DPMD- are technical
divisions, which contribute to service delivery in projects that are undertaken in
the leadership of DTCC. A sizeable share of resources and activities in these
divisions goes into developing and delivering a common pool of products, which
are used to different degrees in various country programmes.
Stratified or Quota Sampling:
Given that the classical random sampling approach will give unrepresentative
samples for this evaluation, we wish to base the selection on a stratified sampling
or quota sampling method, to include at least some countries from each class
appearing under the following criteria:
• Regional: representing the five delivery regions (Africa, AP, LAC, Arab,
EECIS)
• Economic classification: LDC, OLIC, and LMICs as per the OECD
classification for 2001 and 2002 flows (LDC- group of 49 countries
Joint Evaluation of the International Trade Centre
Version Final
108
established by the UN; OLIC - non LDCs with GNP below US$ 760 per
capita in 1998, and LMICs between US $760 and $3030 in 1998)
• Type of economy: market economies and countries in transition
• Population size: small (<10 million), medium/ large (> 10 million)
• Economic performance: countries that posted annual GDP growth and
annual export growth both more than 4.5% during past decade (1990-
2000, appearing in World Bank’s World Development Indicators 2003),
and countries that did not
Because ITC reports its delivery by geographical region, and because each region
has a mix of LDCs OLICs, a quota sampling by regions would give a mix of LDC
and non-LDC countries in each region.
The other classifications for selecting countries are intended to establish control
groups to measure impacts in countries based on different absorption capacities
(size of economy, % of poor population) and based on actual improvements in
trade performance in the past few years.
Another important aspect to be studied is the difference between results and
impacts created by multiple agency/regional interventions, and by single-country
interventions.
Country sample by share of delivery
Region Share in 2001
delivery %
Share in 2002
delivery %
Average share
in delivery %
Number of
samples
Africa 43 43 43 6
Asia Pacific 13 15 14 3
LAC 13 13 13 2
CEE and CIS 19 14 16.5 3
Arab States 5 7 6 1
Total 93* 92 92.5 15
* Global services account for 7% and 8% in these years
Source: ITC annual report 2002
Within each region, the countries shall be chosen on a random basis (picking
chits bearing the country name from a box). Countries that have recent history of
civil disturbances or political turmoil shall be excluded and in case such
countries feature in the sample, there shall be redraws.
At the end of the sampling, countries shall be grouped on the following table to
study the presence of all categories of interest. In case any categories are missing,
Joint Evaluation of the International Trade Centre
Version Final
109
a replacement draw shall be made by deselecting one of the over represented
groups, and selecting until there is representation in all categories of interest. On
12th March, the evaluation team picked a random sample of countries based on
the above method (room 123, ITC office), and came up with the following list:
Results of first stratified random sample
Region Type of economy Size Trade performance
LDC OLIC Transiti
on
Small
M/L Performer Non
performer
Africa
Cape Verde 8 * * *
Mali 20 * *
Ethiopia 19 * *
Namibia 15 * * *
Seychelles 6 * * *
Mozambique 19 *
Asia Pacific
Malaysia 12 * *
Mongolia 6 * *(gdp
only)
Vanuatu 5 * * - -
LAC
Peru 10 * * *
Trinidad &
Tobago 8
* * *
CEE-CIS 3
Turkmenistan 6 * - -
Czech Rep 2 * * - -
Macedonia 1 * * -
Arab
Tunisia 20 * * *
Total at least 2 in
each box
The sample consists of several small island countries, and six of the 15 countries
have less than 5 ITC projects during 2001-2004.
Total database entries: 147
Transition countries: 2
Very small countries: 6
Countries with incomplete data: 4
Joint Evaluation of the International Trade Centre
Version Final
110
Average number of projects: 9, five countries with less than 7 project activity
entries in ITC database for 2001-04.
Therefore, the above sample cannot be considered cost-effective for purposes of
this evaluation. In order to have a more meaningful sample, the stratified sample
for different regions of the world allocated according to ITC budget allocations in
2001 and 2002. The judgemental samples was developed on the basis of meeting
the following criteria:
• Comprehensive coverage of ITC programs and competencies;
• Inclusion of economies at different levels of development; LDCS, OLICs,
and LMICs;
• Coverage of large and small economies; and
• ITC projects that are completed and those that are under implementation.
The following criteria were used to exclude some countries:
• Countries with serious civil disturbances; and
• Countries that had been subject to recent ITC country evaluations.
The result yielded the following list:
Revised list after reiterations:
Country, no of
database records
Income profile Population size Economic Performance
Description LDC OLIC Small
<12m
Med/lar
ge >12m
Performer Non
performer
Africa
Benin, 39 * * *
Burkina Faso, 36 * * *
Kenya, 45 * * *
South Africa, 21 * * *
Tanzania, 52 * * *
Zambia, 22 * * *
Asia Pacific
Viet Nam, 18 * * *
Cambodia, 24 * * *
India, 21 * * *
LAC
Jamaica, 12 * * *
Honduras, 12 * * *
CEE-CIS
Joint Evaluation of the International Trade Centre
Version Final
111
Moldova, 9 * * -
Kyrgyz, 19 * * -
Latvia, 7 * * *
Arab
Tunisia, 20 * * *
Total, 359 5 10 8 7 7 6
The relevant economic data used for the classification appears as an Annex to the
Desk Study.
Sample country groupings by selection profile
Region Income profile Population Economic performance
LDC OLIC Small Medium
/large
Performer Non performer
Africa 6 4 2 3 3 4 2
Asia Pacific 3 1 2 1 2 1 2
LAC 2 0 2 2 0 1 1
CEE-CIS 3 0 3 3 0 1 -
Arab 1 0 1 0 1 0 1
Total 5 10 8 7 7 6
Country fit with major ITC programmes:
JITAP: Benin, Burkina Faso, Kenya, Tanzania,
IF: Cambodia, Tanzania
Sector coverage:
Commodities – coffee, spices, Floriculture – cut flowers, Textiles and Clothing,
Leather
However, there does not seem to be adequate representation of LDC exporters of
textiles and clothing. Textiles and clothing will be addressed in the horizontal
cluster analysis of ITC products and services.
In each of the selected countries, the total number of ITC projects shall be
inventoried and project documents shall be perused and grouped into integrated
projects (IF/JITA) and country specific projects. Within country projects, the
classification of projects shall be made according to the standard ITC
Programmes (7 +3) under which these projects are serviced by ITC.
The five largest projects by value spent in the country shall be taken up for
detailed field evaluations, to demonstrate the possibility of the highest impact in
Joint Evaluation of the International Trade Centre
Version Final
112
each country. Among these, projects that have recently been completed or those
nearing completion shall be given more priority for study, based on the
assumption that it would be easier to locate and meet with the project managers,
principal beneficiaries and other stakeholders in these projects.
Project documents, contact details of key persons at ITC and in the beneficiary
country dealing with the project, and monitoring and evaluation documents if
any, would need to be collected at studied prior to the field mission.
Horizontal Studies:
The countries selected have adequate coverage of all seven ITC Programmes
besides the programmes designed to address the UN common areas of concern.
However, besides the country studies, the evaluation also propose to undertake
field studies at a horizontal level for the major ITC programmes. For some
programmes, the selected countries do not provide a sufficient base to establish
meaningful studies. Therefore, studies on specific programmes shall be carried
out in a few representative countries that are not in the sample.
The following countries have been selected based on the desk study and
successful examples cited by ITC:
• Export led Poverty Reduction Programme: China, Floriculture Project,
Yunan Province; and India, Organic Spices Programme
• World Trade Net: Brazil (Business for Development and UNCTAD XI,
June 2004)
• South- South Trade Programme: South Africa, Kenya
• Export Packaging, Latvia
Joint Evaluation of the International Trade Centre
Version Final
113
Economic data for sample countries to classify size, income and trade
performance
Country Population
2001
thousands
GDP
millions
of
dollars
GDP per
capita
dollars
Annual
growth
total real
product
1990-2001
Export
value
growth
%
1990-2000
Benin 6387 2372 371 4,8 3,4
Burkina Faso 12259 2486 203 4,5 11,9
Kenya 31065 11396 367 2,0 6,3
South Africa 44416 113276 2550* 2,1 2,5
Tanzania 35565 9341 263 3,2 7,4
Zambia 10570 3639 344 0,8 -2,0
Viet Nam 79197 32723 413 7,7 22,7
Cambodia 13478 3404 253 5,0 25,3
India 1033395 477342 462 5,9 9,5
Jamaica 2603 7784 2990 0,2 2,2
Honduras 6619 6386 965 3,1 7,1
Moldova 4276 1479 346 -8,4 -
Kyrgyz 4995 1525 305 -2,9 -
Latvia 2351 7549 3211 -2,2 7,4
Tunisia 9624 19990 2077 4,7 6,0
World avge 6148061 31079820 5058 6,7
Developing countries
and territories
4944824 6526752 1324 4,7 9,1
Countries in Central
and Eastern Europe
336558 806241 2396 -3,9 8,2
Source: UNCTAD Handbook of Statistics 2003
6.3 Note on Country Selection
This section presents the final country selection based on the discussion with,
and comments of, the members of the core group and the management group. In
the Draft Final Inception Report we identified the following criteria for country
selection for country level studies. Thus it modifies the previous section.
First the overall sample was stratified by the relative budget weights of ITC
projects in different regions of the world. Thus, the share of the region in the ITC
budget was used as a guideline for the allocation of country studies among
different regions of the world. Note that this regional distribution ensures
inclusion of economies in transition.
Joint Evaluation of the International Trade Centre
Version Final
114
Distribution of Activities 2001 2002
Africa - 43% Africa - 43%
Asia/Pacific - 13%
Asia/Pacific - 15%
Latin America and the Caribbean -
13%
Latin America and the Caribbean -
13%
CEE and CIS - 19%
CEE and CIS - 14%
Arab States - 5% Arab States - 7%
Global - 7% Global - 8%
Then within this stratification, a judgmental sample was developed on the basis
of meeting the following criteria:
• Comprehensive coverage of ITC programs and competencies;
• Inclusion of economies at different levels of development; LDCS, OLICs,
and LMICs;
• Coverage of large and small economies; and
• ITC projects that are completed and those that are under implementation.
The following criteria were used to exclude some countries:
• Countries with serious civil disturbances; and
• Countries that had been subject to recent ITC country evaluations
On the basis of these criteria we proposed a sample of 10 countries consisting of
the following countries:
Joint Evaluation of the International Trade Centre
Version Final
115
Original Sample
Country, no of
database records
Income profile Population size Economic Performance
Description LDC OLIC Small
<12m
Med/lar
ge >12m
Performer Non
performer
Africa
Burkina Faso, 36 * * *
Kenya, 45 * * *
South Africa, 21 * * *
Tanzania, 52 * * *
Asia Pacific
Viet Nam, 18 * * *
India, 21 * * *
LAC
St Lucia * * *
El Salvador, 12 * * *
CEE-CIS
Moldova, 9 * * -
Kyrgyz, 19 * * -
Joint Evaluation of the International Trade Centre
Version Final
116
At the meeting of the Management Group and the Core Group in April there was
extensive discussion of the country sample. There was general agreement that
the sample should be reduced. One suggestion was to reduce the number of
country studies to about four as in other joint evaluations.
In our view, -- and it seemed to be the view of some members of the
Management and Core Group --, a larger number of country studies than four is
warranted because of the different types of countries where the ITC is working.
The different types of economies include LDCs, OLICs, LMICs and economies in
transition. The sample also includes landlocked countries and small island
economies.
Perhaps the most important factor suggesting a larger sample is that because the
ITC interventions are dispersed over many economies, the ITC interventions
tend to be small even in countries where they are quite active. While many ITC
competences are delivered in a number of the economies where ITC is active,
some are delivered in fewer economies. The sample of 10 countries provides
broad coverage of ITC competences. Thus, within the stratified sample structure
according to different regions of the world, we have reduced the total number of
countries to ten from fifteen, but we have not reduced it further.
In so doing we have reduced the number of LMICs from 3 to 2, but we still
propose to be evaluating some horizontal project activity in LMICs. The key
criteria for selecting the following sample of 11 countries was the coverage of ITC
competences, delivery tracks and types of deliverables. Please note that the
specification of key competences, delivery tracks and deliverables only captures
the most import project activities. The purpose of this survey of key
competences, key delivery tracks and deliverables is to ensure that the sample is
comprehensively covering ITC interventions. It is evident that the sample is
heavily weighted to countries where the ITC is especially active.
The proposed sample is presented and the coverage of ITC competences for these
countries is summarised in the following table.
Joint Evaluation of the International Trade Centre
Version Final
117
Country Selection Survey Of ITC Activities
Country Income profile Population size
Main areas of
competence
Main types of
deliverable
Name of
ITC outputs Country portfolio
LDC OLIC LMIC
Small <
12m
Med/large >
12 m
Country
portfolio
Country
portfolio
sample of
projects
number
of entry*
number
sampled
AFRICA
1 Benin 1 1 2,4,5,10,11,1 1,2 39
2 Burkina Faso 1 1 2,3,4,5,8,1 36
3 Kenya 1 1 1,2,4,5,7 1,6,3,4
E-trade
bridge,
SSTP,
EPRP,
JITAP,
PAckIT,
Executive
forum 28/45 17
4 Tanzania 1 1 1,7,2,3,4,5,15 6,2,4
IF, 11, 1, E-
trade
bridge,
SSTP, 4, 8,
Jitap,15, 34/52 17
ARAB STATES
5 Tunisia 1 1 2,1,3,4,14 6,4 18/21
ASIA PACIFIC
6 India 1 1 1,9,3,15 1,4,5,6
E-trade
bridge 19/21 9
7 Vietnam 1 1 1,9,2,3 6,2,3 18/16
Joint Evaluation of the International Trade Centre
Version Final
118
Country Income profile Population size
Main areas of
competence
Main types of
deliverable
LDC OLIC LMIC
Small <
12m
Med/large >
12 m
Country
portfolio
Country
portfolio
sample of
projects
number
of entry*
number
sampled
8 Kyrgyz 1 1 1,3,15,7 6,2 19
9 Moldova 1 1 1,7,12,15 4,6 9
LAC
10 El Savador 1 1,2,3,4 1,4,5,6
WTN, E-
trade
Bridge,
comp tool
fair, EPRP 12
11 Saint Lucia 1 1,7,15 5, 6
Executive
strategy 8
TOTAL 3 7 1
Joint Evaluation of the International Trade Centre
Version Final
119
Horizontal &
functional studies
Brazil, Multilateral Trade system and business advocacy 1
Executive
forum, SME
comp -
PROCIP,
PACKit 15
China, Integration of
poverty reduction
TRTA 1 EPRP,
South Africa,
Integration of
development
concerns in TRTA 1 1 C,D,1,3,7,5
E-trade
bridge,
SSTP,
EPRP 21 5 or 9
Turkey or
Philippines,
Capacity building
SME
competitiveness
Latvia, export
packaging
When the cell includes a list: it is in decreasing order of importance ( either frequence, or total amount)
Joint Evaluation of the International Trade Centre
Version Final
120
List of technical competences, delivery tracks and deliverables from the TRTA reference framework
on page 29 of the Draft Final Inception Report
List of areas of
technical
competences
In decreasing order of importance, with at least 3
entries related
List of
tracks
1 Export strategy
2 MTS 1 - Global, generic
3 Enterprise management and competitiveness 2 - Multi progr., multi donors
4
Product development by
sector 3- Country / region specific
5 Market analysis and strategic market research
6 Public - private partnership and networking
7 Trade in services
Types of Deliverables
8
Trade information
management 1. Tools
9 E-competence 2
Information
sources
10 Trade finance 3 Advisory
11 Standards and quality 4 Training
12 Supply chain management 5 Networking
13 Export packaging 6 Programmes
14 Legal aspects
15 Institutional strengthening of TSI
Joint Evaluation of the International Trade Centre
Version Final
122
6.4 Annex 4: QA Plan
6.4.1 Introduction to the Quality Assurance Management
System
An active Quality Assurance (QA) is essential to ensure the smooth
implementation of the project, the achievement of project objectives, and an
effective liaison with the stakeholders and Danida. In order to secure this,
administrative procedures have been set up to ensure:
6) communication and time for discussion and feed-back between evaluation
team members
7) fast accessibility to information and documents collected and developed
during the evaluation
8) dialogue with Danida, the Management Group, Core Group, and other
stakeholders
The QA procedures have been developed on the basis of Copenhagen
Development Consulting’s QA-system; quality requirements of the tender
document; the consortium’s organisation; and our approach to the evaluation.
The approach to the QM system is that it should be as simple as possible and as
detailed and complex as necessary. If it is too detailed and complex the system
will not be followed; if it is too simple it will not provide the necessary assurance
and documentation of quality.
The procedures will be presented to all evaluation team members (in writing
and orally) at the earliest possible. The first internal audit will be carried out as
soon as possible after the inception period to ensure that the QM system is
understood and being followed.
6.4.2 Administrative procedures
Communication lines and feed-back
Discussions and feed-back between the evaluation team members is found of
crucial importance to the quality of the evaluation. To ensure this, meetings will
be held in the beginning of the evaluation and at important steps in the process.
Fast access to documents and files will be secured through the use of a web-
server with access for evaluation team-members. In addition a proposal for a
public web site which will provide access to key project documents to Danida,
Joint Evaluation of the International Trade Centre
Version Final
123
the Management Group, the Core Group and the Consultative Group, as well as
the general public is included in the Inception Report.
All relevant information from Danida, the Management Group, and the Core
Group will be communicated to the evaluation team; and all information
gathered by the evaluation team will be communicated to TL and relevant
evaluation team members.
Handling of Documents (paper version and electronic)
A filing system has been developed. This filing system will be used both for the
filing of printed documents, and for electronic filing (web-server or filing system
at project head office). Please refer to Annex 5.
� A paper version of all documents/reports in their final version will be printed
and filed at the project head office.
� A copy of all outgoing documents will be filed in a printed version.
� All electronic versions of documents will be filed on the web-siteand/or at the
project head office.
� Information in hard copy only received by the project head office will be filed
and a list of relevant documents sent to relevant evaluation team members. A
list of filed hard copy documents of importance will be kept on the web-
server.
� Information in hard copy only received by individual team members will be
filed by the individual members. If found relevant a copy will be provided to
TL and/or other relevant team members. A list of filed hard copy documents
of importance will be sent by the expert and will be kept on the web-server.
Guidelines for use of the htp-server
What to save where and when:
� The folder system/structure is the same as the filing system (repetition
of numbers and text)
� The filing system is self-explanatory
� QA-folders will be established for relevant reports and other QA-
activities.
� It is suggested to establish folders with documents and/or lists of
information and data collected
When to save and how to get information about new "files"/papers
� all relevant data must be saved immediately
� when important files have been added to the web site all relevant
“members” will be notified in an e-mail (attachments are not needed).
Back-up of content on the server
Joint Evaluation of the International Trade Centre
Version Final
124
� daily, weekly, and monthly back-up carried out by Liga Stikute
� tapes/CD’s are kept in a different location from the server
� If the web-server is down (not functioning) all documents of
communication must be sent to TL who will save on the project head
office computer for later saving at the web-server
� A back-up email system and storage is available on the email address,
Definition of responsibilities among team-members
There is a management process established in the Inception Report. Each
member of the evaluation team will receive from the Team Leader terms of
reference and a mission allocation for each phase of the study, namely the Desk
Study phase, February 15-March 31, 2004, the Field Study Phase, April 1- June
30, 2004 and the Synthesis Report (July 1, 2004-September 30, 2004).
Financial Procedures
Accounting
Financial procedures for reimbursable expenses will be followed according to
Danida guidelines and procedures. Each of the experts and/or consortium
members will submit an original of all financial documentation to the office of
DMI Associates, 21 rue Longue, BP 1176, F -69201, Lyon Cedex 1, France by
registered mail. Photocopies of all documentation submitted must be retained
prior to sending them. Such documentation should be submitted monthly and all
documentation must be submitted quarterly.
Time sheets for professional time should be signed by the expert and submitted
by registered mail to the Team Leader, Murray Smith, 147 rue de Lausanne, CH-
1202, Geneva, Switzerland for signature and approval on a monthly basis. The
Team Leader will submit all time sheets received within seven working days
from the beginning of the month / within nine working days to the office of DMI
Associates 21 rue Longue, BP 1176, F-69201, Lyon, Cedex 1, France.
Guidelines for accounting for duty travels
All duty travel must be approved by the Team Leader. Normally authorisation
will be included in the terms of reference for the expert for each phase of the
study, but all travel must be authorised by the Team Leader.
Joint Evaluation of the International Trade Centre
Version Final
125
General Financial Reporting
While each expert and consortium member is responsible for maintaining
complete financial records, all original documentation should be submitted to
the office of DMI Associates, 21 rue Longue, BP 1176, F-69201, Lyon, Cedex 1,
France. All accounts must be submitted within five days of the end of each
quarter.
Guidelines for Hiring Local Consultants
CVs for candidates for local consultants will be submitted to the Team Leader for
approval. Interviews will be held if feasible, and a report should be submitted to
the Team Leader if an interview is held.
QA-organisation
A QA team has been formed with the following thee members:
Mrs. Denise Colonna d’Istria will be responsible for technical QA of
1) the evaluation process and results
2) review of technical reports
Mrs. Annette Munk Sørensen will be responsible for:
1) development and maintenance of the QA-plan
2) monitoring timeliness and adherence to the QA-plan
Daily QA will be the responsibility of the Team-leader, Core-Team members, and
Team of Specialists.
According to the Danida Evaluation Guidelines (February 1999) Chapter 6,
the team leader will have to indicate the extent to which:
• Danida guidelines, including the present one, have been used
• the use of methods have been discussed explicitly
• the focus, analysis and results are according to standard
• the language, layout, illustrations, etc are according to
standard
• the professional input of team members is acceptable
The Team Leader will advise all experts of the requirements of Chapter 6 of the
Danida evaluation guidelines and they will be asked to sign a letter confirming
that they are in compliance with the requirements of Chapter 6 of the Danida
evaluation guidelines.
Joint Evaluation of the International Trade Centre
Version Final
126
All mission reports, country reports and other reports must be submitted to the
Team Leader for approval.
Documentation of QA:
A QA folder will be created for each report as a sub-folder on the web-server
with back-up with the server of the consortium leader DMI Associates
The QA-folder will contain:
1) QA form for reports,
2) related e-mails saved as rtf-files (Rich-Text-Format),
3) related versions of reports with “track-changes”.
Quality Assurance Plan
Generally speaking, QA is provided through continuous review of reports,
technical papers, meetings, e-mail discussions, etc.; such reviews are compared
with the work plan and administrative procedures. In addition, QA also covers
financial aspects, scrutiny of vouchers, and auditing.
As well as measuring the evaluation progress against the more obvious time and
quantity standards, the evaluation will also be measured against quality
standards given in Danida guidelines for evaluation.
Quality Assurance of reports
A traditional QA will be made for all major reports. As most work will be
distributed and quality checked over the internet, no signature will be found on
the QA forms. Instead QA will be documented by filling in the QA-form
electronically and saving related e-mails and documents with “track changes” in
the QA-folder for the relevant reports.
Joint Evaluation of the International Trade Centre
Version Final
127
In addition, quality will be assured through review and feed-back during the
process of writing the reports. This feed-back will be given by the Team-Leader,
core team members, the team of specialists, and local consultants as found
relevant by the Team-Leader.
The QA-form for reports has been developed based on the requirements given in
Danida’s “Evaluation Guidelines”.
Layout for reports
A format for layout of report has been developed and this must be used by all
evaluation team members. Annex 3 (ITC-report.dot).
Joint Evaluation of the International Trade Centre
Version Final
129
6.4.3 QA Annexes
Annex 1
QA-Plan (QM 1.v2)
Annex 2
QA form for reports (QM 2.1v2)
Annex 3
ITC-Report.dot
Annex 4
Filing system
Annex 5
Htp-server structure
Joint Evaluation of the International Trade Centre
Version Final
131
6.4.4 QA Annex 1
Quality Assurance Plan
Activity Responsible Participating QA-
responsible
Date Documentation of QA
Daily QA TL + Core Team On-going e-mail, documents, etc.
QA of TOR for local consultants TL Core Team TL On-going e-mail with attached TOR with comments
Accounting TL OV monthly e-mails with comments
QA of inception report & QA-plan TL Core Team/
DCI 2 days before
dead-line
QA form for reports, related e-mails, and
documents with “track changes”.
Verification of declaration letters OV
QA of desk study report TL Core Team / Team of
Specialists / Local
consultants
DCI 3 days before
dead-line
QA form for reports, related e-mails, and
documents with “track changes”.
QA Audit AMS Twice during
the project
QA based on the project intranet site.
QA of field study report TL Core Team DCI 3 days before
dead-line
QA form for reports, related e-mails, and
documents with “track changes”.
QA of draft final synthesis report TL Core Team Technical: DCI
Layout: AMS
3 days before
dead-line
QA form for reports, related e-mails, and
documents with “track changes”.
QA of final synthesis report TL Core Team Technical: DCI
Layout: AMS
3 days before
dead-line
QA form for reports, related e-mails, and
documents with “track changes”.
Legend:
TL Team-Leader
QA Quality Assurance
AMS Annette Munk Sørensen
DCI Denise Colonna d’Istria
Joint Evaluation of the International Trade Centre
Version Final
132
OV Olivier Vautherin, external auditor
Paper versions of this document will be updated in one copy only.
Elaborated by: Controlled by: Approved by: Version: Date of approval:
AMS 1,2 09/02/04
MGS 2,3 10/02/04
Joint Evaluation of the International Trade Centre
Version Final
133
6.4.5 QA Annex 2 QA of Reports
Report:
Author(s):
[name + date and subject of e-mail or document with track changes in
relevant feed-back folder]or [meeting/workshop]
Feed-back
from:
filled in by
author(s)
Subject for QA QA-
person
[initials
or
name]
or [n/a]
Comments
Short comments should be
written in this form; longer
comments in e-mails saved
in the relevant QA-folder –
indicate in this column by
“e-mail” and subject +
date)
Document with
track changes
saved in the
relevant QA-
folder [no / yes +
version no.]
Date
of
QA
Changes
incorporated
[initials +
date +
document
version no.]
Technical QA
Quality of Information
(p. 80-82)
Validity and reliability
(p. 83-85)
Joint Evaluation of the International Trade Centre
Version Final
134
Subject for QA QA-
person
[initials
or
name]
or [n/a]
Comments
Short comments should be
written in this form; longer
comments in e-mails saved
in the relevant QA-folder –
indicate in this column by
“e-mail” and subject +
date)
Document with
track changes
saved in the
relevant QA-
folder [no / yes +
version no.]
Date
of
QA
Changes
incorporated
[initials +
date +
document
version no.]
Independence (p. 86-87)
Interdisciplinary
perspective (p. 88-90)
Evaluation ethics (p.91-93)
Is comparative advantage
of ITC covered?
Is performance of ITC
intervention covered?
Is capacity of ITC and
donor funding modalities
covered?
Others
Structure, language, etc.
Structure of report (p. 98-
99)
Editing (p. 100-102)
Layout
Joint Evaluation of the International Trade Centre
Version Final
135
Subject for QA QA-
person
[initials
or
name]
or [n/a]
Comments
Short comments should be
written in this form; longer
comments in e-mails saved
in the relevant QA-folder –
indicate in this column by
“e-mail” and subject +
date)
Document with
track changes
saved in the
relevant QA-
folder [no / yes +
version no.]
Date
of
QA
Changes
incorporated
[initials +
date +
document
version no.]
Others
Page numbers refer to Danida’s “Evaluation guidelines”
Paper versions of this document will be updated in one copy only.
Elaborated by: Controlled by: Approved by: Version: Date of approval:
AMS 1
Joint Evaluation of the International Trade Centre
Version Final
136
6.4.6 QA Annex 3 Template for Reports
List of contents
Error! No table of contents entries found.
1. First Title
Joint Evaluation of the International Trade Centre
Version Final
137
6.4.7 QA Annex 4 File Structure
Structure of the Filing System
Administration
o Budget
o Contracts
o Travel
o Minutes of meetings
o Activity Reports
o Expenses
Country Studies
o QA
o Draft
o Final
o Notes
o Documents
• Paper based documents
• Electronic documents
Desk Study Report
o QA
o Draft
o Final
Joint Evaluation of the International Trade Centre
Version Final
138
o Notes
o Documents
• Paper based documents
• Electronic documents
Evaluation Methodology
Evaluation Tools
Inception report
o QA
o Draft
o Final
o Notes
o Documents
• Paper based documents
• Electronic documents
Information on ITC
Templates
Terms of Reference
Joint Evaluation of the International Trade Centre
Version Final
139
1. Synthesis Phase
o QA
o Draft
o Final
o Notes
o Documents
• Paper based documents
• Electronic documents
2. Final Report
o QA
o Draft
o Final
o Notes
o Documents
• Paper based documents
• Electronic documents
Joint Evaluation of the International Trade Centre
Version Final
140
6.4.8 Structure of the WEB Server
(matches the filing system)
1. Administration
o Budget
o Contracts
o Travel
o Minutes of meetings
o Activity Reports
o Expenses
o Emails
2. Inception report
o QA
o Draft
o Final
o Notes
o Documents
3. Desk Study Report
o QA
o Draft
o Final
o Notes
Joint Evaluation of the International Trade Centre
Version Final
141
o Documents
4. Field Studies
o QA
o Draft
o Final
o Notes
o Documents
o Reports