External quality assessment of IAOD evaluation function - Final Report 29-06

WIPO - IAOD

EXTERNAL QUALITY ASSESSMENT OF THE EVALUATION FUNCTION

FINAL REPORT

Ian C Davies, CE, Consultant

June 29, 2014

Page 1 of 41

Executive Summary

This report presents the findings, analysis, conclusions and recommendations of the external quality assessment of the evaluation function of the Internal Audit and Oversight Division (IAOD) of the World Intellectual Property Organization (WIPO).

PurposeThe primary purpose of the external assessment is to provide assurance on IAOD evaluation conformity with the United Nations Evaluation Group (UNEG) Norms and Standards. The secondary purpose is to provide recommendations to improve the effectiveness of the IAOD evaluation function including the integration of Human Rights and Gender Equality in evaluations. Questions relative to evaluation resources and quality assurance are also addressed.

Conformity with UNEG norms and standardsThe review concludes that overall, IAOD’s evaluation function is generally consistent with UNEG norms and standards. The review concludes that there are specific areas that partially conform and require the attention of IAOD in order to strengthen their consistency with UNEG norms and standards and to contribute to the performance of the evaluation function. Although taken individually or collectively the assessed gaps in conformity do not threaten the fundamental integrity of the evaluation function, they are sufficiently important that they warrant action in each of the three assessed areas of independence, credibility and utility. Most importantly:

Publish, disseminate and communicate publicly all evaluation reports1.

Address bottlenecks in the conduct of evaluations that threaten their timeliness.

Develop quality standards specific to the IAOD evaluation function and put in place an explicit quality control process to manage for quality each of the processes of programming, planning, implementation, communication and follow up.

Address the evaluation function’s ability to fulfil its learning mandate and contribute to organisational knowledge.

1 It should be noted that the Proposed Revisions to the WIPO Internal Oversight Charter (May 22, 2014) states: The Director, IOD shall publish internal audit and evaluation reports on the WIPO website within 30 days of their issuance. In exceptional cases, if required to protect security, safety or privacy, the Director, IOD may, at his/her discretion, redact or withhold a report in its entirety.

Page 2 of 41

Human rights and gender equality (HRGE)To the question “to what extent are principles and implicit standards related to Human Rights and Gender Equality (HRGE) integrated in evaluations and what revisions to the Evaluation Policy and Strategy should be considered to further that integration?” the review concludes that principles and implicit standards related to Human Rights and Gender Equality (HRGE) are not yet fully integrated into IAOD evaluations.

The review recommends at the operational level:

Maintain regular professional development sessions for IAOD evaluation staff on gender responsive evaluation.

As part of to-be-developed quality control standards for evaluation in IAOD, quality control and quality assurance processes, operationalise and integrate HRGE.

Access specialised HRGE resources so that IAOD evaluations provide assessments and recommendations to the stakeholders, programmes, organisations and networks on HRGE levers for effectiveness and sustainability.

At the organisational level:

Articulate a specific policy component the integration of HRGE in all IAOD evaluations

Develop annual operational plans for the integration of HRGE integration in evaluation.

EffectivenessTo the questions “to what extent does the Evaluation Function contribute to WIPO accountability and learning and to what extent are evaluation results incorporated and used in follow-up activities, how they are monitored and what is their actual impact?” the review concludes that:

IAOD evaluation contributes to WIPO accountability however could contribute more to learning.

Although there is a system in place for monitoring and following up on evaluation recommendations, there is as yet little in the way of observable or reported impact.

The observed application of a predominant audit frame of reference to oversight may constrain the effectiveness of the IAOD Evaluation Function and may limit its ability to exploit fully opportunities for adding value to WIPO and its stakeholders.

Page 3 of 41

The review recommends that IAOD evaluation should:

Consider improving the quality and nature of its engagement with WIPO management .

Systematically include evaluability assessments at the programming stage of work planning for evaluations. .

Focus more systematically on intended users as part of programming and planning processes.

Identify and address systematically the bottlenecks in the evaluation process.

Consider diversifying its portfolio of evaluations to conduct short cycle evaluations.

Revisit the purpose and use of reference groups for evaluation.

ResourcesThe question that was put to the review is: To what extent are the resources available to the Evaluation Function adequate for it to achieve its mandate?

The review concludes that in terms of quality IAOD evaluation staff resources are adequate however the quantity of qualified evaluation staff may not be sufficient.

The review recommends IAOD monitor its level of allocation of qualified evaluation staff to its evaluation function and the feasibility of its intended workplan, based on a systematic and data based analysis of estimated workload.

Quality assuranceThe question that was put to the review is: How adequate is the quality assurance system for the Evaluation Function and in particular the periodicity of reviews?

The review concludes that there is not a sufficiently developed quality assurance system for the Evaluation Function of IAOD.

The review recommends that IAOD should:

Give priority to developing and instituting a quality control process for evaluation based on explicit quality standard.

Develop and institute a quality assurance process using professionally qualified evaluation experts external to IAOD.

Page 4 of 41

TABLE OF CONTENTS

EXECUTIVE SUMMARY 2

1. INTRODUCTION 7

2. DATA COLLECTION 7

3. ASSURANCE ON CONFORMITY 8

3.1 Findings 83.1.1 Independence 93.1.2 Credibility 123.1.3 Utility 16

3.2 Conclusion 19

3.3 Recommendations 19

4. HUMAN RIGHTS AND GENDER EQUALITY 20

4.1 Findings 20

4.2 Analysis 21

4.3 Conclusion 21


5. ASSESSMENT OF EFFECTIVENESS 23

5.1 Findings 235.1.1 The view from WIPO management 235.1.2 Understanding of evaluation 245.1.3 Programming 255.1.4 Evaluation process 25

5.2 Analysis 265.2.1 Evaluation and audit 265.2.2 Organisation 275.2.3 Operations 27

5.3 Conclusion 29


6. ADEQUACY OF RESOURCES 32

6.1 Findings 32

6.2 Conclusion 32

Page 5 of 41

6.3 Recommendation 32

7. ADEQUACY OF QUALITY ASSURANCE PRACTICES 33

7.1 Findings 33

7.2 Conclusion 33


8. WIPO-IAOD EVALUATION FUNCTION SURVEY 34

8.1 Relevance and use of evaluations 34

8.2 Evaluation process 34

8.3 Communication of evaluation findings 35

8.4 Quality of evaluations 35

8.5 Additional comments 35

8.6 Observations 35

9. ADDITIONAL REMARKS 36

10. SUMMARY OF RATINGS 37

Page 6 of 41

1. Introduction

This report presents the findings, analysis, conclusions and recommendations of the external quality assessment of the evaluation function of the Internal Audit and Oversight Division (IAOD) of the World Intellectual Property Organization (WIPO).

The assessment was conducted by the external professional consultant Ian C Davies, Credentialed Evaluator (CE) over the period from April to June 2014.

The assessment is first and foremost an assurance engagement to assess the extent of the Evaluation Function’s conformity with the United Nations Evaluation Group (UNEG) Norms and Standards for evaluation in the UN system.

The secondary purpose of the assessment is to provide recommendations to IAOD on improving guidance for, and effectiveness of, the evaluation function including the integration of Human Rights and Gender Equality in evaluations. Questions relative to evaluation resources and quality assurance are also addressed.

2. Data collection

The assessment was conducted based on data collected through systematic document review, semi-structured interviews, meetings with IAOD staff and a web-based survey questionnaire administered to WIPO management respondents.

The document review included all key guidance documents pertaining to the evaluation function in WIPO, full documentation on IAOD, comprehensive documentation of the evaluation function of IAOD including, but not limited to, working papers, evaluation reports, relevant job descriptions and staff CVs, comprehensive complement of evaluation documents from programming through to planning, conduct, reporting, follow up as well as documentation pertaining to external consultants.

The review comprised a one week intensive onsite visit to WIPO headquarters for meetings, interviews, file reviews and the detailed examination of working papers and web based systems.

A descriptive summary of the results of the web based survey is included at the end of this assessment report.

Page 7 of 41

3. Assurance on conformity

This chapter looks at the extent to which the organisation, systems and processes of IAOD and its Evaluation Section are consistent, i.e. in conformity, with the norms and standards of the United Nations Evaluation Group (UNEG).

Conformity with norms and standards is an ad minima proposition, i.e. it means that the organisation is set up, has procedures and behaves in a manner consistent with what are considered to be minimum criteria.

Examining and assessing conformity allows for flagging areas that may be below standard, i.e. do not conform, and that require immediate attention and correction as they are essential to the functioning of the organisation, as well as areas that conform partially and call for including or strengthening aspects that may be already in place or planned for. It is however key to understand that conformity with norms and standards does not imply nor does it equate with good performance. Conformity is a necessary but not sufficient condition for good performance the question of which is dealt with in chapter 5.

3.1Findings

The assessment undertook to address the following question:

To what extent do the evaluation services and products of the Evaluation Function, in particular the conduct of evaluation processes and the quality of evaluations, conform to UNEG Norms and Standards for Evaluation?

The review drew key findings from the data that were collected and assessed these against the full set of UNEG norms and standards with a focus on UNEG’s normative2 framework for Independence, Credibility and Utility of evaluation function.3 The normative framework is the basis on which UNEG Peer Reviews are conducted of UN organisations with a normative mandate as is the case with WIPO.

2 The UNEG Annual General Meeting, 2012, defined normative work as: “The support to the development of norms and standards in conventions, declarations, regulatory frameworks, agreements, guidelines, codes of practice and other standard setting instruments, at global, regional and national level. Normative work also includes the support to the implementation of these instruments at the policy level, i.e. their integration into legislation, policies and development plans, and to their implementation at the programme level. (UNEG, 2012, p. 5)” 3 Please see http://www.uneval.org/papersandpubs/documentdetail.jsp?doc_id=1484

Page 8 of 41

http://www.uneval.org/papersandpubs/documentdetail.jsp?doc_id=1484

3.1.1 Independence

Independence is a function of statutory and organisational dispositions, as well as explicit and adequate support to behavioural independence. Accordingly the findings on independence are based to a large extent on the examination of WIPO’s key guidance and organisational documents for IAOD’s evaluation function, of working papers and reports for evaluations, of IAOD evaluation staff CVs and position descriptions as well as interviews with IAOD staff.

The following presents key findings for independence against relevant UNEG norms:

N2.3: The governing bodies/heads of organizations are responsible for ensuring that adequate resources are allocated to enable the evaluation function to operate effectively and with due independence.

Responsibility with respect to the allocation of resources to IAOD, including the Evaluation Function, rests with the Director General of WIPO (the head of the organisation). The Program and Budget of WIPO is presented to the General Assembly (the governance level) for approval.4

Status: Conforms

N.5.1: Impartiality is the absence of bias in due process, methodological rigor, consideration and presentation of achievements and challenges. The requirement for impartiality exists at all points in the process: planning, conduct, reporting

The requirement for impartiality is found in the IAOD’s guidance and operational documents and is assessed as consistent with generally accepted practice in the UN system and more generally with evaluation principles such as those found in OECD-DAC guidance.

In addition, the review’s in depth examination of working documents of IAOD’s Evaluation Function does not show evidence of bias or perception thereof.

Status: Conforms

4 It is important to distinguish the level at which lies responsibility for the allocation of resources, which is what the norm refers to, from the question of adequacy of resources of the Evaluation Function to fulfil the mandate given to it, which is addressed in 3.4

Page 9 of 41

N6.1: The evaluation function has to be located independently from the other management functions so that it is free from undue influence and that unbiased and transparent reporting is ensured. It needs to have full discretion in submitting directly its reports for consideration at the appropriate level of decision-making pertaining to the subject of evaluation.

The evaluation function is located in IAOD and, as such, is independent from management functions of WIPO. The independence of IAOD, and therefore of its Evaluation Section, is enshrined in the WIPO Internal Oversight Charter, which states in section D, article 4, : “The Director, IAOD, enjoys functional and operational independence from the Secretariat and Management in the conduct of his/her duties”. It also states that “The Director IAOD is responsible to the Director General (…)” and further states that “ He/she has the authority to initiate, carry out and report on any action, which he/she considers necessary to fulfil his/her mandate.”

Based on the authority and prerogatives conferred to the Director of IAOD by the Charter as well as a systematic review of evaluations completed since 2012 and their transmission, the assessment considers that the norm is respected.

Status: Conforms

N6.2: The head of evaluation must have the independence to supervise and report on evaluations.

Although there is a position of Head of Evaluation Section, it is part of IAOD for which designated responsibility for oversight, including evaluation, rests with the Director of IAOD. The review finds that the independence to supervise evaluations is well defined in guidance documentation and effected based on operational documentation and interviews with staff.

The WIPO evaluation policy states: “The IAOD Evaluation Section will ensure that all evaluation reports are disclosed to the public at the completion of the evaluation process and disseminated widely through the print and electronic media in accordance with WIPO’s disclosure policy”.5

The review found that evaluation reports are not always systematically made available to the public and in cases the practice appears to be to provide them to Member States on request through “Brainloop”6, to make them available on WIPO’s intranet in some cases and to publish public summaries in some instances.

This finding indicates potential inconsistencies with the Revised WIPO Evaluation Policy which states that : “All evaluation reports are posted on the WIPO website and are accessible to the public.”7

5 IAOD Revised Evaluation Policy – May 4, 2010 – article 596 A secure password protected cloud based storage facility http://www.brainloop.com/en/products/brainloop-secure-dataroom.html 7 Revised Evaluation Policy (May 4, 2010) paragraph 23. Protection from outside interference.

Page 10 of 41

http://www.brainloop.com/en/products/brainloop-secure-dataroom.html

The review did not examine the extent to which WIPO’s disclosure policy had a bearing on the above finding nor is this issue clearly addressed from a public information perspective. Because public perception of independence, and transparency, are important to the credibility of the evaluation function, its products and more broadly WIPO’s activities, the relationship, if any, between WIPO’s disclosure policy and public reporting of evaluations should be clearly addressed. More importantly public reporting should be consistent with WIPO policy and the intent expressed in the proposed revised Charter.8

Status: Partially conforms

N6.3: To avoid conflict of interest and undue pressure, evaluators need to be independent, implying that members of an evaluation team must not have been directly responsible for the policy-setting, design, or overall management of the subject of evaluation, nor expect to be in the near future.

From a review of the documentation made available to it, including CVs of staff of the evaluation function, as well as interviews with said, the norm as stated above is met.

Status: Conforms

N6.4: Evaluators must have no vested interest and have the full freedom to conduct impartially their evaluative work, without potential negative effects on their career development. They must be able to express their opinion in a free manner.

The review finds that the WIPO Internal Oversight Charter9, Evaluation Policy as well the quality control processes in place in IAOD at each stage of the evaluation and related processes ensure that the evaluators do not have vested interests and are able to conduct their activities impartially. This finding extends to the external consultants that are retained for IAOD evaluations.

Status: Conforms

8 It should be noted that the Proposed Revisions to the WIPO Internal Oversight Charter (May 22, 2014) states: The Director, IOD shall publish internal audit and evaluation reports on the WIPO website within 30 days of their issuance. In exceptional cases, if required to protect security, safety or privacy, the Director, IOD may, at his/her discretion, redact or withhold a report in its entirety.9 WIPO Internal Oversight Charter

Page 11 of 41

3.1.2 Credibility

Credibility is to a large extent based on the perception that stakeholders have of evaluation. While there are factors that affect perception and fall outside the reasonable control of the evaluation function, good practice as reflected in the following norms shows that there are key aspects of the way in which evaluation is conducted that affect positively and constructively credibility. Credibility is important because, among other things, it increases the likelihood of evaluation use.

The following presents key findings for credibility against relevant UNEG norms:

N4.2: The evaluation plan can be the result of a cyclical or purposive selection of evaluation topics. The purpose, nature and scope of evaluation must be clear to evaluators and stakeholders. The plan for conducting each evaluation must ensure due process to ascertain the timely completion of the mandate, and consideration of the most cost-effective way to obtain and analyze the necessary information.

The review finds that evaluation planning at the organisational level is both cyclical and purposive based on the WIPO Evaluation Section: 2010-2015 Strategic Plan. Generally, from a planning perspective, the strategic plan is consistent with good practices in evaluation and coherent with the Charter and the Evaluation Policy.

At the level of specific evaluations, and on the basis of document review and interviews, the review finds that, generally, their purpose, nature and scope is well understood by both the evaluators of IAOD and by stakeholders who have participated in them. The review was not able to ascertain with a sufficient degree of confidence the extent to which the purpose, nature and scope of specific evaluations was clear to externally contracted consultants, however with responsibility for the evaluation retained by IAOD in all cases risks are appropriately mitigated.

The systematic review10 of three final evaluation reports11 based on the UNEG Quality Checklist for Evaluation Reports (2010)12 shows weaknesses in section 3. Evaluation Purpose, Objective (s) and Scope with an average rating across all three reports of 1.513 (on a four point rating scale where 1= poor and 4=excellent).

10 The review of evaluation reports was effected separately by two independent reviewers.11 Country Portfolio Evaluation Thailand. Program 1 (Patent Law). Improvement of National, Sub-regional and Regional IP Institutional and User Capacity.12 http://www.uneval.org/papersandpubs/documentdetail.jsp?doc_id=607 13 A maximum 3 was given only once for the Program 1 report.

Page 12 of 41


As well, based on the examination of working papers for evaluations and interviews, the review finds that the processes for reaching agreement on terms of reference, inception and on reporting with primary stakeholders may put at risk the timely completion of the mandate.

Although generally the factors contributing to the credibility of evaluation identified in this norm are addressed by the IAOD evaluation function there are areas requiring strengthening for the evaluation reports and the timeliness of completion.


N5.2: Impartiality increases the credibility of evaluation and reduces the bias in the data gathering, analysis, findings, conclusions and recommendations. Impartiality provides legitimacy to evaluation and reduces the potential for conflict of interest.

Building on the findings in N5.1 and based on evaluation reports and working papers, the review examined the methodology used in each of the three evaluations the reports of which were assessed, with a focus on data collection methods and sources, findings, conclusions and recommendations. The review found no evidence of bias.

In addition, data provided through interviews with stakeholders involved in IAOD evaluations as well through a web based survey questionnaire14, provide no indication of bias. To the survey question “To what extent are evaluations conducted transparently, independently and impartially?”, respondent ratings averaged at 3.73 (on a five point rating scale where 1= not at all and 5=fully)

Provisions for avoiding conflict of interest exist for external consultants involved with evaluations.

Status: Conforms

N8.1: Each evaluation should employ design, planning and implementation processes that are inherently quality oriented

Based on its examination on working documents related to evaluations, interviews with evaluation staff as well with stakeholders involved in evaluations, the review finds that there is systematic effort by the IAOD Evaluation Section to carry out key steps in the evaluation process in a manner that is consistent with general principles of good practice.

14 The survey consisted of 10 questions with 5 point Likert scales and one open ended question. Invitations were sent to 34 recipients with 14 respondents (including partial collections) for a response rate of 41%.

Page 13 of 41

Reference is made to UNEG Norms and Standards however the review did not find an explicit quality control process in place to manage for quality each of the processes of design, planning and implementation. The review notes that IAOD uses the UNEG Checklist for reports when conducting its peer review quality checks.


N9: The head of the evaluation function, evaluation staff and evaluators should have proven competencies to manage and/or conduct evaluations15

The review examined the job descriptions, curriculum vitae, of IAOD evaluation function staff and finds that, given the European context and the corresponding stage of development of professional evaluation and related training, they have collectively and experientially proven competencies to manage and conduct evaluations.

Status: Conforms

N10.1: Transparency and consultation with the major stakeholders are essential features in all stages of the evaluation process. This improves the credibility and quality of the evaluation. It can facilitate consensus building and ownership of the findings, conclusions and recommendations.

Based on interviews with WIPO stakeholders who had participated in evaluations, the results of the survey16, the examination of working papers and the assessment of evaluation reports, the review finds that, generally, key stakeholders are consulted as part of the evaluation.

Stakeholder participation could be strengthened through more systematic involvement in the programming phase of evaluation. Stakeholder analysis should be undertaken systematically for all evaluations. Evaluation reports should improve their reporting on describing stakeholder consultation processes as well as the rationale for these. Full access to all evaluation reports would further increase transparency.

Status: Conforms

15 The formulation of Norm 9 – Competencies for Evaluation is a synthesis of its components 91, 9.2, 9.316 From the survey summary: The extent to which evaluation processes engage with the survey respondents and their stakeholders in ways that make evaluations useful received an average rating of 3.15 with the extremes of the rating scale only being used by one respondent each.

Page 14 of 41

N11: Evaluators must have personal and professional integrity, respect people and their rights, confidentiality of data and sources, and be sensitive to beliefs as well as issues of gender and human rights.17

From its examination of working papers and interviews with WIPO stakeholders and IAOD staff the review considers the professional deportment of the IAOD Evaluation Function staff to be consistent with the above.

Status: Conforms

N8.2: Evaluation reports must present in a complete and balanced way the evidence, findings, conclusions and recommendations. They must be brief and to the point and easy to understand. They must explain the methodology followed, highlight the methodological limitations of the evaluation, key concerns and evidenced-based findings, dissident views and consequent conclusions, recommendations and lessons. They must have an executive summary that encapsulates the essence of the information contained in the report, and facilitate dissemination and distillation of lessons.

Notwithstanding that some areas of the evaluation reports18 that the review examined in detail offer opportunities for quality improvements, the review finds that, generally, reports adhere to the principle of fair and balanced reporting. 19

In addition, the survey found that on balance, respondents indicated that the findings were communicated in a constructive manner.

Status: Conforms

N10.2: documentation on evaluations in easily consultable and readable form should also contribute to both transparency and legitimacy

Please see observations under N6.2


17 The formulation of Norm 11 – Evaluation Ethics is a synthesis of its components 11.1, 11.2, 11.3, 11.4, 11.518 Country Portfolio Evaluation Thailand. Program 1 (Patent Law). Improvement of National, Sub-regional and Regional IP Institutional and User Capacity.19 Item 6.4 - UNEG Quality Checklist for Evaluation Reports (2010)

Page 15 of 41

3.1.3 Utility

Utility recognises the fundamental purpose of evaluation as a discipline that contributes to learning and progress through the provision of reliable and meaningful information that can be used. In other words, evaluation that is not useful and used does not have value. In this sense evaluation is different from social science research which does not necessarily have to be used to be considered valuable. As such the concept of utility is central to sound evaluation.

The following presents key findings for utility against relevant UNEG norms:

13.1 Evaluation contributes to knowledge building and organizational improvement. Evaluations should be conducted and evaluation findings and recommendations presented in a manner that is easily understood by target audiences.

The review finds that, overall, in guidance documents for the Evaluation Function the learning aspect of evaluation, i.e. as a component of knowledge building, is put forward. In particular the WIPO Evaluation Policy highlights the role of the IAOD Evaluation Section in contributing to the Organisation’s learning20.

However, based on the examination of the selected evaluation reports21, the feedback provided by WIPO stakeholders who had participated in IAOD evaluations and to a lesser extent the responses to the survey, the review finds that in practice the contribution of evaluation to knowledge building and organizational improvement is still weak and of little consequence.

From an evaluation output perspective the review finds that the “lessons learned” tend either to be identified “shortcomings” to which solutions are proposed. These solutions are too specific to the evaluand and its context to be considered lessons learned.

Or they are re-statements of well know principles of good practice, e.g. “The key lesson here is that the Project design and management style/approach have an important effect on the outcomes of Projects. It is a matter that should therefore be taken seriously.”

The review further finds that in most cases recommendations tend to be general and insufficiently specific to be actionable.

20 Part 3 section 4 paragraph (p.12): i) Promoting and supporting best practice in developmental evaluation and developing

appropriate and user-friendly mechanisms for the collection, publication and dissemination of lessons learned.

ii) Creating for a with internal and external stakeholders to share and discuss findings and feeding into future decision-making

iii) Developing and managing a public WIPO web site dealing with evaluation.21 Country Portfolio Evaluation Thailand. Program 1 (Patent Law). Improvement of National, Sub-regional and Regional IP Institutional and User Capacity.

Page 16 of 41

Interviews with WIPO stakeholders highlight concerns with the lack of substantive contribution of evaluation to the “business of IP” together with a lack of understanding of IP and insufficient knowledge of the administrative, programmatic and management issues of the organisation. As a result, and although there is generally a positive appreciation of the work of the IAOD evaluation function, there is little perception, understanding and recognition of its contribution to organisational improvement and knowledge.


N2.6: The Governing Bodies and/or Heads of organizations and of the evaluation functions are responsible for ensuring that evaluation contributes to decision making and management. They should ensure that a system is in place for explicit planning for evaluation and for systematic consideration of the findings, conclusions and recommendations contained in evaluations. They should ensure appropriate follow-up measures including an action plan, or equivalent appropriate tools, with clear accountability for the implementation of the approved recommendations.

WIPO instituted a web based recommendation management system, TeamCentral, in March 2013, migrating from an Excel open recommendation spreadsheet. The review finds that there is a systematic process whereby the Director of IAOD provides to the Director General of WIPO regular updates on the status of the Oversight Recommendations. The updates are simple statistical descriptions of the status of implementation of recommendations according to variables such as programmes and importance. The update encompasses all oversight recommendations. The updates do not present analysis.

The review did not examine the extent to which specific recommendations from evaluations contribute to decision-making and management. Based on data collected through interviews with WIPO stakeholders and through the survey, the review finds that the recommendations and their follow up tend to be perceived by management more as accountability and compliance checks rather than substantive inputs to management decision making.

Based on summaries of ratings from Client Satisfaction Questionnaires provided to it by IAOD, the review notes that, to the statement “The lessons learned from this audit/evaluation have a positive impact on your management processes and will help you achieve the objectives of your program/unit/project” the average rating was 4 “I agree” (for 7 evaluations on a rating scale from 5 – Totally agree, to 1 – Totally disagree)


Page 17 of 41

N4.1 Proper application of the evaluation function implies that there is a clear intent to use evaluation findings. In the context of limited resources, the planning and selection of evaluation work has to be carefully done. Evaluations must be chosen and undertaken in a timely manner so that they can and do inform decision-making with relevant and timely information. Planning for evaluation must be an explicit part of planning and budgeting of the evaluation function and/or the organization as a whole. Annual or multi-year evaluation work programmes should be made public.

Building on the findings presented in N2.3, the 2010-2015 Evaluation Strategy as well as consultations by the Director of IAOD with senior management of WIPO as part of the evaluation programming process, the review finds that there is clear intent to make the evaluations relevant and useful to management. As well care is taken in the programming of evaluations to make clear its dependence on allocation and levels of resources available to the evaluation function. Planning for evaluation is an explicit part of IAOD’s programme and budget planning which itself feeds into WIPO’s overall program and budget on a cyclical basis. The 2010-2015 Evaluation Strategy is publicly accessible on WIPO’s website.

Status: Conforms

N4.2 The evaluation plan can be the result of a cyclical or purposive selection of evaluation topics. The purpose, nature and scope of evaluation must be clear to evaluators and stakeholders. The plan for conducting each evaluation must ensure due process to ascertain the timely completion of the mandate, and consideration of the most cost-effective way to obtain and analyze the necessary information.

Please see N4.1

Status: Conforms

N2.7: governing bodies and/or heads of organizations and of the evaluation function are responsible for ensuring that there is a repository of evaluation and a mechanism for stilling and disseminating lessons to improve organizational learning and systemic improvement

The review finds that there is an archive for evaluations. Please see N2.6 and N13.1.


Page 18 of 41

3.2Conclusion

The review concludes that overall, IAOD’s evaluation function is generally consistent with UNEG norms and standards.

The review concludes that there are specific areas that partially conform and require the attention of IAOD in order to strengthen their consistency with UNEG norms and standards and to contribute to the performance of the evaluation function.

Although taken individually or collectively the assessed gaps in conformity do not threaten the fundamental integrity of the evaluation function, they are sufficiently important that they warrant action in each of the three assessed areas of independence, credibility and utility.

3.3Recommendations

IAOD should take action required to:

Publish, disseminate and communicate publicly all evaluation reports22.

Address bottlenecks in the conduct of evaluations that threaten their timeliness.

Develop quality standards specific to the IAOD evaluation function and put in place an explicit quality control process to manage for quality each of the processes of programming, planning, implementation, communication and follow up.

Address the evaluation function’s ability to fulfil its learning mandate and contribute to organisational knowledge (the more specific aspects of which are dealt with further in this report).

22 It should be noted that the Proposed Revisions to the WIPO Internal Oversight Charter (May 22, 2014) states: The Director, IOD shall publish internal audit and evaluation reports on the WIPO website within 30 days of their issuance. In exceptional cases, if required to protect security, safety or privacy, the Director, IOD may, at his/her discretion, redact or withhold a report in its entirety.

Page 19 of 41

4. Human rights and gender equality

Human rights and gender equality are a cornerstone of the UN’s mission and values. As such they have garnered priority and system wide attention. In particular these issues are recognised as key to economic and social development, among other things, and they should be woven into the fabric of UN organisations, their interventions and their evaluation. It is on this latter aspect that evaluation in WIPO has an opportunity through its evaluations and recommendations to support the inclusion of HRGE considerations in the WIPO interventions it evaluates.

4.1Findings

The assessment undertook to address the following question:

To what extent are principles and implicit standards related to Human Rights and Gender Equality (HRGE) integrated in evaluations and what revisions to the Evaluation Policy and Strategy should be considered to further that integration?

The review drew key findings from the data that were collected and assessed these against UNEG norms and standards with a focus on HRGE and as well the principles, tasks and checklist highlighted in the UNEG guidance document: Integrating Human Rights and Gender Equality in Evaluation23

The review examined the three evaluation reports24 that were submitted to it and found that in none of these was there reference to gender and human rights25 . The review further examined the evaluation report “Technology and Innovation Support Centers (TISCs)” and the summary of the evaluation report26 “Support Services to the Intergovernmental Committee on Intellectual Property and Genetic Resources, Traditional Knowledge and Folklore (IGC)” and found no reference to gender and human rights.

From the examination of evaluation working papers and interviews with staff the review found little evidence of systematic integration of HRGE in evaluation.

23 http://www.unevaluation.org/HRGE_Guidance 24 Country Portfolio Evaluation Thailand. Program 1 (Patent Law). Improvement of National, Sub-regional and Regional IP Institutional and User Capacity.25 Section 8 of the UNEG Quality checklist for evaluation reports - http://www.uneval.org/papersandpubs/documentdetail.jsp?doc_id=607 26 The review was unable to access publicly the full evaluation report http://www.wipo.int/about-wipo/en/oversight/iaod/evaluation/

Page 20 of 41

http://www.wipo.int/about-wipo/en/oversight/iaod/evaluation/


http://www.unevaluation.org/HRGE_Guidance

The review found that the WIPO Evaluation Policy contains only two references to gender, one as an example of a theme that may be addressed within a defined sector and the other as part of the section of Evaluation Ethics.

It further found that the WIPO Evaluation Strategy refers to gender as part of its explanation of coverage criterion and goes onto state: “For all its independent evaluations, the Evaluation Section will uphold the UN Systemobligations and commitments to gender equality and human rights, making sure that these are increasingly mainstreamed into all of its independent evaluation work.”

The review did take note of the appointment of an IAOD internal audit staff person as focal point on gender within WIPO and to the UNEG Taskforce on Gender Equality and Human Rights working with, inter alia, the UN SWAP - Meta-Review/Evaluation Scoring Tool27 .

4.2Analysis

The findings indicate that principles and implicit standards related to Human Rights and Gender Equality (HRGE) are not yet integrated fully in IAOD evaluations.

The UNEG Gender related Norms and Standards provide a framework for gender responsive evaluations which are wholly applicable to the various evaluands of IAOD. Considerations of HRGE aspects is relevant to the evaluation of WIPO interventions and may be significantly important in areas such as Intellectual Property and Genetic Resources, Traditional Knowledge and Folklore.28

4.3Conclusion

The review concludes that principles and implicit standards related to Human Rights and Gender Equality (HRGE) are not yet fully integrated into IAOD evaluations. Because of the importance of HRGE it should be integrated fully to evaluation guidance, i.e. the Oversight Charter and Evaluation Policy, as well as to the Evaluation Strategies.

27 http://uneval.org/papersandpubs/documentdetail.jsp?doc_id=1452 28 For example see: Gender dimensions of intellectual property and traditional medicinal knowledge http://www.undp.org/content/dam/aplaws/publication/en/publications/environment-energy/www-ee-library/biodiversity/gender-dimensions-of-intellectual-property-and-traditional-medicinal-knowledge/Gender_IP_and_TKM_Final_27_Apr07.pdf

Page 21 of 41

http://www.undp.org/content/dam/aplaws/publication/en/publications/environment-energy/www-ee-library/biodiversity/gender-dimensions-of-intellectual-property-and-traditional-medicinal-knowledge/Gender_IP_and_TKM_Final_27_Apr07.pdf



http://uneval.org/papersandpubs/documentdetail.jsp?doc_id=1452

4.4Recommendations

At the operational level:

Maintain regular professional development sessions for IAOD evaluation staff on gender responsive evaluation.

As part of to-be-developed quality control standards for evaluation in IAOD, quality control and quality assurance processes, operationalise and integrate HRGE.

Access specialised HRGE resources so that IAOD evaluations provide assessments and recommendations to the stakeholders, programmes, organisations and networks on HRGE levers for effectiveness and sustainability.

At the organisational level:

Articulate a specific policy component the integration of HRGE in all IAOD evaluations

Develop annual operational plans for the integration of HRGE integration in evaluation.

Page 22 of 41

5. Assessment of effectiveness

Different from the thrust and nature of the examination and assessment conducted on conformity by the review29, this chapter addresses the question of performance of IAOD’s evaluation function relative to its goals, the mission of WIPO and its contribution to development in Member states. The question of performance is an ad maximis proposition, i.e. to what extent is the evaluation function managed to achieve its goals and what can be done to improve it.

Because use of evaluation is integral to its effectiveness and impact on the organisation and its stakeholders, it is also addressed in this chapter.

5.1Findings

The assessment undertook to address the following questions:

Taking into account IAOD capacity and policy, to what extent does the Evaluation Function contribute to WIPO accountability and learning, as set forth in the Internal Oversight Charter and expressed in the expectations of WIPO management?

To what extent are evaluation results incorporated and used in follow-up activities, how they are monitored and what is their actual impact?

The review drew its findings from interviews with WIPO management staff, IAOD staff, the survey of WIPO management staff as well as documentation such as the Oversight Charter, the proposed revisions to the Oversight Charter, the Evaluation Policy, the 2010-2015 Evaluation Strategy, evaluation reports and working papers.

As well, key aspects relating to use, follow up and monitoring are addressed in Chapter 3, 3.1.3 Utility, N4.1.

5.1.1 The view from WIPO management

Overall the review finds that WIPO management are receptive to and supportive of evaluation in WIPO. Generally WIPO management would like more evaluations of their programmes and areas of work, and consider the work carried out by the evaluation function to be relatively independent, impartial and constructive.

29 Please see chapter 3.

Page 23 of 41

On the other hand, there is a view that the evaluation function does not reach out enough to management to well understand its needs and that of the organisation as a whole. The fact that IAOD has the ability to “oversee” the entire organisation puts it in a unique position which provides it with a comparative advantage that is not seen to be sufficiently exploited. There are opportunities to provide input and support at the policy level of WIPO which do not appear to be recognised and acted on at present.

The management view is formulated that the evaluation function in IAOD has still not reached maturity. It is seen as insufficiently distinct from internal audit and from a control approach, and has yet to establish firmly its substantive competence and credibility. Its niche remains unclear and there is a still a way to go in engaging with management before it can capture the sense of value that evaluation deserves.

A majority of interview and survey respondents consider evaluation in WIPO to be primarily about fulfilling accountability requirements. And although there is an understanding of the learning aspect of evaluation, respondents generally consider that evaluation does not provide significantly for learning and for improving practices.

Evaluation recommendations are on balance considered too vague or general to be meaningful, and for some, reflect a lack of substantive knowledge of the evaluand, i.e. the subject areas that are evaluated.

The system for follow up of recommendations is considered to be more of a control and compliance exercise, with little value added in terms of operational management or improvement of practices, than a learning function.

5.1.2 Understanding of evaluation

There are differences in level of appreciation of the evaluation function according to the degree of direct implication of respondents in an evaluation. Generally the greater the implication the more positive the appreciation. This may suggest, among other things, that the evaluation process itself has inherent value that is equal to if not superior to the perceived value, and hence the utility and use, of the evaluation report including its recommendations.

This said, evaluations that are carried out “at the request” of management in a specific area tend to be benefit from what could best be termed a “positive pre-disposition” of managers who know about evaluation, albeit in general terms, understand its distinction from control and audit, and are interested in the added value of external assessment of performance and the learning potential evaluation brings.

Page 24 of 41

On the other hand, the review found that for those management respondents less directly involved in evaluation, the distinction between evaluation and the other functions of IAOD, in particular internal audit but as well control approaches in general, was not clear.

As a result, management less familiar with evaluation tend to have a less positive view of evaluation, are not particularly interested in requesting it, consider it a control exercise and do not see much added value from evaluation to their management and to the performance of their programmes.

5.1.3 Programming

The positive correlation between appreciation for evaluation and extent of direct participation in it is also found to carry through to the appreciation of the relevance of the topic that is evaluated.

In all cases however there is a clear expectation and oft formulated request from management for more systematic and direct involvement with IAOD’s Evaluation Section in the programming and planning of evaluations, i.e. with the evaluators themselves. There is a view that the evaluation function does not engage sufficiently in dialogue with management and that, as a result, it does not meet well the needs and requirements of the management of WIPO.

Generally the reasons motivating more involvement in programming of evaluations tend to fall into two categories. One set of reasons has to do with evaluating programmes or components thereof that are known not to be functioning well or that face significant challenges, in order to better understand why, identify possible solutions and actions to implement, monitor and adjust these.

The other set of reasons falls into a category which could be best summed up as learning for improvement, i.e. where a programme is functioning acceptably and management is looking for ways to improve and scale up.

5.1.4 Evaluation process

For WIPO management involved more directly in evaluations there is a common view that evaluation processes tend to take too long and incur transaction costs that are not always commensurate with the benefits to management, the programme, the organisation and related stakeholders.

While appreciative of the thoroughness and attention given by the evaluation function to the conduct of evaluations and effort to engage constructively during the process, there is a view that evaluation should be “quick and light” to be timely, responsive and relevant to management and organisational issues in WIPO.

Page 25 of 41

The view and appreciation from WIPO management on external consultants and their use is at best mixed. Although they tend to be retained by IAOD’s evaluation function for their IP expertise it does not appear that their input translates into substantive added value for management. As well the role and extent of responsibility for the evaluation of external consultants does not always appear clear to management.

5.2Analysis

Overall IAOD evaluation is conceived, planned, carried out, perceived and received as an accountability rather than a learning function. This despite the fact that learning is put forward as a key objective in WIPO evaluation guidance documents as well as in IAOD evaluation related documentation.

As such, the unique added value that evaluation can provide as a discipline and profession by drawing on its extensive and deep body of theory and practice, remains largely under-utilised. Instead the evaluation function in WIPO is limited to a significantly and relatively restricted conception and application as an “accountability tool”, with accountability understood in the classic sense of “The obligation to account for a responsibility conferred”.

Quite aside from the truism that just because it is written does not mean that it will necessarily get done, different perspectives on this reality provide some measure of explanation for the state of the evaluation function’s effectiveness, use and impact in WIPO.

5.2.1 Evaluation and audit

The fundamental distinction and differences between evaluation and audit is not reflected in a manner that is sufficiently clear, systematic and appropriate in the organisation, systems and processes for evaluation in WIPO.

This said, the fact that IAOD’s examination and validation of WIPO’s Program Performance Report (PPR) has moved from the evaluation function to the audit function is for example wholly consistent with the fundamental assurance remit of audit rather than that of evaluation.

While audit may be considered as a type of “evaluative” exercise, it is a process that is superimposed on an accountability relationship for the purpose of providing assurance. Audit considers the “what” question, e.g. what was done, against expectations.

Evaluation addresses the range of questions from the “what”, to the “why”, the “how” and, most importantly, the “so what”, for “who”, under “what circumstances”. While evaluation as a discipline provides a almost limitless array of approaches, methodologies and techniques for determining value, it does not provide the best configurations for assurance and control purposes. This is the specialised purview of audit.

Page 26 of 41

5.2.2 Organisation

PositionThe organisational position of evaluation in an oversight function brings with it known and specific challenges, but may also bring benefits depending on overall context. In the case of WIPO a clear benefit is the independence from operational management of IAOD in which the evaluation function is located.

The basic challenge is one of identity of evaluation, both within the oversight function and for stakeholders external to it. Within the oversight function “evaluation” as conceived, programmed, managed, conducted and followed up is sometimes difficult to distinguish from performance audit.

GuidanceThis is to some extent the case with the current and proposed guidance frameworks for IAOD’s evaluation function. The definitions of evaluation in the Oversight Charter, and moreover those of the proposed revisions to the Charter, correspond to those applicable to a performance audit, i.e. examining the “what”, more than focussing on the determination of value.

From a management and stakeholder view point external to the oversight function, evaluation is also often difficult to understand as something more than an accountability, and specifically a control, mechanism. As such, that perception tends not to be conducive to the type of engagement that allows for and nurtures mutual learning. Furthermore that perception conditions the way in which recommendations and their follow up are treated, i.e. as control requirements rather than as added value to management.

5.2.3 Operations

Programming and planningThe overarching audit approach within IAOD is also reflected in the use of risk assessment as the initial frame of reference for prioritising its annual work plan. The revised evaluation policy states in article 25: “Planning priorities will be based on risk-assessment, size and needs for evaluation.” While risk assessment is suited to assurance and performance audit, it is not the most useful frame for evaluation.

As such, and in addition to consultation with management on selection of evaluation topics, a framework that is anchored in a value proposition permits considerations beyond achievement of objectives. A evaluation specific value oriented frame of reference allows for a greater choice of evaluation approaches and methodologies, in particular those more suited to nature of the work WIPO in the domain of IP, e.g. systems based, network approaches, rights based, equity focussed, capacity assessment, gender lens, social and intellectual capital, public good, normative, complexity approaches, developmental evaluation, etc.

Page 27 of 41

Questions of evaluability should also be addressed systematically as part of programming and work planning stages, rather than once the evaluation has been agreed to and is being planned. Systematic stakeholder analysis can form part of programming and planning of evaluations.

EfficiencyWith respect to the efficiency of the evaluation process, more direct engagement and dialogue with management from a learning perspective would likely reduce the time spent and effort expended in what appears to a be a relatively formal exchange of documentation and correspondence to arrive at agreements at planning and reporting stages.

From the perspective of staff of IAOD the back and forth appears sometimes more like “a pattern of passive resistance” than a genuine effort to craft jointly a process that will provide value added learning. Again this might be due to the perception some management staff have of evaluation as a control and accountability tool, a perception however that may be reinforced by the way evaluation is defined and presented in guidance documentation.

Learning Resource GroupThe Learning Resource Group (LRG) is a kind of reference committee for evaluations intended to “ (…) increase ownership, transparency and learning effect of evaluations”.30 For the LRG ToR that the review examined31 there was majority participation of six WIPO management staff and three external members.

Although the review was not able to assess systematically the extent to which generally the LRG contributed to learning and impact, interviews with management indicated a positive assessment of participation in LRG and its utility for providing a practical understanding of the evaluation function in WIPO.

What appears less clear however, both from an examination of the ToR and from the different perspectives of management on the LRG, is its primary purpose, e.g. whether it’s for validation of the evaluation, expert input, governance, quality assurance, facilitating logistics, or a combination thereof.

As is often the case with reference groups in evaluations for which the purpose might not always be clearly and consistently understood among participants, independence and efficiency of process may become issues.

External consultantsThe use of external consultants by the IAOD evaluation function is systematic in all cases. This practice is understandable in view of the limited staff resources that IAOD may affect to evaluations and their associated requirements of expertise in various domains of IP that IAOD should not be expected reasonably to possess.

30 Country Program evaluation (CPE) – Thailand – TORs Learning Resource Group (LRG)31 Ibid

Page 28 of 41

As well, although external consultants typically fulfil a significant role in evaluations, IAOD through its evaluation function retains full ownership and responsibility for the evaluation32.

The primary role of the external consultant is not however consistently clear, e.g. whether it is for IP content expertise, evaluation expertise, additional human resources, negotiating local arrangements, knowledge of local context, etc.

While each of the above examples may be justified for hiring external resources, it is important to be clear in each evaluation about the primary purpose of the external consultant for reasons of credibility, efficiency and effectiveness of the evaluation and its process.

At present the approach of the evaluation function appears to be to find external consultants who combine both IP content expertise with evaluation expertise, the upshot of which is that the evaluation function sees itself in the position of “tapping into the same limited pool of evaluators”.

The findings of the review, most notably the appreciation of WIPO management that evaluations may lack substantive value as well as of understanding and knowledge of the areas that are being evaluated, together with the positive appreciation of the constructive way in which the evaluation function conducts evaluation, suggests that the primary purpose of the external consultants should be specific IP expertise related to the evaluand.

The evaluation expertise resides with the staff of the evaluation function of IAOD and it would appear less necessary to resort to additional evaluation expertise through external consultants unless there is a clear and explicit reason to do so.

5.3Conclusion

The review concludes that the Evaluation Function contributes to WIPO accountability however could contribute more to learning as expressed in the expectations of WIPO management and in view of the potential that evaluation offers for organisational learning.33

The review further concludes that although there is a system in place for monitoring and following up on evaluation recommendations, there is as yet little in the way of observable or reported impact.

The review concludes that the observed application of a predominant audit frame of reference to oversight may constrain the effectiveness of the IAOD

32 Which is different from some other evaluation functions of public institutions in where the evaluation report is written by the external consultant and considered as such, i.e. not a product of the evaluation function per se but that of the external consultant.33 Please refer to the work of Chris Argyris, Donald Schön, Ron Heifetz, Gregory Bateson and Peter Senge for an understanding of single, double and triple loop learning.

Page 29 of 41

Evaluation Function and may limit its ability to exploit fully opportunities for adding value to WIPO and its stakeholders.

5.4Recommendations

The distinction between audit and evaluation, in which the primary purpose of audit is to provide assurance and that of evaluation to determine value, has critical implications for evaluation at all levels, from organisational to functional to operational.

The distinction is of a fundamental nature such that it has implications for foundational guidance documents of WIPO such as the Oversight Charter and the Evaluation Policy.

As well, it has implications for the way in which the evaluation function is resourced and managed in relation to the other functions of the oversight division so as to promote complementarity and synergy.

Finally, it has implications for programming and conducting evaluations particularly with respect to approaches and methodologies, including quality control and assurance processes.

The review has not acquired knowledge of the organisational context of WIPO sufficient and comprehensive enough to understand and appreciate the extent to which recommendations of a fundamental nature at the organisational level are realistically feasible and actionable.

For this reason the review is not in an informed position to provide practical recommendations at organisational and policy levels consistent with its assessment that the application of a frame of reference more appropriate to evaluation will increase significantly its effectiveness, learning impact and value for WIPO.

The recommendations that follow focus therefore on the operational level recognising that they should assist in improving the evaluation function within the existing organisational and policy frameworks.

IAOD should consider improving the quality and nature of its engagement with WIPO management in line with its expressed desire for more direct and meaningful involvement with the evaluation function and with particular attention to programming of evaluations.

The evaluation function should systematically include evaluability assessments at the programming stage of work planning for evaluations. It should also use stakeholder analysis systematically and when useful as part of programming and planning.

As part of programming and planning processes the evaluation function should focus more systematically on intended users of evaluation by

Page 30 of 41

applying utilisation-focussed approaches and frameworks such as the Utilisation Focussed Checklist of Michael Quinn Patton34.

The evaluation function should identify systematically the bottlenecks in the evaluation process that hinder its efficiency, some of which are noted in the discussion, and take steps to address them such as: more direct and collaborative engagement at the planning stage, an agreed communication plan for a “no surprises” approach, a clear and explicit distinction between findings and conclusions both in terms of process for preparing the report and in its published form.

The evaluation function should consider diversifying its portfolio of evaluations in order to conduct where it judges worthwhile, short cycle evaluations such as real time evaluation.

The evaluation function should revisit the purpose and use of reference groups for evaluation, i.e. such as the Learning Resource Group, to determine and make explicit their primary purpose while avoiding “multi-purpose” mandates that may affect independence of the evaluation or otherwise create confusion as to respective roles and responsibilities of participants. For example a steering committee, the function of which is to govern, i.e. support the overall orientation of, the evaluation, is different from a reference group which acts as a sounding board based on the specialised knowledge it has of the evaluand and its context, both of which are in turn different from a management group which acts as the primary project and contract management interface between the commissioner and the evaluator.

In all cases there should an appropriate understanding and application of the procedures and respective roles required to protect evaluation independence, i.e. respecting the evaluator’s independence in exercising judgment to reach and formulate conclusions without interference, pressure or influence from the LRG or other groups.

The evaluation function should, in each case in which it resorts to external consultants, make clear and explicit the primary purpose of the consultant’s contribution, i.e. whether it is for IP content expertise and/or for another reason. The use of external consultants should be consistent with the recognition that evaluation expertise rests within the evaluation function of IAOD and that IAOD is not, and should not, be expected to possess IP content expertise. It follows that external consultants with IP content expertise do not also need to have evaluation expertise in all cases.

34 http://www.wmich.edu/evalctr/archive_checklists/ufe.pdf

Page 31 of 41

http://www.wmich.edu/evalctr/archive_checklists/ufe.pdf

6. Adequacy of resources

The question that was put to the review is: To what extent are the resources available to the Evaluation Function adequate for it to achieve its mandate?

The review addressed this question from two perspectives, i.e. the quality and quantity of staff resources allocated to the evaluation function.

6.1Findings

The question of quality is addressed in Chapter 3 – Assurance on conformity – 3.1.2 Credibility N9 and is found to be adequate.

On the question of quantity the review based its assessment on available comparative data, including that which is in Annex 3 of the WIPO revised Evaluation Policy, as well as the staff allocation to the evaluation function of comparable oversight services that also include audit such as OECD, i.e. 2 full time staff, and the Council of Europe, i.e. 2 full time staff.

More importantly the review assessed the adequacy of staffing levels to realise the workplan described in the WIPO 2010-2015 Evaluation Strategy:

Independent strategic evaluations: 2 in period of 6 yearsIndependent country level evaluations: 10Thematic: 6Program: 5

6.2Conclusion

Taking into account the model on which the evaluation function of IAOD is based, i.e. leadership and full participation in the evaluation with the additional contribution of contracted external consultants, the specialised competency requirements for evaluators and in light of the work programme that is proposed, the review concludes that in terms of quality IAOD evaluation staff resources are adequate however the quantity of qualified evaluation staff may not be sufficient.

6.3Recommendation

IAOD should monitor its level of allocation of qualified evaluation staff to its evaluation function and the feasibility of its intended workplan, based on a systematic and data based analysis of estimated workload.

Page 32 of 41

7. Adequacy of quality assurance practices

The question that was put to the review is: How adequate is the quality assurance system for the Evaluation Function and in particular the periodicity of reviews?

The review addressed this question primarily from the perspective of principles for quality control and assurance of specific evaluations, consistent with UNEG Norms and Standards for Evaluation. The review considered the periodicity of review as determined in the Evaluation Policy.

7.1Findings

The review did not find in place a quality assurance process consistent with accepted principles. The review did not find explicit quality standards for quality control of evaluation however did note that the 2010-2015 Evaluation Strategy indicates that by end of the strategy, evaluation quality assurance standards will have been developed and that 80% of WIPO evaluations will have applied them.

The review further found that there appears to be a lack of clarity and understanding with respect to the distinction between the supervisory function of the hierarchical superior and quality assurance, e.g. in the case of the country programme evaluation for Thailand it is stated that “The Director, IAOD, will conduct quality assurance”.

The review noted that the Evaluation Policy calls for a review every 3 years.

7.2Conclusion The review concludes that there is not a sufficiently developed quality assurance system for the Evaluation Function of IAOD.

The 3 year periodicity of reviews appears reasonable.

Page 33 of 41

7.3Recommendations

IAOD should give priority to developing and instituting a quality control process for evaluation based on explicit quality standard. Quality control should be the responsibility of the evaluation team leader.

IAOD should develop and institute a quality assurance process using professionally qualified evaluation experts external to IAOD to provide assurance and advice at key points in the evaluation process, e.g. planning based on approach papers, terms of reference and inception reports; analysis and assessment based on findings and preliminary conclusions; final reporting.

Page 34 of 41

8. WIPO-IAOD Evaluation Function Survey

The survey consisted of 10 multiple choice questions and one open question. The link to the online survey was sent to 34 WIPO staff members. It was completed by 14 people, making for a response rate of 41%. Out of the 14 respondents, 11 answered all multiple choice questions. All multiple choice questions used a rating scale from 1 (lowest) to 5 (highest).

8.1Relevance and use of evaluations

The majority of respondents indicated that WIPO evaluations focus on subjects that are most useful to them. 14% of the respondents rated this question with a 5, 64% with a 4 and 21% with a 3. With an average of 3.93, this is the question which received the highest average score.

The responses indicate that WIPO evaluations are perceived at being slightly better at contributing to accountability requirements than at contributing to learning requirements. The average rating given for the evaluations’ contribution to accountability requirements is 3.50, while it is 2.93 for the evaluations’ contribution to learning requirements. Three respondents indicated that the evaluations did not at all contribute to their learning requirements.

There was some disagreement on the extent to which evaluations contribute to establishing best practice. According to 3 respondents this was not at all the case, while for 1 respondent this was fully the case and 3 others rated this question with a 4. With an average score of 2.83, this question is among those that received the lowest average score.

8.2Evaluation process

The extent to which evaluation processes engage with the survey respondents and their stakeholders in ways that make evaluations useful received an average rating of 3.15 with the extremes of the rating scale only being used by one respondent each.

The question to what extent evaluations were conducted a) transparently, b) independently, and c) impartially mostly received average to positive feedback with transparency receiving a lower average score (3.33) than independence (3.64) and impartiality (3.73).

Page 35 of 41

8.3Communication of evaluation findings

Respondents were asked to what extent evaluation findings are communicated to them and their stakeholders in a a) useful, b) constructive and c) timely manner. The responses were on average slightly on the positive side with the “constructive” aspect of communication receiving a higher average score (3.67) than the two other aspects (3.17 for both). For each aspect, one person responded that this was “not at all” the case.

On the question to what extent evaluation findings were communicated in an impartial way with adequate levels of technical and political credibility, most respondents (42%) gave a medium rating (3 points), with the average being slightly lower (2.83).

8.4Quality of evaluations

On the question whether evaluations are technically sound, there was an interesting division between 42% of the respondents who rated this question rather negatively with a 2 and 50% of the respondents who rated it rather positively with a 4. One person gave an average rating (3) to this question, while the extremes of the rating scale (1 and 5) were not used.

None of the respondents had a strong opinion on the extent to which evaluation reports are of high quality as all responses range from 2 to 4 points (average of 3.09).

8.5Additional comments

The opportunity to provide additional comments was only used by two respondents. One of them recommends a better integration of M&E into the process of program planning, further capacity building of staff on M&E and a stronger participation of staff members in evaluation.

The other comment refers to the lack of understanding by IAOD of the respondent’s substantive area of work and of the division of responsibilities therein, while stressing that the overall experience of working with IAOD has otherwise been positive.

8.6Observations

Most ratings are around the medium value, with relatively few strong (negative/positive) opinions. The range of average ratings per question goes from 2.83 to 3.93, while the average of the averages is at 3.29. The responses on the evaluations’ contribution to learning and to the creation of best practice go into the same direction and indeed underline that the evaluations are seen as less useful for knowledge creation or changing practices than for accountability purposes.

Page 36 of 41

9. Additional remarks

Although it is not within the agreed scope of the review to assess evaluation in WIPO outside of the IAOD evaluation function, the review notes that the requirements of programmes for evaluations over their life cycle do not appear to be covered explicitly in WIPO.

As noted by a respondent: “Program and Budget and Program Performance Reporting provide a framework for planning and monitoring. In this context, there is a vacuum on evaluation which is neither budgeted for nor otherwise conceptualized, let alone planned; this vacuum cannot be filled by IAOD’s evaluation function in its current state, nor is it sure whether it should be”.

To the extent that evaluations in WIPO may be commissioned and conducted outside the remit of IAOD, there should be a clear and explicit WIPO policy covering non-IAOD evaluations.

Such a policy should address conformity of all WIPO evaluations with UNEG Norms and Standards, including inter alia those related to HRGE, quality control and assurance processes, competencies of commissioners and evaluators, communication and dissemination, as well as relationship and coordination with IAOD.

Page 37 of 41

10. Summary of ratings

IAOD conformity with UNEG Norms ConformsPartially conforms

Does not conform

Independence

N2.3: The governing bodies/heads of organizations are responsible for ensuring that adequate resources are allocated to enable the evaluation function to operate effectively and with due independence.

N.5.1: Impartiality is the absence of bias in due process, methodological rigor, consideration and presentation of achievements and challenges. The requirement for impartiality exists at all points in the process: planning, conduct, reporting

N6.1: The evaluation function has to be located independently from the other management functions so that it is free from undue influence and that unbiased and transparent reporting is ensured. It needs to have full discretion in submitting directly its reports for consideration at the appropriate level of decision-making pertaining to the subject of evaluation.

N6.2: The head of evaluation must have the independence to supervise and report on evaluations.

N6.3: To avoid conflict of interest and undue pressure, evaluators need to be independent, implying that members of an evaluation team must not have been directly responsible for the policy-setting, design, or overall management of the subject of evaluation, nor expect to be in the near future.

N6.4: Evaluators must have no vested interest and have the full freedom to conduct impartially their evaluative work, without potential negative effects on their career development. They must be able to express their opinion in a free manner.

Page 38 of 41

Credibility

N4.2: The evaluation plan can be the result of a cyclical or purposive selection of evaluation topics. The purpose, nature and scope of evaluation must be clear to evaluators and stakeholders. The plan for conducting each evaluation must ensure due process to ascertain the timely completion of the mandate, and consideration of the most cost-effective way to obtain and analyze the necessary information.

N5.2: Impartiality increases the credibility of evaluation and reduces the bias in the data gathering, analysis, findings, conclusions and recommendations. Impartiality provides legitimacy to evaluation and reduces the potential for conflict of interest.

N8.1: Each evaluation should employ design, planning and implementation processes that are inherently quality oriented

N9: The head of the evaluation function, evaluation staff and evaluators should have proven competencies to manage and/or conduct evaluations

N10.1: Transparency and consultation with the major stakeholders are essential features in all stages of the evaluation process. This improves the credibility and quality of the evaluation. It can facilitate consensus building and ownership of the findings, conclusions and recommendations.

N11: Evaluators must have personal and professional integrity, respect people and their rights, confidentiality of data and sources, and be sensitive to beliefs as well as issues of gender and human rights

N8.2: Evaluation reports must present in a complete and balanced way the evidence, findings, conclusions and recommendations. They must be brief and to the point and easy to understand. They must explain the methodology followed, highlight the methodological limitations of

Page 39 of 41

the evaluation, key concerns and evidenced-based findings, dissident views and consequent conclusions, recommendations and lessons. They must have an executive summary that encapsulates the essence of the information contained in the report, and facilitate dissemination and distillation of lessons.

N10.2: documentation on evaluations in easily consultable and readable form should also contribute to both transparency and legitimacy

Utility

13.1 Evaluation contributes to knowledge building and organizational improvement. Evaluations should be conducted and evaluation findings and recommendations presented in a manner that is easily understood by target audiences.

N2.6: The Governing Bodies and/or Heads of organizations and of the evaluation functions are responsible for ensuring that evaluation contributes to decision making and management. They should ensure that a system is in place for explicit planning for evaluation and for systematic consideration of the findings, conclusions and recommendations contained in evaluations. They should ensure appropriate follow-up measures including an action plan, or equivalent appropriate tools, with clear accountability for the implementation of the approved recommendations.

N4.1 Proper application of the evaluation function implies that there is a clear intent to use evaluation findings. In the context of limited resources, the planning and selection of evaluation work has to be carefully done. Evaluations must be chosen and undertaken in a timely manner so that they can and do inform decision-making with relevant and timely information. Planning for evaluation must be an explicit part of planning and budgeting of the evaluation function and/or the organization as a whole. Annual or multi-year evaluation work programmes

Page 40 of 41

should be made public.

N4.2 The evaluation plan can be the result of a cyclical or purposive selection of evaluation topics. The purpose, nature and scope of evaluation must be clear to evaluators and stakeholders. The plan for conducting each evaluation must ensure due process to ascertain the timely completion of the mandate, and consideration of the most cost-effective way to obtain and analyze the necessary information.

N2.7: governing bodies and/or heads of organizations and of the evaluation function are responsible for ensuring that there is a repository of evaluation and a mechanism for stilling and disseminating lessons to improve organizational learning and systemic improvement

Page 41 of 41

Documents

External quality assessment of IAOD evaluation function - Final Report 29-06