ENAPEscola Nacional de Administração Pública

“Partnership for Public Sector Management Excellence” Project(2003-2007)

Final Evaluation Report

October 2007

A Cooperation Project on Technology Transfer between Escola Nacional de Administração Pública and the Canada School of Public Service (CSPS)

Supported by the Canadian International Development Agency (CIDA)

Canada-Brazil Technology Transfer Fund

ENAP

Final Evaluation Report

October 2007

Partnership for Public Sector Management Excellence (2003-2007)

A Cooperation Project on Technology Transfer between Escola Nacional de Administração Pública (ENAP) and the Canada School of Public Service (CSPS)

Supported by the Canadian International Development Agency (CIDA)

Canada-Brazil Technology Transfer Fund – Phase II

1

2

Table of Contents Executive Summary ……………………………………………………….….. 03 1. Introduction ……………………………………………………………….... 11 Purpose, Content and Structure of the report Key Audience 2. Project Background and objectives …………………………………….. 12 Brief historical overview Project objectives – goal and purpose Project results – impact, outcomes and outputs Main activities Project Management Financial Resources 3. Evaluation Process ……………………………………………………….. 16 Evaluation Preparation Process Evaluation process 4. Evaluation Findings ……………………………………………………...... 18 Program Effectiveness Program Efficiency Cross-cutting Themes – Gender Equality and Diversity Sustainability Conclusions …………………………………………………….……………... 37 Lessons Learned and Good Practices …………………………………… 39 Recommendations …………………………………………………………… 42 Future perspectives ………………………………………………………….. 43 Annexes

1. Terms of reference 2. Evaluation Issues Matrix 3. Schedule and List of Interviews in Brazil and Canada 4. List of Documents Reviewed 5. Presentation to the Project’s Final Meeting

3

Executive Summary 1. Introduction The project “A Partnership for Public Sector Management Excellence” is a technology transfer cooperation project between the Escola Nacional de Administração Pública (ENAP) and the Canada School of Public Service (CSPS) carried on between March 2003 and September 2007. Aiming to improve Brazilian Federal Public management capacity to achieve greater equity in Brazil, the project focused on strengthening ENAP’s capacity to act as a leading institution for human resource development at public service in Brazil. As final results, the project purposed:

Improved capacity to better manage governmental programs and a change agenda systematically promoted; Innovative learning programs designed and delivered for executive and

managers; ENAP’s capacity enhanced for the conception and delivery of products and

services in management development through CSPS’ technology; Local partners provided with better learning management programs and

federal executives and managers in the States trained and developed in leading public management topics.

This report presents the evaluation results of the project, addressing the following main issues:

Achieved Results: o ENAP institutional development and empowerment o Brazilian executives’ capacity-development o Strengthening of National and International Partnership o Unexpected Results

Efficiency Gender and diversity Sustainability

Information collection activities comprised a document review, individual and group interviews at ENAP office, in Brasília, on site visits, interviews held by telephone and interviews held in Canada. A total of 16 interviews (including 5 focus group discussions) were held in Brazil and 4 in Canada. At the total, 26 persons were directly interviewed and a set of 20 documents analysed. 2. Achievement of Results Notable progress towards expected results has been achieved and remarkable unexpected results have emerged in different components of the project. Regarding ENAP Institutional Development and Empowerment, the project was effective in:

4

New competencies developed in ENAP Staff

The evaluation indicates a strong acquirement of interpersonal competencies by ENAP staff in different directorates. The main perceived changes are related to following abilities: openness for accepting new methodologies and learning technologies for designing and delivering learning events; proactive attitude in face of demands and problems; results based management knowledge; negotiation skills; self confidence; ability to work in teams and team building; and awareness of diversity issues.

Capacity to adapt and deliver new courses according to the Brazilian reality

ENAP instructors and technicians have been strongly benefited by the experience of adapting new courses. During the project, ENAP adapted and delivered 4 CSPS courses on the issues of leadership, development of public policies, integrated resources management and managing human performance. Part of one of the courses was incorporated into the schedule of other regular and tailor-made ENAP courses. One distance learning course was also adapted and has already been delivered to more than 1,200 Brazilian civil servants.

Acquisition and appropriation of new methodologies

New methodologies for increasing dialogue in learning events, such as the Open Space Technology and the Armchair Discussions, brought by the contact with CSPS, have been incorporated in ENAP’s activities. The same happened to new research approaches, as the Action-Research Roundtable, applied twice by ENAP. The adapted courses also brought important new perspectives for learning methodologies. The adoption of new methodologies is described as a strategic shift of ENAP, which was already a decision by ENAP’s presidency and was strongly enhanced by the project.

Increased capacity to attract executives

There is an increasing presence of top level leaders in ENAP activities, due to the focus on leadership courses, brought by the project. Demands for these courses are consistently coming in from federal agencies which are asking for special classes, closed and delivered only for their senior level executives. The Evaluation Reports of these courses indicate a very high level of satisfaction with both the contents and the methodological approaches.

Enhanced recognition of ENAP as a space for developing public executives

The positive image of ENAP perceived by the federal administration and partner schools as a locus for learning and exchanging ideas has been enhanced by the project. ENAP developed strategies to meet customized demands through the association of its know-how to the customer’s needs, as well as to create an appropriate environment for learning and knowledge sharing among peers. Dynamism, proactive approach, innovation, and generosity (culture of sharing knowledge) are some highlighted characteristics enhanced by the Project.

Contributions to the National Policy for Human Resource Development

5

The cooperation with Canada improved ENAP's qualifications in the area of competence-based management, which today is key for the National Policy for Human Resource Development (stated in February 2006 by the Presidential Decree nb. 5,707), ensuring a privileged position for ENAP to discuss training policies.

Regarding executives’ capacity development, the project achieved significant progress. From March 2003 to July 2007, learning events of the project reached 1,070 Brazilian federal executives (of which almost 70% are women) in Brazil and in foreign missions. Impact evaluation reports indicate that executives reached by the courses are acquiring new competencies for leadership, specifically:

Focusing on the client (ability to better listen to and understand the team and client demands). Interpersonal relationships (ability to listen to the team, provide coaching,

provide and receive adequate feedback, self-evaluation and team evaluation). Ability to deal with changes (adaptability to change processes and transitions

in the government). There are indications that these competencies are promoting benefits in the workplace, especially regarding leader-team relationship and inter-ministerial practices. Many activities of the project catalyzed contacts and linkages between Brazilian and Canadian executives, both in Brazil and in Canada. ENAP had an important role in keeping these executives in contact. Notwithstanding progress made, evaluation interviews suggested that some of these linkages are hard to sustain. Most of the executives that have been in Canada and are not from ENAP staff, have not kept the links with foreign experts / executives. The project had also as medium term result the purpose of strengthening partnerships and networks in both national and international level. In this aspect, the project enhanced ENAP links with 11 schools of government of its Partnership Program and 95 institutions of the National Network of Schools of Government in Brazil, providing increasing opportunities for learning and exchanging new contents and learning methodologies and technologies. It reinforced ENAP's role to act more and more based on the logic of supporting the institutional development of these organizations. Selected five partner schools, covering a strategic geographical distribution, are enthusiastically taking part of the planning of a new ENAP-CSPS cooperation project. International partnerships were also enhanced, mainly by two international forums held at ENAP during the project, which were attended by 145 representatives from ENAP and CSPS partner schools and from 13 countries, besides Brazil and Canada: Germany, Argentina, China, Spain, France, Guatemala, Portugal, Kenya, Dominican Republic, Russia, Sweden, Ukraine, and Panama. Unintended results were also expressive. Main achievements are:

Strengthening of the Distance-Learning sector at ENAP, with considerable gains both to the cooperation ENAP-CSPS and to the learning programmes in Brazil.

6

The incorporation of new methodologies and contents in the career training for Specialists in Public Policy and Governmental Management and in the specializations in public management (latu sensu)., Integration among different ENAP teams and units.

3. Program Efficiency Efficiency in overall management of the project, good monitoring and evaluation tools and good communication between Canadian and Brazilian management teams are landmarks of this project. Great value for money was noted by ENAP and CSPS executives as a characteristic of the project, which is perfectly supported by the annual reports. A considerable set of new activities and an extension of the project length were done within the original budget. Systematization of information is also a landmark in the project. Regular registration of course adaptation processes facilitated their use by other ENAP units and partner schools. Standardized tools for planning and reporting missions and learning events in Canada and Brazil facilitated the evaluation of these activities and provided comparable information for discussing action plans. Visits of foreign experts were also carefully prepared and vital information on the Brazilian public service was always previously provided. This seems to be crucial for making the intervention of foreign instructors more effective in the local context. 4. Gender and Diversity Sensitization for gender equality and diversity issues may be recognized as an emerging competency to some members of the ENAP team, as in the adaptation of e-learning courses and in the delivery of modules of indigenous education and African-Brazilian health. This may be related to project’s specific activities, such as a research about gender and race in Brazilian public service and a specific sensitization workshop. Nonetheless, the Canadian attitudes and actions in addressing diversity were more effective in terms of sensitization for these issues than a specific workshop activity. The integration of gender and diversity issues in ENAP’s pedagogic project and in courses’ schedules are still challenges to further projects. Some stakeholders suggested that more attention should be paid to already existing Brazilian information/ knowledge on this issue. 5. Sustainability The collected data indicate ENAP is perfectly enabled to continue delivering the adapted courses, as well as sharing them with a network of partner schools. All the adapted courses have been integrated into ENAP’s catalogue and are being delivered in a regular basis. The main challenge now is how to share the courses with partner schools, as the courses require very specific structures (instructional material and logistics) which are not easily available elsewhere. During the project an increased demand for ENAP’s courses is perceived, as well as a sustained shift towards a focus on leaders and executives.

7

On the other hand, the enhancement of e-learning is contributing to a significant enlargement of ENAP’s reach – both numerically (from 6,000 students reached by e-learning in 2003 to estimated 17,000 in 2007), as regionally (from all Brazilian regions). This is perceived, inside and outside ENAP, as important step toward democratization of knowledge. 6. Conclusion and Recommendations Project outcomes Overall, the outcomes of the project were consistently achieved:

Improved capacity to better manage governmental programs and a change agenda systematically promoted. Impact evaluation of learning events indicates that this has been achieved by Brazilian executives, to some extent. However, there is still a great demand for training in leadership competencies, as the interviews have shown. The incorporation of methodologies as the Action-Research Roundtable, promoted important shifts towards research and innovation.

New learning programs designed and delivered for executive and managers.

This result was successfully achieved. Under the Project, since 2003, 31 learning events were conducted at ENAP, 4 lectures were delivered at partner schools, 20 foreign missions went to Canada, 4 CSPS courses were adapted by ENAP and are being regularly delivered for Brazilian executives and managers (760 civil servants until August 2007) and one CSPS e-learning course was reformulated by ENAP and delivered for more than 1,200 civil servants.

Enhanced ENAP’s capacity for the design and delivery of management

development learning products and services through Canadian Technology. Evaluation shows this is also a fully achieved result for the project.

Local partners provided with new management learning programs for federal

executives and managers in the States for their training and development in leading public management topics. The project provided important contact of partner schools with new contents, methodologies and technologies in public management topics, mainly by learning events at ENAP and lectures with Canadian experts in the partner schools. Instructional materials were also made available for local partners. The adaptation of new courses to specific demands of partner schools, however, is yet a challenge to the next cooperation project.

Lessons Learned and Good Practices Project Management

The project was marked by excellent relations/interaction between ENAP and CSPS management teams in Brazil and in Canada, largely as a result of a clear definition of responsibilities, clear and ongoing communications between

8

the parties, a management approach focused on optimizing outcomes and the joint definition of mechanisms to solve problems. Ongoing and swift interaction with the CIDA Office in Brasília and with the

Brazilian Cooperation Agency – ABC. Focusing on building relations among the key actors is fundamental for

ensuring their involvement in the activities from the outset. The need to define demands accurately – preparation of terms of reference

and making the guest experts aware of the context of ENAP and of the Brazilian federal public administration systematically, with the aim of delivering quality products that are in tune with the Brazilian reality. The outcomes anticipated in the project should be seen as part of the

responsibilities of the partners, making sure that they are included in their planning. Such projects would benefit greatly from a Steering Committee, in particular

the new coming project with many and new stakeholders involved. Among other responsibilities, the committee could help to ensure that the transfer of expertise is leading the expected outcomes, and to address challenges as they emerge. The gender and diversity strategies should be defined with the support from

key actors and from an external consultant. The evaluation component should be designed and have its strategies defined

with the support from an external consultant, who should be subsequently involved in building evaluation tools, monitoring activities and drawing up evaluation reports. Flexibility on the part of the partner in Canada, which has been ensuring a

sound technical-financial management of the activities, such as, for example, in the acquisition of more books, in the translation of documents and texts, in the publication of more titles, and in the negotiation of the copyrights of the courses “Leadership: Reflection and Action” and “Leading Transitions.” Flexibility on the part of the CIDA Brazil office to solve problems faced to cover

expenses in Brazil (simultaneous translation, translation of written materials, and fees of consultants/researchers). Flexibility on the part of the Canadian Embassy in Brasilia that agreed to make

the payments related to the execution of activities in Brazil. It favoured a more agile and safe project implementation.

Learning

Practical learning: • The contact with the CSPS experience in building leadership in the

public service (through visits and participation in courses) led to new possibilities in terms of learning contents and methodological approaches for ENAP teams.

• Many interviewees praised the appropriateness of the topics addressed in the courses and their format, highlighting the sensitization to “what being a leader is all about, the opportunity to exchange ideas with Brazilian and Canadian peers, the importance of fostering horizontal relations to mobilize teams around outcomes” as gains.

• As regards the relations between ENAP and CSPS teams, they mentioned the “confidence, clarity and professionalism established

9

between them and the possibility of exchanging ideas and learning jointly” as highly positive factors.

Multiplying strategy:

• Recruiting high level executives and managers to take part in the training activities favors a multiplication of the learning within the teams and the possibility of training new team members (recommendation from superiors).

• Making available more places in courses/events held in Brazil to participants from the same institution to favor the introduction of new post-training practices and changes in their organizations (so as to enhance their influence/impact of changes).

Awareness of differences between the public administrations of Canada and

Brazil: • It is important to prepare foreign and Brazilian executives for

missions/visits by providing them with contextualizing and supporting documents/texts for them to understand how the public administrations and institutions of both countries (specificities and differences) operate and to adjust the development of products and services to the needs and practices of the beneficiaries.

Strategy for knowledge acquisition and sharing:

• It is important to always send two ENAP civil servants to attend courses at CSPS, particularly for processes of adapting courses, which require more intense and longer activities. This is a strategy that favors the retention and dissemination of knowledge and a greater integration between ENAP's learning staff and research teams.

Recommendations

Some alerts were done regarding the regional diversity: in the next project, attention should be given to preparing trainers and instructors to deal with a larger diversity / plurality of regional contexts. New CSPS courses were suggested for further adaptation, in particular those

related to leadership development and human performance management: “Leading for Results”, “The Courage to Lead in the Public Service” and “The Art of Coaching”. More time is required for the visits of Brazilians to Canada, in order to improve

the knowledge sharing and absorption. In particular, more time would be required for practical applications of information shared. As suggested by one of the interviewees of CSPS, more time would be required for “job shadowing”, which could be quite useful in this kind of initiative. The dynamic of holding an international activity requires complex preparations

which involve the intense participation of the teams of both schools in the design of actions and adaptations to the needs of the Brazilian public administration. Therefore, in order to make the best use possible of the teams, it is necessary to avoid any agenda overlapping among ENAP's different units. The calendar of ENAP's international actions should continue to integrate the

strategic planning process of the technical teams, so as to add value to the

10

activities being developed by them and to avoid unforeseen activities or undue burdens. ENAP should be better prepared to take on administrative activities which are

also being carried out by the technical teams, so that they may focus exclusively on using new knowledge and methodologies. Contacts with Canadian experts: the program of short visits of Brazilians

executives to Canada and of Canadian executives to Brazil clearly revealed the need to refine the program by identifying areas of common interest more precisely, anticipating potential further developments. Multiplying effect: the strategy of recruiting participants to attend courses at

ENAP was based on selecting high hierarchical level executives and managers who can influence, mobilize and recommend the same training to other executives or to members of their teams. This strategy was correct and should be continued and strengthened, with the aim of improving cost-benefit aspects.

11

1. Introduction Purpose, Content and Structure of the report This report aims to present the main findings of a results evaluation of the Project “A Partnership for Public Sector Management Excellence” – technology transfer cooperation between Escola Nacional de Administração Pública (ENAP – National School of Public Administration) and the Canada School of Public Service (CSPS). The report has 4 main sections. Firstly it presents the historical background of the project, describing the objectives, expected outcomes and outputs purposed by the project and summarizing the main activities developed during the past four years. Next, the report describes the evaluation process and the methodology designed for the assessment of results. The following section presents the main findings of the evaluation, describing achieved results in different project components. Last, it presents overarching conclusions, lessons learned, recommendations and future perspectives for the partnership between the stakeholders. Key Audience The results evaluation is both a form of giving account of the project achievements to Canadian and Brazilian governments – through their cooperation agencies – and providing useful information for planning further projects on the topics addressed by this report. So, the key audience of this text is composed by executives and management staff of the Canadian International Development Agency (CIDA) and the Agência Brasileira de Cooperação (ABC), as well as by CSPS’ and ENAP’s management team.

12

2. Project Background and objectives Brief historical overview In March 2003, a Memorandum of Understanding was signed between ENAP and CSPS (former CCMD) starting a four-year Project “A Partnership for Public Sector Management Excellence – a technology transfer cooperation between Escola Nacional de Administração Pública (ENAP – National School of Public Administration) and the Canada School of Public Service (CSPS)”. The project was collaboratively planned, assessed and designed by ENAP and CCMD to act as a vehicle for the transfer of CCMD’S unique expertise in the field of public sector learning and its significant role in public sector renewal, and to create enduring partnerships between learning professionals in Brazil and Canada in order to provide mutual benefits. The project was not affected by CCMD’s amalgamation with two other agencies to become the CSPS. The Canadian International Development Agency (CIDA) has made a major commitment to this project through its Technology Transfer Cooperation Program. Project objectives – goal and purpose The goal of the Project is to increase management competencies of Brazilian federal executives and managers by strengthening ENAP as a centre of excellence in public management through the transfer of Canadian technology. Project results – impact, outcomes and outputs Based on the Canadian expertise, the expected impacts, outcomes and outputs of the project are: Impacts:

Brazilian Public service more agile, efficient, result-oriented, citizen-focus and committed to achieving greater equity. Stronger linkages and relationships developed between the Brazilian and

Canadian Governments Outcomes:

Improved capacity to better manage governmental programs and a change agenda systematically promoted. New learning programs designed and delivered for executive and managers. Enhanced capacity in the design and delivery of management development

learning products and services at ENAP through Canadian Technology. Local partners provided with new management learning programs for federal

executives and managers in the States for their training and development in leading public management topics.

13

Outputs:

CSPS action learning methodology adapted and incorporated into ENAP´s new or regular management development programs offered to public servants and to ENAP’s Partner Institutions. Expertise enhanced at ENAP to design and implement new learning events

based on CSPS’s experience / technology. Content and methodology of CSPS courses transferred to ENAP. Learning Technicians of the Brazilian Federal Government have developed

new competencies. CSPS contents/technologies adapted and incorporated into ENAP’s new or

regular courses delivered to public servants and ENAP’s Partner Schools. Partnership reinforced with state schools. Federal public executives and managers have developed new competencies. CSPS’s research roundtable methodology integrated and adapted by ENAP. Mechanisms for ongoing exchanges of information / publications / research

between ENAP and CSPS established. CSPS information on networks of research experts and public executives

made available to Brazilian public sector through ENAP. Linkages between ENAP’s and CSPS’s libraries established. CSPS documents and information on leadership and public management

made available in Portuguese to ENAP, Partner Institutions and Brazilian public servants. Learning material and documents based on Canadian technology made

available to ENAP, Partner Institutions and federal public executives and managers. Relationship between Canadian and Brazilian senior executives established. Best practices in public administration from both countries shared and

disseminated. Mutual learning on activities in the field of public sector reforms shared. Strengthened cooperative relationships in international activities. A global perspective on public sector reform provided to participants. ENAP and CSPS positioned as centers of excellence in public management. Best practices exchanged in the International Seminar published and

disseminated for Brazilian and Canadian public managers. Network of ENAP’s and CSPS’s partners expanded.

Main activities The project has performed the following activities during the past 4 years: 2003

5 learning events – courses, workshops, lectures – were held at ENAP, delivered by CSPS experts and attended by 169 Brazilian civil servants. 70 civil servants attended one lecture delivered by CSPS expert at a Brazilian

Partner School in Recife, Pernambuco (Northeast Region). 06 civil servants from ENAP and 01 executive from the Ministry of Justice

attended courses and learning events at CSPS/Ottawa.

14

2004 3 learning events – followed by meetings and working sessions – were held at

ENAP, which were delivered by CSPS experts and attended by 128 Brazilian civil servants. 55 civil servants attended one lecture delivered by CSPS experts at a

Brazilian Partner School in Rio de Janeiro (Southeast Region). 06 ENAP officials and 01 executive of the Presidency of the Republic attended

courses and learning events at CSPS/Ottawa. 2005

4 learning events – followed by meetings and working sessions – were held at ENAP (totaling 60 hours), which were attended by 279 Brazilian civil servants. 41 civil servants attended one lecture delivered by CSPS experts at a

Brazilian Partner School in Belém – State of Pará (North Region). 05 ENAP civil servants, 01 executive from the Special Secretariat on Policies

for the Promotion of Racial Equality – SEPPIR and 01 executive from the Ministry of Social Development attended courses and learning events at CSPS/Ottawa. Action-Research Roundtable on “Competency-Based Management”, with

publication of a book. Researches: “Gender and Race/Ethnic group: A Matter of Competencies?”

and “Leadership Competencies in Public Service and Gender Perspective” 2006

5 learning events – followed by meetings and working sessions – were held at ENAP (totalling 63 hours), which were attended by 167 Brazilian civil servants. 36 civil servants attended one lecture delivered by CSPS experts at a

Brazilian Partner School in Curitiba – State of Paraná (South Region). 04 ENAP civil servants participated of technical missions to CSPS/Ottawa and

the Netherlands (Global Engagement Network – GEN Program). Brazil-Canada International Open Forum “To Learn, To Innovate, To

Cooperate: a conversation among peers”. Action-Research Roundtable on “Distance Learning”, with publication of a

book. 2007

2 learning events were held at ENAP (totalling 31 class hours), which were attended by 89 Brazilian civil servants. 09 ENAP civil servants and 02 executives from the Ministry of Planning,

Budget and Management attended courses and learning events at CSPS/Ottawa.

Project Management Management teams at ENAP and CSPS have been coordinating the Project activities during the 4-year project. The activities and results are officially presented during the General Annual Meetings attended by the Canadian and Brazilian Project Managers and representatives from CIDA and ABC.

15

Financial Resources In order to meet the mandate of this project, a request for CND$ 1,635,068 in funding has been made to CIDA, to help offset the expected total project costs of CND$ 5,182,041. CSPS was expected to contribute CND$ 2,019,350 (from which CND$ 1,200,000 of CSPS Intellectual Property), while ENAP was planning a financial outlay of CND$1,527,623 over the three year duration of the project. The current Project finished on September 30th 2007. Follow-up activities In October 2006, in an effort to maintain the established partnership and disseminate the knowledge sharing in Brazil, ENAP and CSPS submitted to CIDA (under the Brazil-Canada Knowledge Exchange for Equity Promotion Program – KEEP) a new project proposal called “Governance Capacity-Building”. The Project was recommended for approval by CIDA and Agência Brasileira de Cooperação (ABC) and it will be implemented from 2007 to 2009, if it is approved by the Canadian Minister of International Cooperation.

16

3. Evaluation Process Evaluation Preparation Process The evaluation process was prepared jointly by ENAP, CSPS and the Groupe-conseil Baastel Itée, and proposed the following steps:

Document and literature review. Review and adjust the Evaluation Issues (Annex 2); Development for each issue the proper questions, data sources and data

collection methods; Development of a Data collection Matrix; Development of interview protocols; Development of data collection tools; Interviews with key stakeholders in Brazil and Canada.

Evaluation process The data collection was made by the local consultant, Daniel Schroeter Simião, supported by ENAP’s Project Management team, through on site visits, interviews and focus group discussion at ENAP office, in Brasília, or by telephone. Some interviews were conducted in Canada by the consultant Evan Green – Groupe-conseil Baastel Itée. A total of 16 interviews (including 5 focus group discussions) were held in Brazil and 4 in Canada. At the total, 26 persons were directly interviewed. The interviewees were composed of the following groups:

1. ENAP Staff a. Strategic level b. Technical level: coordinators and technicians c. Facilitators / instructors

2. Executives and managers in federal government directly reached by the project

3. Partners (partner school, CIDA) 4. CSPS staff

a. Project management team b. Learning specialists / directors

The list of the persons interviewed and their positions is found in Annex 3. All the interviews in Brazil were tape recorded, transcribed and analysed. Besides the interviews, a set of documents was taken into account (see the complete list in Annex 4), namely: Project document (including the logical framework) Annual reports Activity reports (including impact and reaction assessment surveys) Publications resulted from the project (books and Cadernos ENAP)

17

This report was prepared by the local consultant and reviewed by the Canadian consultant Evan Green – Groupe-conseil Baastel Itée and ENAP’s Project Management Team. Evaluation Limits and Constraints The original project had not been prepared with the idea that it would be evaluated by its outcomes and therefore it does not allow for an ex-ante / ex-post evaluation. So the solution agreed between ENAP and CSPS – and approved by CIDA – was to define an evaluation matrix with issues and questions allowing for one to focus on perceived changes, based on the perspective of different actors and crossing different sources. A strong constraint was the short time available (two months) for extensive interview coverage and for in-depth analysis of the considerable amount of information produced by the project. This was compensated for by some in-depth interviews with key stakeholders and with the important support received by management team.

18

4. Evaluation Findings Program Effectiveness 1. ENAP Institutional Development and Empowerment 1.1 New competencies developed in ENAP Staff The evaluation indicates a strong acquirement of competencies by ENAP staff. Every staff member interviewed, in three different directorates of ENAP (Directorate for Managerial Development, Directorate for Professional Training and Directorate for Communications and Research), has stressed important shifts in interpersonal competencies. The main perceived changes are related to ability for innovating, assertiveness, self-knowledge and self-esteem. It is noteworthy that different members of ENAP staff, with different participation in project activities, have reported similar skills and attitudes developed concerning new competencies acquired (especially in areas like improved attention paid to hearing colleagues and learning). This may indicate good practices / tools for sharing information and experiences among ENAP teams. The following competencies were frequently referred to in the interviews:

- Openness for accepting new methodologies and learning technologies for developing learning events: the team feels qualified to think and develop ENAP’s products and services using innovative approaches;

- Proactive attitude: the team analyzes the demands and problems and responds to them in a swift, professional, and objective manner;

- Results based management knowledge: clear goals, clearly allocated resources, clearly defined deadlines, analysis of risks and actors;

- Negotiation skills: recognition of cultural differences, concertation (consultation/dialogue), consensus-building

- Self-confidence: recognition of existing knowledge and competencies and of learning and exchange opportunities with partners

- Ability to work in teams and team building: recognition of the complementariness of capacities and integrated mobilization of talents, with the aim of ensuring the best performance possible to the team and achieving actual outcomes;

- Awareness of diversity issues. During focus group discussions, a clear generational gap was perceived however, in the analysis of limitations and potencial of developing new competencies. In particular, the younger stakeholders from among the staff seemed to be more enthusiastically involved. On the question of openness for new methodologies, it is noteworthy that enthusiasm or innovation is becoming pervasive. As a programme coordinator said:

19

“The contact with the Project is encouraging us to the idea of: ‘yes, we do need to do that in a different way’! When we were reviewing, for instance, the (training) Programme for Development of Operational Managers and Supervisors (Desenvolvimento de Gerentes Operacionais e Supervisores – DGOS), discussing our experience with the partner schools, the example of the project encouraged the instructors to think about new learning alternatives; I am saying this to them: ‘let’s go and go out, go out from the classroom space, indeed’.”

1.2 Capacity to adapt and deliver new courses according to the Brazilian reality During the project, ENAP adapted and delivered 4 CSPS courses on the issues of leadership, development of public policies, integrated resources management and managing human performance. Until August 2007, these courses have formed 23 classes, reaching 760 Brazilian civil servants, as follows: Course Number of classes

formed since 2003 Total of public servants reached

Challenges of Public Policy Development: an overview

9 484

Leadership, Reflection and Action (LRA)

8

162

Integrated Resource Management

2

43

Managing Human Performance (MHP)

4

71

TOTAL

23

760

Part of the “Challenges of Public Policy Development: An Overview” course was incorporated into the schedule of the following ENAP courses, under the name “Workshop on Public Policy development”:

- Career training for Specialists in Public Policies and Governmental Management (3 induction courses and 1 continuous development course for career progression),

- 3 Specialization courses in Public Management, - 2 tailor-made Specialization Courses for the Ministry of Education and

Ministry of Health Improved capacity to adapt and deliver leadership courses was also highlighted as one of the abilities learned by the staff during the project. According to the Director for Managerial Development:

20

“The main gain afforded by the cooperation with CSPS, with CIDA's support, was to enable ENAP to build the capacity to develop contents and new methodologies for executives. Besides bringing these contents to Brazil and adapting them successfully with the support from Canadian professors, we also learned how to best adapt them and their methodology.”

This ability has also resulted in the adaptation of an e-learning course (“Passage to E-learning”) already delivered to more than 1,200 Brazilian civil servants. The quality of the adaptation of courses on leadership – namely Leadership Reflection and Action (LRA) and Managing Human Performance (MHP) may attest to the excellent level of satisfaction as recorded in the efficacy evaluation reports of the courses, which was also confirmed by the instructors. Although adaptation is successfully evaluated, some issues regarding the adaptation of specific courses may still require attention. For example, the Integrated Resources Management (IRM) course is perceived as being ‘too Canadian’, by some ENAP management level. 1.3 Acquisition and appropriation of new methodologies During the project the following new methodologies were consistently appropriated by ENAP: Open Space Technology (OST): Brought by the Canadian experts for the Brazil-Canada International Open Forum “To Learn, To Innovate and To Cooperate: a conversation among peers”, some aspects of this OST methodology was then appropriated and utilized by a Brazilian consultant, Heloisa Nogueira, in further workshops at ENAP. According to her:

“OST showed me there are good and powerful methodologies for dialogue – this was an old concern of mine. After that, I have used some aspects of it in every event I facilitated at ENAP – it is a simple way for making people talk to another in an intelligent way”

Action-Research Roundtable (ARRT): Two ARRTs were implemented by ENAP during the project. After a good experience with an ARRT on “Competency-based management”, the ENAP team decided to implement another one on “E-learning in Public Organizations”. These resulted in two books published by ENAP entitled: “Gestão por Competências em Organizações de Governo” (Competence-based management in Governmental Organizations) and “Educação a Distância em Organizações Públicas” (Distance Learning in Public Organizations). These books are available for download at ENAP internet site. From March to August 2007, there were 2,020 downloads of the two books (at an average of 336 downloads per month – 198 of the competence-based management book and 138 of the e-learning one), which indicates they are of great interest of Brazilian executives.

21

Coffee with Debate (adapted from Armchair Discussions): Inspired by the CSPS Armchair Discussions, this has become a regular activity at ENAP; it is performed early in the morning and attracts executives of federal offices to the School for sharing information and discussing issues of the governmental agenda in a casual environment. Since 2003, 26 meetings have occurred, reaching more than 1,000 public servants. New learning technologies: New approaches on instructional design have been used both in e-learning courses and traditional course programmes. The adoption of new methodologies is described as a strategic shift of ENAP; This is not only due to the project with CSPS, however. Many people interviewed reinforced that this shift was already a decision made by ENAP’s presidency prior to the project, which was strongly enhanced by the Project. 1.4 Increased capacity to attract executives The adapted courses brought a new audience (executives) to ENAP by combining state-of-the-art contents, innovative methodologies and renowned experts in a space for peers to learn and exchange ideas. One of the most successful indicators for this is a new kind of demand coming from some agencies, who are asking for special classes: closed courses and delivered only for their senior level executives. This was very recently done with the following courses: Leadership, Reflection and Action:

- 4 classes already held in the Program for Managerial Development demanded by the Agência Nacional de Vigilância Sanitária – ANVISA (Brazilian Health Surveillance Agency)

Managing Human Performance:

- 4 classes will be held in the Program for Managerial Development demanded by the Agência Nacional de Vigilância Sanitária – ANVISA (Brazilian Health Surveillance Agency)

- 01 closed class was held to Instituto do Patrimônio Histórico e Arquitetônico Nacional – IPHAN (Institute for Historical and Architectural Properties)

- Other demands from: Ministry of Education, Fundo Nacional para o Desenvolvimento da Educação – FNDE (National Found for the Development of Education), Military Public Prosecutor's Office, Federal Court of the Federal District.

The Evaluation Reports of these courses (as filled out by senior level executives) indicates a very high level of satisfaction with both the contents and the methodological approaches. This has impacted on the capacity of the courses to further attract new senior executives for future classes. Some statistical data helps to confirm this:

22

From the evaluations of the 4 LRA classes for ANVISA, delivered in August 2007, the percentage of evaluations with high scores – between 8 and 10 (in a 0-10 scale) – for rating the importance and applicability of the contents are demonstrated below in the diagram:

Percentage of evaluations on "8" to "10"

98 98

93

96

10099

92

96

88

90

92

94

96

98

100

102

Class1 Class2 Class3 Class4

%

Importance of thecontentsAplicability of thecontents

As the diagram above demonstrates, the vast majority of evaluations clearly scored these courses very highly. In addition, in the qualitative evaluation, participants recurrently said things like: “The event seemed very important to me. It was a moment for reflection and should be preserved at the institution (ANVISA). Activities like that should be more frequent among the management levels. My suggestion is that all the directors of the Agency may take part on activities of such a nature”. It is important to note that focusing on top levels is proving to be an effective choice for ensuring institutional commitment to the courses. Top level executives have tended to then suggest the course to his/her team. 1.5 Enhanced recognition of ENAP as a space for developing public executives The positive image of ENAP perceived by the federal administration and partner schools as a locus for learning and exchanging ideas has been enhanced by the project, and evidence for this is found in two main areas. Firstly, some of the partners were very clear in saying the image of ENAP is changing. Among the highlighted characteristics in the interviews are: dynamism, proactive approach, innovation, generosity (culture of sharing knowledge). As a Director of the Federal Service for Data Processing said:

“I was not very familiar with ENAP and I must confess that I thought that it was not the dynamic, proactive space that it actually is. I thought that it was a more

23

conservative, bureaucratic institution. I was very surprised with what I saw here in the past three years. Its concern with organizing spaces for cooperation, sharing knowledge, and integrating both domestic and international partnerships fosters the evolution of our public institutions”

The second aspect relates to the qualification of ENAP to meet customized demands, which means that the products and services developed by the School in the areas of leadership and human resources management can integrate training programs demanded by public institutions (ENAP’s know-how associated to the customer’s needs). The segmentation of audiences and differentiated training strategies, which were incorporated by ENAP under the Project, are also other aspects that contribute to the recognition of the Brazilian School as an appropriate space for developing public executives. 1.6 Contributions to the National Policy for Human Resource Development The Canadian knowledge contributed to the role ENAP is effectively playing in the implementation of and reflection about strategic policies in capacity building – especially as it relates to human resource development policy. The Brazilian government has recently published a decree specifying competency-based management as a strategic choice, and has appointed ENAP as the reference for capacity building in federal public service:

“Art. 6 – Entities of direct federal public administration (…) shall include into their training plans actions focused on building capacity to their staff to assume positions of superior advising and direction. (…) It will be ENAP’s responsibility to promote, to elaborate and to execute training actions referred above, so as to coordinate and supervise programs for managerial training of civil servants promoted by other schools of government of the federal public administration.” (Federal Decree nb. 5,707 – February 23rd, 2006.)

The Presidential Decree also states that ENAP is a member of the Managing Committee of the National Policy for Human Resource Development and responsible for the coordination of the system of federal schools of government. The cooperation with Canada improved ENAP's qualifications in the area of competence-based management, which today is key for the National Policy for Human Resource Development, ensuring a privileged position for ENAP to discuss training policies. As the president of ENAP, Ms. Helena Kerr do Amaral, said:

“The competency-based management policy adopted by the Canadian government was an initial benchmark for our discussions. ENAP's mission already included the imperative of ‘building capacities in civil servants.’ Since the presidential decree for human resource development, a key need for the national policy is using Competence-based Management as a training model.”

24

2. Brazilian executives’ capacity development The project purpose included “to increase management competencies of Brazilian federal executives and managers” through the transfer of Canadian technology. From March 2003 to July 2007, learning events of the project reached 1.070 Brazilian federal executives (of which almost 70% are women) in Brazil and in foreign missions, as the graph below illustrates:

246

190

327

207

100

1070

59

127

230

144

71

631

37

63

97

63

29

289

2003

2004

2005

2006

2007

2003-2007

Total of trained civil servants Women Men

239

183

320

203

1034

89

36

11

4

7

7

72003

2004

2005

2006

2007

2003-2007

Total of trained civil servants in Brazil Total of trained civil servants in foreign missions

These data also indicate a strong presence of women (almost 70%) among the beneficiaries. Considering that to increase women participation in senior and

25

management levels is still a challenge to Brazilian public service, this could be said as a quite good indicator. The distribution of beneficiaries along the 4-year series indicates a regular flow of participants at learning events, so as a regular delivering of activities along the project. Four issues were established at the evaluation matrix in order to assess results related to the component of management competencies of the executives reached by the project, as set bellow. 2.1 New competencies in leadership New competencies for leadership acquired or enhanced through participation in the learning events of the project were frequently mentioned by the beneficiaries. This was identified by the analysis of impact evaluation reports produced after 3 months of each course of the project, and was confirmed through the direct interviews of executives who have taken part of the learning events. The following competencies were frequently identified in the analysed material and discussions:

Focusing on the client (ability to better listen to and understand the team and other actors’ demands): this is a competency which has resulted in more effective inter- and intra-ministerial relations and actions, particularly in the process of designing and implementing multi-sectoral public policies involving agreements among the actors involved. Interpersonal relationships (ability to listen to the team, provide coaching,

provide and receive adequate feedback, self-evaluation and team evaluation): competency which has resulted in improvements in the performance of teams – quality and productivity – accountability and confidence-based relations Ability to deal with changes (adaptability to change processes and transitions

in the government): competence developed to deal with changes and transitions as opportunities to learn, teach and innovate, mobilizing teams to face new challenges.

2.2 Direct and indirect benefits in the workplace Impact evaluation reports provide good tools for assessing the impacts on the workplace. The report on the Leadership Reflection and Action (LRA) course was done with a qualitative analysis of a selected sample of executives, and indicates a strong impact on the teamwork:

35% registered an impact on support and context factors (so as composing an efficient and integrated team; having support for the decision-making, etc.) 20% registered an impact on the way tasks were executed; and, 15% stressed positive changes related to the use of new abilities.

Interviews done for this final evaluation indicated changes in the leader-team relationship, mainly concerning the openness for receiving feedback and putting oneself “in the place” of the other. Executives emphasize the importance of coaching techniques to improve the performance and outcomes of the teams, as well as

26

identify the importance of being able to see different capacities and allocate them to the correct activities. They are better prepared to recruit, train and manage teams with complementary and multidisciplinary profiles. In some cases, the impact of these changes goes beyond the relation with the team only, and touches on relationships with other areas/ sectors, as noted by the following statement of one executive:

“We are from a support area (responsible for the development of the Federal Government Multi-Year Plan), and this focus on trying to understand the specificity of other ministries, trying to get a plan which is closer to their needs, this increases their commitment with the plan we conceive.”

This focus on ‘the other’ is resulting in more customer-oriented practices. The impact on team building was also expressed, namely in high tech areas – such as distance learning in a virtual environment –, and associated with the Action-Research Roundtable (ARRT) on e-Learning. For example, an executive of the Federal Service of Data Processing stated that:

“The Roundtable afforded me an opportunity to review certain necessary team organization concepts, profiles and capacities, particularly when setting up multidisciplinary teams in an area in which everything is quite new, such as distance learning in a virtual environment. Today I have a very consistent team. We defined the required profiles and the outcomes achieved with this team have been very positive.”

2.3 Increased demand for executive capacity development All executives interviewed considered the focus on attitudinal and leadership competencies a great characteristic of the learning events of the project. Nonetheless, they still feel that more training should be offered regarding these aspects. The Canadian methodology plays an important role, in the executives’ point of view, and therefore they would like to see more such activities offered. This was supported by reaction evaluations of the LRA (Leadership, Reflection and Action) and MHP (Managing Human Performance) courses. Some of the beneficiaries interviewed also reported that they had shared the benefits/knowledge of the course within their workplace. However, this seems to be an exception in the case of one isolated individual, while another beneficiary clearly stated that a more pervasive approach is still required. 2.4 Sustained networking between Canadian and Brazilian Executives Many activities of the project catalyzed contacts and linkages between Brazilian and Canadian executives, both in Brazil and in Canada. ENAP had an important role in keeping these executives in contact – through visits and forums. Notwithstanding progress made, evaluation interviews suggested that some of these linkages are hard to sustain. Most of the executives that have been in Canada and

27

are not from ENAP staff, have not kept the links with foreign experts / executives. Some factors that may be related to this are: short time of contact during visits and no institutional plan for specific further actions to sustain linkages. 3. Partnership The project had as medium term results the purpose of strengthening partnerships and networks at both national and international levels (outputs included: network of ENAP’s and CSPS’s partners expanded; CSPS information on networks of research experts and public executives made available to Brazilian public sector through ENAP; partnership reinforced with state schools). In order to assess this component, the evaluation issues matrix (EIM) established the following criteria for evaluation: 3.1 (National level) Increased exposure of partner schools to new contents /

methodology ENAP has a Partnership Program with 11 local schools of government, since 1996, and, in 2003, a National Network of Schools of Government was created, integrating 95 schools at the federal, state and municipal levels, coordinated by ENAP. The Project strengthened both the Partnership Program and the Network, providing increasing opportunities for learning and exchanging new contents and learning methodologies and technologies, reinforcing ENAP's role to act more and more based on the logic of supporting the institutional development of these organizations. This successful strategy will be adopted for guiding the new Project with Canada. The contact of partner schools to these new contents and learning methodologies and technologies was achieved mainly through:

courses and learning events at ENAP, including meetings of the National Network; lectures of foreign experts and workshops delivered at selected partner

schools (4 schools of government in different Brazilian regions – North, Northeast, Southeast and South); international forums (such as the Brazil-Canada International Open Forum and

the International Seminar on Distance Learning); 11 CSPS titles translated into Portuguese and published under the Project in

the series named Cadernos ENAP, which were distributed to the partner schools:

1. Moving from the Heroic to the everyday: Lessons Learned from Leading Horizontal Projects

2. A Foundation for Developing Risk Management Learning Strategies in the Public Service

3. The Learning Journey: A guide to achieving excellence 4. Conference Guide 5. A Canadian Model of Public Administration 6. The Renewal of Organizations: How to Manage Transitions in the

Workforce 7. The Horizontal Challenge: Line Departments, Central Agencies and

Leadership 8. An Initial Exploration of the Literature on Innovation 9. A Primer on Risk Management in the Public Service

28

10. A Primer on Knowledge Management in the Public Service 11. Learning@large: an E-learning guide for managers – the experience of

the Canadian Government in E-Learning; and 02 publications as a result of the Action-Research Roundtables on

competence-based management and distance learning. 3.2 (National level) Enhanced partnerships with schools of government in the states

and municipalities New methodologies were stressed by ENAP’s executives as being good products for offering to the partners, and, in this manner, a way of strengthening partners’ network. One of the partner schools confirmed that learning events and innovative methodologies were the most important benefit they received from their contact with ENAP. This enhancement may be perceived through witnessing the enthusiasm of the 5 selected partner schools that are taking part of the new cooperation project (Governance Capacity Building Project). In the first meeting, held at ENAP on September 25 and 26, 2007, for the joint discussion of the new Project with ENAP, CSPS and all the Brazilian partners, all selected partner schools presented good and similar expectations concerning capacity building and learning together. Based on the experience of ENAP in the Partnership Program and in the National Network of Schools of Government, the new cooperation project to be implemented in the 2007-2009 period is both a huge opportunity – for contributing toward the institutional development of partners – and a great challenge – as new conditions must be created for carrying out international activities in different local realities and for building capacities to incorporate and adapt new learning and research contents and methodologies. Taking into account the work already carried out with these partners, there is an expectation and a desire to turn these schools into centers for disseminating new knowledge and good practices in their regions – in the role of “champion schools” – expanding the reach of the Project and its benefits to other schools. 3.3 (International Level) Expansion of CSPS and ENAP partners network Although this is difficult to measure, the expansion of the partners’ network at the international level occurred as a consequence of two main international events held at ENAP during the project:

The Brazil-Canada International Forum “To Learn, To Innovate, To Cooperate: a conversation among peers”. The Forum was attended by 73 executives from ENAP and CSPS partner schools and from the following countries: France, Spain, Portugal, Germany, Sweden, Russia, Ukraine, United States of America, South Africa, Kenya, China, Argentina, Dominican Republic and Guatemala. The International Seminar on Distance Learning, which was attended by 72

representatives of 35 different institutions – 28 from Brazil and 7 from Canada,

29

France, Spain and Panama. The Seminar was followed by the Blended Learning Workshop.

Both events were highlighted as moments of strong sharing of experiences and meeting of new partners, which resulted in further contacts. According to the president of a local school of government:

“Taking part at the International Forum made it possible to us to get in touch with people from different places of the world. We have heard experiences from different places. As the methodology asked, we have let our contacts and our demands, and actually we got some answers, namely from the Portuguese School, which have send to us some law and procedures. As for the International Seminar, we were so enthusiastic that we offered to hold the next International Seminar on Distance Learning in Curitiba – State of Paraná.”

Furthermore, ENAP adopted the strategy of fostering, whenever possible, learning events with the presence of different international partners, favoring the exchange of experiences among peers and paving the way for new cooperation possibilities for both ENAP itself and CSPS. Both schools had the chance to get in contact with the international partners of each other. The result was the expansion in the international partners’ network of both schools. For instance, ENAP established bilateral relations with South Africa and Ireland (through the Global Engagement Network Program coordinated by CSPS), and CSPS with France and Spain. 4. Unintended Results As results that had not been originally anticipated in the Project, the following ones should be highlighted: 4.1 Strengthening of ENAP's Distance Learning Program The major unintended result of the project may have been the strengthening of the Distance Learning Program at ENAP. This achievement was noted as being both the result of the project and as a condition that supported the ongoing dialogue and mutual exchange of information between Brazil and Canada. With respect to the exchange of knowledge in the area of Distance Learning, the CSPS contribution has been essential for the speedy evolution of this branch at ENAP and it is of strategic importance for training civil servants in countries of continental size, such as Brazil and Canada. The following steps led to the development of distance learning at ENAP:

In October 2004, ENAP's General Coordinator for Distance Learning attended an E-Learning Methodology Transfer Workshop at CSPS. In May 2005, two CSPS experts delivered a Lecture on E-Learning Trends

and Challenges at ENAP, followed by a Workshop on Virtual Education: Factors of Success in Instructional Design. In 2006, an Action-Research Roundtable (ARRT) on Distance Learning was

held at ENAP.

30

In April 2007, two ENAP technical experts from the Distance Learning Branch participated in a two-week internship at CSPS Blended Learning Center. In June 2007, the International Seminar on Distance Learning at ENAP was

attended by 72 representatives of 35 different institutions – 28 from Brazil and 07 from Canada, France, Spain and Panama. The Seminar was followed by the Blended Learning Workshop. In 2007, three consultancies for improving ENAP’s distance learning courses

were held: a survey for mapping out the profile of students of ENAP's E-Learning courses; instructional planning of E-learning courses; and a study for implementation of virtual communities at ENAP's Distance Learning branch. From 2004 to 2007, two CSPS courses were donated to ENAP: Passage to E-

learning and Organizing for Deliberate Innovation. In addition, ENAP technical experts browsed virtual courses offered by CSPS Campus Direct virtual school for 90 days. CSPS documents and learning materials were translated into Portuguese, and ENAP published the CSPS document “Learning@Large: an e-learning guide for managers” in the series Cadernos ENAP. Lastly, 11 books on distance learning were donated to ENAP´s library.

All the activities listed above contributed to important developments at ENAP, including the following:

The ENAP Virtual School was created in 2005, with a valuable contribution from the CSPS Centre of Blended Learning team; Distance Learning at ENAP is now responsible for 49,5% of the School’s

training. In 2006, 11.346 civil servants from all Brazilian regions were trained in distance learning courses. In 2007, this figure is expected to increase to 17,000. The cost of distance learning courses per trained civil servant at ENAP corresponds to one quarter the cost of classroom courses (according to 2006 Budget), which is an efficiency gain. Revision of distance learning courses in the light of the blended learning

approach. New instructional design approaches have benefited up to 20,000 civil

servants in all Ministries. The Passage to E-learning course donated by CSPS was adapted and

redesigned in a more dynamic and interactive way. The new format of the course was adopted by the Canadian School. More than 1,200 Brazilian civil servants have completed this course and it was donated to other Brazilian government agencies, with the authorization from CSPS. International networks were enhanced and virtual communities were created.

The virtual community named Praxis was created after the International Seminar on Distance Leaning. Creation of a community of practice in connection with the ENAP course

“Meeting Citizens’ Needs”. ENAP provides reference material on distance learning in public

organizations. CSPS has also been benefited by this cooperation on Distance Learning in the following ways:

31

CSPS had the opportunity to design and pilot a blended learning workshop based on its online Blended Learning Guide. It now can be used as workshop with CSPS clients and can be quickly customized when needed. Dynamic course renewal: the Passage to E-learning course donated to ENAP

was adapted, updated and returned to CSPS. ENAP’s Meeting Citizen’s Needs course was donated to CSPS. CSPS’ international visibility was increased through the contact with ENAP

national and international partners. The reach of CSPS’ networks was also expanded.

4.2 Inclusion of new contents and methodologies in courses of the Directorate for

Professional Training Because of a greater convergence between CSPS learning programs and ENAP's Directorate for Managerial Development, the project covered continued education programs more intensively. However, Canadian contents and methodologies were seen to be applicable to courses developed within ENAP’s Directorate for Professional Training, resulting in the incorporation of new modules and methodologies. A practical methodology for developing public policies extracted from the CSPS course "Challenges in Public Policy Development: an overview" was incorporated as a module in the career training for Specialists in Public Policies and Governmental Management (induction courses and continuous development course for career progression) and in specializations (latu sensu) in public management. 4.3 Greater integration among ENAP units and teams The project required an intensification of integrated actions among different ENAP units (international cooperation, managerial development, research, professional training, events, editing, library), which contributed to strengthening the capacity to work as a team and as a learning organization – an inherent challenge for a government school. 5. Achievement of Outcomes: Overall, the outcomes of the project were consistently achieved, although to different degrees:

Improved capacity to better manage governmental programs and a change agenda systematically promoted. Impact evaluation of learning events indicates that this has been achieved by Brazilian executives, to some extent. However, there is still a great demand for training in leadership competencies, as the interviews have shown. The incorporation of methodologies as in the Action-Research Roundtable, promoted important shifts towards research and innovation.

New learning programs designed and delivered for executive and managers.

This result was successfully achieved. Since 2003, 31 learning events were conducted at ENAP, 4 lectures were delivered at partner schools, 20 foreign

32

missions went to Canada, 4 CSPS courses were adapted by ENAP and are being regularly delivered for Brazilian executives and managers (760 civil servants until August 2007) and one CSPS e-learning course was reformulated by ENAP and delivered for more than 1,200 civil servants.

Enhanced ENAP’s capacity for the design and delivery of management

development learning products and services through Canadian Technology. Evaluation shows this is also a fully achieved result for the project.

Local partners provided with new management learning programs for federal

executives and managers in the States for their training and development in leading public management topics. The project provided partner schools with new contents, methodologies and technologies in public management topics, mainly by learning events at ENAP and lectures with Canadian experts in the partner schools. Instructional material was also made available for local partners. 11 CSPS titles were translated into Portuguese and published in the Cadernos ENAP series (distributed to partner schools and available for download at ENAP internet site). The adaptation of new courses to specific demands of partner schools, however, is yet a challenge to be addressed in the next cooperation project.

Some figures give an overview of the scope of Project outcomes:

1,070 Brazilian civil servants trained, 631 of which were women and 289 were men (in two events held in 2003, no record by gender was made).

1,034 Brazilian civil servants trained in Brazil and 36 in Canada. 31 courses/learning events held at ENAP; 04 lectures delivered in

partner schools in the North, Northeast, Southeast and South regions of Brazil; and 20 missions/courses held in Canada.

04 CSPS classroom courses adapted by ENAP to the Brazilian reality and offered regularly by the School. Until August 2007, 23 classes had already completed these courses at ENAP, totaling 760 trained civil servants.

01 CSPS distance course was redesigned by ENAP and donated back to the Canadian School. Over 1,200 Brazilian civil servants have completed this distance learning course so far.

New approaches to the instructional design of distance learning courses adopted by ENAP as a result of the cooperation with CSPS have benefited about 20,000 Brazilian civil servants in all ministries of the federal public administration.

05 Canadian executives visited 06 different Brazilian ministries and exchanged experiences with them.

02 CSPS activities/methodologies were incorporated by ENAP: Armchair Discussion and Action-Research Roundtable. 26 Armchair Discussions were held at ENAP, which were attended by over 1,000 civil servants. 02 Roundtables were held by ENAP, resulting in 02 publications, 1,480 copies of which were distributed and made available for download on ENAP's website.

33

11 CSPS publications were translated into Portuguese and published in 09 Cadernos ENAP on different topics related to public management, such as risk management, innovation, knowledge management, management of transitions in the labor force and distance learning.

In the first half of 2007 alone, 5,272 downloads of the 09 Cadernos ENAP and of the 02 books on Action-Research Roundtables were made.

184 books were donated by Canada and incorporated into ENAP's Library, which is a benchmark in public management..

Program Efficiency Efficiency in overall management of the project, good monitoring and evaluation tools and good communication between Canadian and Brazilian management teams were noted as landmarks of this project.

Managing, monitoring and evaluation instruments were constantly negotiated and agreed between the Project management teams of ENAP and CSPS since the Project design phase, resulting in optimized resources, swift actions, correction of planned steps/activities, opportunities appropriately used and problem-solving. Concerning evaluation strategies, it might be highlighted:

• The annual evaluation meetings to review anticipated activities and results achieved with the aim of defining the action plan for the following year; and

• Efficacy evaluation tools applied to the learning events (reaction and impact reports, that were used to assess the satisfaction of participants and improve future activities)

Great value for money was noted by both ENAP and CSPS executives as a characteristic of the project:

• Non-anticipated activities were carried out without additional resources; • Negotiations focused on reducing costs and on enhancing the quality and use

of Project resources in developing new activities; • Solutions for transferring funds agreed upon with CSPS and the Embassy of

Canada, ensuring safer payments without delays. Management teams provided a very efficient use of the project’s founds, constantly monitoring the use of resources and the implementation of activities. The project was supposed to finish on March 2007, when the planned activities have indeed being delivered. However, as there was still some resources available, the management teams proposed a 6 month extension of the project, what enabled new activities to be implemented, already having in mind a new cooperation proposal between ENAP and CSPS. Systematization of information is also a landmark in the project. Regular registration of course adaptation processes facilitated their use by other ENAP units and partner schools: instructional design, methodological approaches, instructor training processes, reference bibliography, learning materials. It is an initial investment which facilitated the work of the team and improved ENAP actions with its clients.

34

Standardized tools for planning and reporting missions and learning events in Canada and Brazil facilitated the evaluation of these activities and provided comparable information for discussing action plans. Visits of foreign experts were also carefully prepared and vital information on the Brazilian public service was always previously provided. This seems to be crucial for making the intervention of foreign instructors more effective in the local context. A very positive evaluation of Canadian experts’ performance was noted in the Reaction Evaluation Reports, in terms of the applicability of the contents. ENAP’s management team succeeded in operating as mediators between international and local knowledge. The existence of a new project proposal is also a success indicator for the project as a whole, suggesting that the relationship among the stakeholders has become stronger and active. Interviews indicate that the close and warm relationship between Brazilian and Canadian teams was a crucial factor for enhancing the partnership and that this should be maintained in further cooperation. Cross-cutting Themes – Gender Equality and Diversity One important achievement of the Project concerning the gender issue was the strong participation of women in the learning events (70%), although no specific strategy was designed in advance. Specific activities were done concerning both gender equality and diversity throughout the project:

• Meeting to discuss the research on gender and racial diversity in the public service, that was carried out under the project, with the participation of CSPS, ENAP, Special Secretariats for Women’s Policies, for Promotion of Racial Equality Policies and for Human Rights, Experts and Academics;

• Meetings between Canadian and Brazilian executives to discuss gender and diversity topics: a mission of Brazilian executives to Canada and the “Meeting of Women in Inclusive Policies” in Brazil;

• "Reflections on Diversity and Inequality" workshop held with instructors and course coordinators delivered by ENAP for the purpose of including the diversity dimension in courses and learning events held by the School.

Learning about the Canadian policy for diversity either in courses and specific visits to discuss this topic or through CSPS instructional models had an impact on different ENAP teams:

• Distance learning: incorporation of regional diversity elements into the design of the course “Meeting Citizens’ Needs.”

• Specialization: module about public policy development in the tailor-made specialization for the Ministry of Education – workshop on indigenous education; and for the Ministry of Health – workshop on health care for the African-Brazilian population.

• Research: “Gender, race and leadership competencies in the Federal Civil Service” (published as a Caderno ENAP and disseminated to the Federal Government and the National Network of Schools of Government).

35

Curiously, many interviews suggested that the Canadian attitudes and actions in addressing diversity were more effective in terms of sensitization for these issues than a specific workshop activity. The integration of gender and diversity issues in ENAP courses’ schedules seems to rely more on particular initiatives – inspired by examples seen abroad – than a strong institutional concern. Some stakeholders suggested that more attention should be paid to already existing Brazilian information/ knowledge on this issue. Nonetheless, there is a good demand for the results of the research carried out under the project, as exemplified by the high download rates of the document from ENAP’s website (from January to August 2007, there were 1,534 downloads of the Caderno ENAP “Gender, race and leadership competencies in the Federal Civil Service”, an average of 192 downloads per month) The main challenges identified by ENAP in order to contribute to the inclusion of gender and diversity issues as crosscutting themes in the training of civil servants are:

• Documenting the material produced in the Workshop “Reflections on Diversity and Inequality” and disseminating it to schools of government.

• Inclusion of the diversity dimension in ENAP's pedagogic project. • Offering training courses on the diversity topic on a regular basis for ENAP

staff and instructors. Sustainability As the focus of the project was in exchanging innovative learning contents, methodologies and technologies, the ability of Brazilian partner to continue working with new approaches and share them with other schools of government is crucial to the project’s sustainability. The data shown at the section on “Program Effectiveness” indicate ENAP is perfectly enabled to continue delivering the adapted courses, as well as sharing them with a network of partner schools, since the procedures adopted under the Project for knowledge documentation, management and dissemination make it possible to access and share good practices to other ENAP units and partners. The knowledge and experiences acquired in courses/visits to Canada were regular shared on workshops held at ENAP both for the technical staff of the School and for teams of beneficiary agencies. This procedure enabled the knowledge sharing beyond project activities. All the adapted courses have been integrated into ENAP’s catalogue and are being delivered in a regular basis. The main challenge now is how to share the courses with partner schools, as the courses require very specific structures (instructional material and logistics) which are not easily available elsewhere. Another positive indicator for the sustainability of the project is that ENAP has developed the capacity to meet increasing demands for courses on leadership

36

development and human resource management, especially due to the incorporation of new contents and learning methodologies in those areas. The shift towards a stronger attendance of executives and leaders in learning events at ENAP is clearly perceived, and the School is recognized as a partner that plays a major role in promoting improvements in public management practices, particularly as a result of its role in building capacities in executives, decision-makers. On the other hand, the enhancement of e-learning is contributing to a significant enlargement of ENAP’s reach – both numerically (from 6,000 students reached by e-learning in 2003 to estimated 17,000 in 2007) as well as regionally (from all Brazilian regions). This is perceived, inside and outside ENAP, as an important step toward democratization of knowledge. The existence of a new project proposal is closely related to the sustainability of this project. The sustained enforcement of the links between ENAP and partner schools, so as with Special Secretariats for Women’s Policies, for Promotion of Racial Equality Policies and for Human Rights, and with the Ministry for Social Development is leading to a strong commitment with the continuity of partnerships related to capacity development in public service.

37

Conclusions Regarding ENAP Institutional Development and Empowerment, the project was very effective in almost all issues from the evaluation matrix:

New competencies developed in ENAP Staff o Strong acquirement of interpersonal competencies by ENAP staff

related to: openness for accepting new methodologies and learning technologies for designing and delivering learning events; proactive attitude in face of demands and problems; results based management knowledge; negotiation skills; self confidence; ability to work in teams and team building; and awareness of diversity issues.

Capacity to adapt and deliver new courses according to the Brazilian reality

o ENAP adapted and delivered 4 CSPS courses on the issues of leadership, development of public policies, integrated resources management and managing human performance.

o Part of one of the courses was incorporated into the schedule of other regular and tailor-made ENAP courses.

o One distance learning course was also adapted and has already been delivered to more than 1,200 Brazilian civil servants.

Acquisition and appropriation of new methodologies

o New methodologies for increasing dialogue in learning events and new research approaches have been incorporated in ENAP’s activities.

Increased capacity to attract executives

o Increasing presence of top level leaders in ENAP activities. o Demands for leadership courses are consistently coming in from

federal agencies.

Enhanced recognition of ENAP as a space for developing public executives o Positive image of ENAP perceived by the federal administration and

partner schools as a locus for learning and exchanging ideas has been enhanced by the project.

Contributions to the National Policy on Human Resource Development

o Cooperation with Canada improved ENAP's qualifications in the area of competence-based management, which today is key for the National Policy for Human Resource Development (stated in February 2006 by the Presidential Decree nb. 5,707), ensuring a privileged position for ENAP to discuss training policies.

Regarding Brazilian executives’ capacity development, the project achieved significant progress:

38

Learning events of the project reached 1,070 Brazilian federal executives (of which almost 70% are women) in Brazil and in foreign missions. Impact evaluation reports indicates that executives reached by the courses

are acquiring new competencies for leadership, specifically: o Focusing on the client (ability to better listen to and understand the

team and other actor’s demands) o Interpersonal relationships (ability to listen to the team, provide

coaching, provide and receive adequate feedback, self-evaluation and team evaluation)

o Ability to deal with changes (adaptability to change processes and transitions in the government)

Competencies are promoting benefits in the workplace, especially regarding leader-team relationship and inter-ministerial practices.

Concerning the strengthening of partnerships and networks:

The project enhanced ENAP links with 95 other schools of government in Brazil Selected five partner schools, covering a strategic geographical distribution,

are enthusiastically taking part of the planning of a new ENAP-CSPS cooperation project. International partnerships were also enhanced, mainly by two international

forums held at ENAP during the project. Unintended results were also expressive. Main achievements are:

Strengthening of the e-Learning sector at ENAP, with considerable gains both to the cooperation ENAP-CSPS and to the learning programmes in Brazil. Incorporation of new methodologies and contents in the career training for

Specialists in Public Policy and Governmental Management and in the specializations in public management (latu sensu). Integration among different ENAP teams and units.

39

Lessons Learned and Good Practices Project Management The project was marked by excellent relations/interaction between ENAP and CSPS management teams in Brazil and in Canada, largely as a result of a clear definition of responsibilities, clear and ongoing communications between the parties, a management approach focused on optimizing outcomes and the joint definition of mechanisms to solve problems. The following characteristics are crucial factors of success observed during the whole project management cycle and should be maintained in further cooperation:

Ongoing and swift interaction with the CIDA Office in Brasília and with the Brazilian Cooperation Agency – ABC was also mentioned as a good practice. Focusing on building relations among the key actors is fundamental for

ensuring their involvement in the activities from the outset. It is important to define demands precisely, so as to better adapt and meet the

needs of the beneficiaries. The preparation of terms of reference and making guest experts aware of the context of ENAP and of the Brazilian federal public administration systematically, helped to ensure that the delivery of quality products would be more in tune with the contextual reality. The outcomes anticipated in the project should be seen as part of the

responsibilities of the partners, making sure that they are included in their planning. Such projects would benefit greatly from a Steering Committee, in particular

the new coming project with many and new stakeholders involved. Among other responsibilities, the committee could help to ensure that the transfer of expertise is leading the expected outcomes, and to address challenges as they emerge. The gender and diversity strategies should be defined with the support from

key actors and from an external consultant. The evaluation component should be designed and have its strategies defined

with the support from an external consultant, who should be subsequently involved in building evaluation tools, monitoring activities and drawing up evaluation reports. Flexibility on the part of the Canadian partners is key practice noted. This has

helped to ensure a sound technical-financial management of the activities, such as, for example, in the acquisition of more books, in the translation of documents and texts, in the publication of more titles, and in the negotiation of the copyrights of the courses “Leadership: Reflection and Action” and “Leading Transitions.” Flexibility on the part of the CIDA Brazil office to solve problems faced in terms

of covering expenses in Brazil (e.g. simultaneous translation, translation of written materials, and fees of consultants/researchers) also assisted in project successes.

40

Flexibility on the part of the Canadian Embassy in Brasilia that agreed to make the payments related to the execution of activities in Brazil. It favoured a more agile and safe project implementation.

Learning

Practical learning: • The contact with the CSPS experience in building leadership in the

public service (through visits and participation in courses) led to new possibilities in terms of learning contents and methodological approaches for ENAP teams.

• Many interviewees praised the appropriateness of the topics addressed in the courses and their format, highlighting the sensitization to “what being a leader is all about, the opportunity to exchange ideas with Brazilian and Canadian peers, the importance of fostering horizontal relations to mobilize teams around outcomes” as gains.

• As regards the relations between ENAP and CSPS teams, they mentioned the “confidence, clarity and professionalism established between them and the possibility of exchanging ideas and learning jointly” as highly positive factors.

Multiplying strategy:

• Recruiting high level executives and managers to take part in the training activities favors a multiplication of the learning within the teams and the possibility of training new team members (recommendation from superiors).

• Making available more places in courses/events held in Brazil to participants from the same institution to favor the introduction of new post-training practices and changes in their organizations (so as to enhance their influence/impact of changes).

Awareness of differences between the public administrations of Canada and

Brazil • It is important to prepare foreign and Brazilian executives for

missions/visits by providing them with contextualizing and supporting documents/texts for them to understand how the public administrations and institutions of both countries (specificities and differences) operate and to adjust the development of products and services to the needs and practices of the beneficiaries.

Strategy for knowledge acquisition and sharing:

• It is important to always send two ENAP civil servants to attend courses at CSPS, particularly for processes of adapting courses, which require more intense and longer activities. This is a strategy that favors the retention and dissemination of knowledge and a greater integration between ENAP's learning staff and research teams.

41

Recommendations

Some alerts were done regarding the regional diversity: in the next project, attention should be given to preparing trainers and instructors to deal with a larger diversity / plurality of regional contexts. New CSPS courses were suggested for further adaptation, in particular those

related to leadership development and human performance management: “Leading for Results”, “The Courage to Lead in the Public Service” and “The Art of Coaching”. More time is required for the visits of Brazilians to Canada, in order to improve

the knowledge sharing and absorption. In particular, more time would be required for practical applications of information shared. As suggested by one of the interviewees of CSPS, more time would be required for “job shadowing”, which could be quite useful in this kind of initiative. The dynamic of holding an international activity requires complex preparations

which involve the intense participation of the teams of both schools in the design of actions and adaptations to the needs of the Brazilian public administration. Therefore, in order to make the best use possible of the teams, it is necessary to avoid any agenda overlapping among ENAP's different units. The calendar of ENAP's international actions should continue to integrate the

strategic planning process of the technical teams, so as to add value to the activities being developed by them and to avoid unforeseen activities or undue burdens. ENAP should be better prepared to take on administrative activities which are

also being carried out by the technical teams, so that they may focus exclusively on using new knowledge and methodologies. Contacts with Canadian experts: the program of short visits of Brazilians

executives to Canada and of Canadian executives to Brazil clearly revealed the need to refine the program by identifying areas of common interest more precisely, anticipating potential further developments. Multiplying effect: the strategy of recruiting participants to attend courses at

ENAP was based on selecting high hierarchical level executives and managers who can influence, mobilize and recommend the same training to other executives or to members of their teams. This strategy was correct and should be continued and strengthened, with the aim of improving cost-benefit aspects.

42

Future perspectives In October 2006, ENAP and CSPS submitted to CIDA (under the Brazil-Canada Knowledge Exchange for Equity Promotion Program – KEEP) a new project proposal called “Governance Capacity Building”. The Project was recommended for approval by CIDA and Agência Brasileira de Cooperação (ABC) and it will be implemented from 2007 to 2009, if it is approved by the Canadian Minister of International Cooperation. This proposal benefited from the institutional links and lessons learned during the “Partnership for Public Sector Management Excellence” project, especially from the enforcement of ties with partner schools in other Brazilian regions and with key federal offices such as the Special Secretariats and the Ministry for Social Development. If the purpose is approved by the Canadian Minister of International Cooperation, it will certainly be a consistent way of giving continuity to the main gains achieved by the project evaluated.

43

ANNEX 1

Terms of Reference for the Local Consultant 1. Background In 2003, ENAP and CSPS began the implementation of a Project “A Partnership for Public Sector Management Excellence – a technology transfer cooperation between Escola Nacional de Administração Pública (ENAP-Brazil) and the Canada School of Public Service (CSPS)”. The project purpose is to increase the management competencies of Brazilian federal executives and managers, by the strengthening of ENAP as a centre of excellence in public management. The instrument used to ultimately achieve this purpose is through the transfer of the Canadian Technology. The end results should be:

• Improve capacity to better manage governmental programs and their change agenda which has been systematically promoted;

• Innovative learning programs designed and delivered for executives and managers;

• Enhanced capacity for the conception and delivery of products and services in management development;

• Improved learning management programs and trained federal executives and managers in leading public management topics.

ENAP, CSPS and CIDA are conducting an evaluation before the end of the project scheduled for September 2007 in order to determine whether the planned objectives and results were met and to identify best practices, lessons learned and recommendations for improvement for beneficiaries and stakeholders. A new project entitled “Governance Capacity-building Project” between ENAP and CSPS was recommended for approval by CIDA. The evaluation will therefore allow for better planning of this second project. 2. Reasons for the Evaluation The evaluation will examine the degree to which the project met its planned objectives and results, and will draw significant and useful findings and key lessons learned from the experience. The findings of the evaluation will be used to inform the design of the next project, and will be used in the planning of the future expected to be launched at the end of calendar year 2007. 3. Evaluation Framework Guided by the Issues Matrix in the Workplan, the evaluation will focus on the results achieved to date by the Project. Issues of particular interest that will be addressed by the evaluation are:

1. Achievement of Results, more specifically regarding:

• The nature and extent of ENAP’s institutional development and empowerment

44

• The Executives Capacity development in public management

• Partnerships and linkages at the national and international levels

• Unintended results 2. Efficiency 3. Gender / diversity 4. Sustainability 5. Lessons Learned 6. Recommendations for Improvement

4. Evaluator Qualifications A Brazilian consultant will carry out the evaluation under ENAP’s supervision. The consultant is expected to be:

• A reliable and effective evaluation manager with relevant experience in conducting evaluations of similar nature and a proven record delivering professional results.

• Fluent in Portuguese and adequate writing abilities in English.

• Acquainted with results-based management orientation and practices.

• Experienced in project /programme evaluations.

• Expertise in assessment methodology, data collection and analysis

• Experienced in capacity building methodologies and institutional strengthening.

• Experienced in working in the education and pubic sector capacity development fields.

5. Management of the Evaluation Management of the evaluation is the responsibility of ENAP and CSPS. See Section - 4. Roles and Responsibilities – in the Evaluation Workplan. 6. Evaluation Schedule The Consultant is expected to carry out its evaluation responsibilities between August 6 and September 30, 2007. The preliminary schedule and tasks of all involved in the project Evaluation are the following:

Evaluation Schedule of Activities Activities

ENAP Team

Local Consultant

Baastel CSPS Target Date

Develop evaluation workplan √ August 3 Review evaluation workplan √ √ August 7 Develop local consultant’s workplan √ August 8 Review local consultant ‘s workplan √ √ √ August 10 Develop data collection tools √ August 20 Review protocols √ √ √ August 21 Provide on-going support to local evaluation team √ On-going Collect data and information √ √ August 22-31

45

Activities ENAP Team

Local Consultant

Baastel CSPS Target Date

Analyse data and information √ √ September 1-5 Produce preliminary evaluation report √ √ September 14 Review preliminary evaluation report √ √ √ September 18 Produce final evaluation report √ √ September 25 7. Level of Effort The estimated level of efforts, in hours, required for the local consultant is estimated as follows:

Activities Level of effort

Review of the documents and the literature

6

Preparation of a detailed work plan 2 Develop data collection tools 4 Holding of interviews with the partners

10

Preparation of the preliminary report and the presentation, containing the results of the evaluation

18

Participation in the Final Evaluation Meeting

8

Preparation of the final version of the Evaluation Report

2

Total number of hours 50 8. Cost of Evaluation It is estimated that the cost of this evaluation including all fees and expenses would be approximately CAN$ 13,000.00. The basis of this estimate is given in the following sections.

• Consultants' fees: 50 hours x CAN$ 260.00 Total: $ 13,000.00

The level of effort and expenses are indicative. The Consultant will submit a final budget to ENAP as part of the workplan. The budget total will not exceed the amount indicated above. 9. Method of Payment For providing the consultancy services, the CONTRACTOR will pay the CONTRACTEE the amount of CAN$ 13,000.00 (thirteen thousand Canadian dollars) in a single installment. Payment will be made after the products have been delivered. 10. Intellectual Property

46

All work performed under this agreement shall remain the exclusive property of ENAP and CSPS.

47

ANNEX 2

Evaluation Matrix Evaluation Issues Matrix

ISSUES QUESTIONS SOURCES METHODS 1. RESULTS 1.1 ENAP INSTITUTIONAL DEVELOPMENT AND EMPOWERMENT 1.1.1 New competencies developed by ENAP staff

a. Which competencies have been developed by the staff? b. In what measure has the staff changed procedures and behaviors related to these competencies?

ENAP staff. Interview with selected staff.

1.1.2 Capacity to adapt and deliver new courses according to the Brazilian reality

a. How did ENAP adapt and deliver new courses? b. How did the beneficiaries consider attended courses suitable for their own reality?

ENAP staff; Project annual reports. Beneficiaries

Interview with selected staff. Document review

1.1.3 Acquisition and appropriation of new methodologies

a. In what measure have the delivered courses incorporated new methodologies? b. In what measure did ENAP change strategies towards new methodologies?

ENAP staff; Project annual reports.

Interview with selected staff. Document review

1.1.4 Increased capacity to attract executives / strategic managers (increase of demand and change in the audience)

a. How did the demand for ENAP’s courses change along the project implementation (number and profile of executives attending delivered courses)?

ENAP’s annual reports. ENAP staff Executives

Document review Interview

1.1.5 Enhanced recognition of ENAP as a space for developing public executives

a. Is ENAP being more demanded for capacity building?

ENAP staff

Interview with selected managers;

1.1.6 Contributions to the National Policy for Human Resource Development

a. Is ENAP in higher demand for participating in the implementation of the public policy on human resource development? b. How the project contributed to the inclusion of the theme “competence-based management” in the discussions about the policy?

ENAP staff; Annual reports

Interview with selected staff/ managers; Document review; Federal Decree nb. 5,707/2006

1.2 EXECUTIVES CAPACITY-DEVELOPMENT 1.2.1 New competencies in leadership (Brazilian Executives that attended the courses at CSPS and the LRA course at ENAP)

a. Which competencies have been developed by the beneficiaries (executives)? b. In what measure have they changed procedures and behaviors related to these competencies?

Selected executives / managers Course Reports

In deep interview Document review

48

ISSUES QUESTIONS SOURCES METHODS 1.2.2 Direct / indirect benefits / effects of these new competencies in the work place

a. How did the training affect their relationship to co-workers (superiors and subalterns)? b. Are these competencies shared with / replicated to co-workers? c. How did these competencies impact over the efficiency of delivering public policies/ programs?

Selected executives / managers Course Reports

In deep interview Document review

1.2.3 Increased demand for executive’s capacity development (indicator of the needs for additional training on leadership)

a. In which measure is the changing of procedures creating new demands of executives / managers affected by the project? b. What kind of demands?

Selected executives / managers Course Reports

In deep interview Document review

1.2.4 Sustained / increased networking between Canadian and Brazilian executives

a. Is there a regular exchange of information between Brazilian and Canadian executives?

Selected executives / managers ENAP staff Course Reports

In deep interview Document review

1.3 PARTNERSHIP 1.3.1 National Level 1.3.1.1 Increased exposure of Partner Schools to new contents / methodologies

a. How did partner schools take part in the project? b. How did they get access to new methodologies? c. Did the new methods correspond to partners´ needs? What do they think about it quality?

Partner schools Annual reports

Interview Document review

1.3.1.2 Enhanced partnerships with schools of government in the states and municipalities (within the network of 95 schools of government)

a. What kind of partnership has been established amongst the schools network along the project? b. How do they relate to the project’s implementation?

ENAP staff/ managers

Interview with selected staff/ managers.

1.3.2 International Level 1.3.2.1 Expansion of CSPS and ENAP partners network (to events – Forum, Distance Learning Event)

a. Have there been new contacts with international partners along the project’s implementation? How? How many?

ENAP staff/ managers Project annual reports

Interview with selected staff/ managers. Document review

1.4 UNINTENDED RESULTS 1.4.1 Development of E-learning (Establishment of the Virtual School, Reach that leads to democratization of learning – by region)

a. Is there a virtual school established under ENAP coordination? b. How does it work and who does it reach?

ENAP staff/ managers Project annual reports

Interview with selected staff/ managers. Document review

1.4.2 Enhanced capacity to manage change / to mobilize, to optimize team building within ENAP (Experience from the International Forum)

a. How has ENAP handled changes and team building issues during the project? b. Has there been any positive changes? Please explain.

ENAP staff/ managers project annual reports

Interview with selected staff/ managers. Document review

49

ISSUES QUESTIONS SOURCES METHODS 1.4.4 State careers / Specialisation of Public Policy Courses (inclusion of the Public Policy Development Course methodology)

a. Have there been new methodologies incorporated to the already existing courses? Which? How many?

ENAP staff/ managers Project annual reports

Interview with selected staff/ managers. Document review

2. EFFICIENCY 2.1 Value for money (optimization of the resources, extension of the project from 3 to 4,5 years)

a. Were the results in the LFA achieved with the amount of resources originally established? b. Were the resources efficiently used?

ENAP staff/ managers Project LFA and Budget Annual reports

Interview with selected staff/ managers.

2.2 Management performance

a. How adequate are the performance indicators for the project goal, purpose and activities? b. How did the project identify and manage risks and changes? c. Were creative solutions developed and implemented for them? d. How information has been systematized in project management?

Project LFA Project annual reports ENAP staff

Document review Interview

2.3 Relations CSPS – ENAP (example the continuation of the partnership between Distance Learning teams – future development course)

a. Is the relationship between the stakeholders resulting in further partnerships? Does it have any further perspectives? b. Are there joint actions being developed besides the project? How do they relate to the project?

ENAP staff/ managers

Interview with selected staff/ managers.

2.4 Monitoring and reporting (activities based vs. results based)

a. How frequent and accurate have been the monitoring and evaluation reports of the project? b. How have these reports’ findings been incorporated into the project implementation?

Annual reports Annual work plans Reaction and impact evaluation reports

Document review

3. GENDER / DIVERSITY 3.1 Increased awareness regarding gender and diversity issues

a. How are gender and diversity issues being incorporated at the practice of executives / managers reached by the project (policy making, projects, documents)?

ENAP staf/ managers Key executives/ managers;

Interview with ENAP staff Interview with selected key executives

3.2 Integration of gender and diversity in ENAP’s Guidelines in view of course design and delivery

a. How are gender and diversity issues being integrated into course’s agendas? b. How are these issues addressed in the selection of the course participants and intended project beneficiaries?

Annual reports Course’s agenda Course’s reports and evaluations

Document review

3.3 Exposure of Secretariats to issues related to gender equity and diversity issues

a. Which Secretariats have been exposed to gender and diversity issues, and how?

ENAP staff interview

3.4 Increased awareness and sensitization of trainers to gender and diversity issues

a. How are gender and diversity issues being integrated into course’s agendas?

Course’s agenda, reports and evaluations ENAP Staff

Document review In deep interviews with

50

ISSUES QUESTIONS SOURCES METHODS ENAP staff 4. SUSTAINABILITY 4.1 Sustained / increased demand for specifics course programs (Leadership)

a. Is there regular demand for specific courses? / Are new demands coming in regularly? b. Has there been a regular number of on-demand courses in the past years? c. Are demands increasing? From where are they coming?

ENAP Staff / managers Annual course’s schedule

In deep interviews Document review

4.2 Effective transfer of knowledge & skills leading to adoption, appropriation and adaptation of new contents

a. In what measure have the contents of courses and programs delivered changed in the last 3 years? b. How do these contents address local/ regional context (constraints and opportunities)? c. To what extent have new contents been adapted?

ENAP Staff Annual reports Course’s agenda, reports and evaluations

Interview Document review

4.3 Systematization and institutionalization of new contents

a. Are new contents being consistently incorporated by ENAP’s course coordinators?

ENAP Staff Annual course’s schedule / portfolio

Interview Document review

4.4 E-Learning – establishment of a permanent Virtual School

a. Is there a virtual school offering and delivering regular courses?

Annual reports ENAP staff

Document review Interviews

4.5 Increased and more diversified reach (by region)

a. Have there been participants from all Brazilian regions at delivered courses? b. How has regional coverage changed in the past 3 years?

Annual reports Course’s agenda, reports and evaluations ENAP staff

Document review Interviews

4.6 Progressive shift from a traditional transfer of knowledge model to a more high tech E-Learning model

a. How many courses have been delivered by e-learning methodology in the past 3 years? b. What does this represent from the total courses delivered each year?

ENAP Staff Annual reports

Interview Document review

4.7 Higher reach associated to lower costs of E-Learning model

a. How many people are now being reached by courses based on E-Learning model, compared to those at traditional courses? b. What does this represent in cost-benefit terms?

Annual reports ENAP staff

Document review Interviews

4.8 Networking of schools a. Which schools are officially networking? b. Has there been a consistent exchange between them?

Annual reports ENAP staff / managers

Document review Interviews

4.9 Consolidation of ENAP as a Leader / Model institution / Centre of excellence

a. Which images / conceptions are associated to ENAP by key executives / managers in Federal offices? b. What kind of recognition (awards, references, evaluations) is ENAP receiving in public administration?

Selected executives / managers

In deep interview

5. LESSONS LEARNED

51

ISSUES QUESTIONS SOURCES METHODS 5.1 Adaptation of contents and methodologies

a. Which problems and solutions were observed concerning the adaptation of contents and methodologies to the Brazilian reality?

ENAP staff / consultants

Interviews

5.2 Follow-up of activities (Forum, visits of executives)

a. Which lessons were learned from follow-up activities?

ENAP staff / consultants

Interviews

5.3 Increased / enhanced relations at top / senior levels

a. Which lessons were learned concerning top level executives?

ENAP staff / consultants

Interviews

5.4 Adaptation of courses to local / regional reality

a. Which lessons were learned concerning adaptation of courses to local reality?

ENAP staff / consultants

Interviews

5.5 Reciprocity – ENAP and Partner Schools learning from each other

a. What has ENAP learned from partner schools during the project?

ENAP staff / consultants

Interviews

52

ANNEX 3

Schedule and List of Interviews Schedule and List of Stakeholders Interviewed in Brazil:

1. ENAP Staff a. Strategic level:

Date / Time / Place Name/ Position 1. 06/09/07 / 10h00-11h00 /

ENAP Helena Kerr do Amaral – President of ENAP

2. 23/08/07 / 15h30-17h00/ ENAP

Margaret Baroni – Director for Managerial Development/ ENAP

3. 06/09/07 / 13h00-14h30 / ENAP

Cláudia Conde – Head of the International Cooperation Office / ENAP

b. Technical level: coordinators and technicians 4. 24/08/07 / 14h30-15h30 /

ENAP Focus Group: 1. Juliana Silveira L. de Souza – Coordinator General for Communications / Directorate for Communications and Research / ENAP 2. Clarice Gomes de Oliveira – Advisor of the Coordination General for Research / Directorate for Communications and Research / ENAP 3. Letícia Leal Lengruber – Course Coordinator/ Coordination General for Training Programs / Directorate for Managerial Development / ENAP

5. 24/08/07 / 16h00-17h00 / ENAP

Focus Group: 1. Paulo Carvalho – Director for Professional Training / ENAP 2. Ligia Aparecida de Arruda Camargo Lacerda – Coordinator General for Especialization Course / Directorate for Professional Training / ENAP 3. José Mendes de Oliveira – Course Coordinator / Coordination General for Training Projects / Directorate for Managerial Development / ENAP

6. 06/09/07 / 9h00-10h00 / ENAP

Sonisley Santos Machado – Coordinator General for Training Programs / Directorate for Managerial Development / ENAP

7. 06/09/07 / 11h30-12h30 / ENAP

Focus Group: 1. Tarcilena Polisseni Cota Nascimento – Coordinator General for Distance Learning / Directorate for Managerial Development / ENAP 2. Maria Fernanda Borges – Technician on Educational Issues / Coordination General for Distance Learning / ENAP

53

c. Facilitators / instructors 8. 24/08/07 / 13h00-14h00 /

ENAP Luiz Alberto Andreola – ENAP instructor of the “Leadership: Reflexion and Action” course

9. 30/08/07 / 15h30-16h30 / ENAP

Celina Romagnolli Dantas – Advisor of the Director for Human Resources Management/ Presidential Staff Office (Casa Civil) / Presidency of the Republic (ENAP instructor of the “Managing Human Performance” course)

10. 30/08/07 / 17h00-18h00 / São Paulo (by phone)

Heloísa Nogueira – ENAP Facilitator in the Brazil-Canada International Open Forum “To Learn, To Innovate, To Cooperate: a conversation among peers” and in the International Seminar on Distance Learning

2. Executives and managers in federal government directly reached by the

project 11. 31/08/07 / 10h00-11h00 /

ENAP Focus Group 1. Selma Roller Quintella – Director for Human Resources Management/ Presidential Staff Office (Casa Civil) / Presidency of the Republic 2. Erick Fagundes Ribeiro – Head of the Division for Implementation of the Management Model of the Multi-Year Plan / Department for Management of the Multi-Year Plan / Secretariat for Planning and Strategic Investments / Ministry of Planning, Budget and Management

12. 24/08/07 / 10h00-11h00 / ENAP

Beatrice Kassar do Valle – Coordinator General for Programs Evaluation / Secretariat for Planning and Strategic Investments / Ministry of Planning, Budget and Management (attended the “Leading for Results” course at CSPS)

13. 28/08/07 / 09h30-10h30 / Serpro – Belo Horizonte

Margareth Alves de Almeida – Manager of the Department for Educational Technologies and Methodologies / Federal Data Processing Service – Serpro (participated in the Action-Research Roundtable on Distance Learning)

14. 31/08/07 / 14h30-15h30 / ENAP

Renata Alves Carvalho – Head of the International Cooperation Unit / International Relations Office / Brazilian Health Surveillance Agency (attended the “Leadership: Reflection and Action” course at ENAP)

3.Partners

15. 31/08/07 / 17h00-18h00 / Curitiba (by phone)

Focus Group Carlos Homero Giacomini – President of the Municipal Institute for Public Administration of

54

Curitiba (State of Parana) Newton Dan Faoro – Director for Adminitrative Planning / Municipal Institute for Public Administration of Curitiba (State of Parana)

16. 13/09/07 / 10h30-11h30 / Recife (by phone)

Simone Direito – Development Officer / CIDA Brazil

List of Stakeholders Interviewed in Canada:

Ms. Hanny Toxopeus – Director General of the International Cooperation and Strategic Change Directorate, CSPS Mr. Ian Creery – Former Project Director, International Cooperation and

Strategic Change Directorate, CSPS Ms. Barbara Lukaszewicz – Director of the Blended Learning Center, CSPS Mr. Michel Bourdon – Director, Leading Transitions and Leadership,

Leadership Development and Senior Leaders Program, CSPS

55

ANNEX 4

List of Documents Reviewed Project documents Original project description Annual activities report 2003-2004 Annual activities report 2004-2005 Annual activities report 2005-2006 Annual activities report 2006-2007 Action Plan 2004-2005 Action Plan 2005-2006 Action Plan 2006-2007

Evaluation reports: Reaction Evaluation Report of the workshop: “Developing Competencies for

Preparing Training Plans”, 2004 Evaluation of the “Leadership, Reflection and Action” Course: Impact on the

Work. Reaction Evaluation Report of the course “Challenges of Public Policies

Development”, 2004 Impact Evaluation of the course “Challenges of Public Policies Development”,

2004 Reaction Evaluation Report of the “Integrated Resources Management”

Course, 2005. Impact Evaluation of the “Integrated Resources Management” Course, 2006. Reaction Evaluation Report of the workshop: “ Reflections on Diversity and

Inequality”, 2006 Reaction Evaluation Report of the course “Managing Human Performance”,

2006 Impact Evaluation of the course “Managing Human Performance”, 2006 Reaction Evaluation Report of the “International Open Forum Brazil-Canada:

to Learn, to Innovate and to Cooperate – a conversation among peers”, 2006. Reaction Evaluation Report of the “Leadership, Reflection and Action” Course,

4 classes for ANVISA, 2007 Reaction Evaluation Report of the following lectures:

o "Making Policy in the 21st century: The policy environment as seen in the Canadian Perspective", delivered by the expert David Johnston. FUNDAJ, Recife-PE, Aug. 2003.

o “Performance Management”, delivered by Chantal Lussier and France Grenier. IMAP, Curitiba-PR. Aug.2006.

o “Emotional Intelligence and Today’s Leader”, delivered by Michel Bourdon; and “Leading Transitions: The Art of Champions”, delivered by John Flynn. ENSP, Rio de Janeiro-RJ. Jun. 2004.

Reports of missions and learning events in Canada Report of participation at the “Building Resilience” course, 2004.

56

Report of technical visit on Distance Education – CSPS, Campus Direct, 2004 Report of participation at the “Negotiating Skills” course, 2004 Report of participation at the “Challenges of Public Policy Development: an

Overview” course, 2005 Reports of participation at the “Leading for Results” course, 2007.

Publications Gênero, Raça e Competências de Direção no Serviço Público Federal

(Gender, race and leadership competencies at the Federal Public Service), Cadernos ENAP, 31, 2006.

Gestão por Competências em Organizações de Governo (Competence-based management in governemental organizations). ENAP, 2006

Educação a Distância em Organizações Públicas (Distance Learning in Public Organizations). ENAP, 2007.

Newsletter “A Partnership for public sector Management Excellence” Other documents Final Report for the “International Open Forum Brazil-Canada: to Learn, to

Innovate and to Cooperate – a conversation among peers”, 2006. Detailed Proposal for the Project “Governance Capacity-Building” between

ENAP and CSPS (2007-2009). Report on the publication downloads from ENAP’s site – January to August

2007. Report of in-coming demands for ENAP’s courses – Dec. 2006 to Mai 2007. Federal Decree 5.707/2006, that states the National Policy on Human

Resources Development List of books and journals published by the project Draft learning text for a course on “Training for Trainers on Diversity and

Inequality”, 2006. Power-Point presentation of project’s activities (2003-2007), ENAP, 2007

57

ANNEX 5

Presentation to the Project’s Final Meeting

58

Partnership for Public Sector Management Excellence

2003-2007

Final Evaluation

Brasília, September 25th, 2007

A Cooperation Project on Technology Transfer between Escola Nacionalde Administração Pública (ENAP) and the Canada School of Public Service (CSPS)

Supported by the Canadian International Development Agency (CIDA)Canada-Brazil Technology Transfer Fund – Phase II

The Genesis of Evaluation

In the beginning, God created the heavens and the earth…

then he created the plants, the animals, man and woman…

The evening of the sixth day, God stopped, contemplated his work and said:

“This is all very good”.

The Genesis of Evaluation

An archangel heard him and said to God: “Very good? Very good compared to what?

What criteria are you basing your judgement on?What problem did we have

that merited this kind of investment?What were your objectives?

How can you say that you’ve met them?”

The Genesis of Evaluation

“Six days to do that.Isn’t that a bit long for an all-powerful God?

Finally, aren’t you a bit too involved in this project to be able to express an objective

opinion?”

The Genesis of EvaluationOn the seventh day, God’s rest was seriously disrupted by

all these questions.

On the eighth day, God called his archangel and said to him, “Lucifer, go to Hell!”

Lucifer had just invented government intervention evaluations and discovered the risks.

(“The Genesis of Evaluation” was extrated from the presentation prepared by theGroupe-conseil Baastel Itée for the Practical Training Workshop on Evaluation and

Results-Based Management – Gatineau, from July 30 to August 01, 2007)

Evaluate Outcomes• Integrated perspective

– The outcomes result from a process in which many actions combine with one another. For this reason, we will not present the outcomes of each activity separately. Therefore, we organized the main outcomes related to the project in one way or another by impact area:

1. Brazilian Executives2. ENAP3. Domestic and International Partners

Evaluate Outcomes• Framework of the Presentation

– Methodology adopted– Points of view of the different stakeholders:

1. Brazilian public executives 2. ENAP3. Domestic and international partners

– Non-anticipated outcomes – Other aspects: efficiency, sustainability, gender

& diversity– Lessons learned

Methodology• The outcomes refer to changes in certain

aspects of reality. How can such changes be measured and in relation to what?

– Challenge: the original project had not been prepared with the idea that it would be evaluated by its outcomes and therefore it does not allow for an ex-ante / ex-post evaluation

– Solution: definition of an evaluation matrix with issues and questions allowing for one to focus on perceived changes, based on the perspective of different actors and crossing different sources.

Evaluation matrix Questions that guided the evaluation

• ENAP's institutional strengthening – New competencies development: incorporation of new state-of-the-art

contents into public management, innovative learning methodologies and technologies, capacity to develop and adapt courses, learning products and services. Enhanced capacity to attract executives, being recognized as a benchmark center, building new institutional links with federal agencies and national and foreign schools of government, contributions to the National Policy for Human Resource Development.

• Brazilian public executives – capacity-building– New leadership competencies, application to the work environment,

perception of the need to acquire new knowledge and skills, expansion of the contact network with national, Canadian and international peers.

• Domestic and international partnerships– Exposure of partner schools to new contents and methodologies,

strengthening of domestic partnerships, expansion of the network of foreign partners of ENAP and CSPS.

• Unexpected outcomes • Efficiency• Gender and Diversity• Sustainability• Lessons Learned

Sources• Interviews

11 interviews and 5 focal groups in Brazil, 4 interviews in Canada. 26 people directly consulted.

– ENAP Team:• Executives• Technical experts• Instructors and

consultants– Beneficiaries

• Brazilian public executives

– Partners• Partner schools in

Brazil• CSPS• CIDA

• Documents – Project documents

• Original project• Annual activity reports

– Evaluations of courses, events and missions

• Reaction reports • Impact reports • Mission reports

– Publications– Administrative

documents • Control over course

demands, download reports from ENAP'swebsite, etc.

Outcomes1. Brazilian public executives

Capacity building

Adaptability to deal with changes should be a key competency of the government.

Learning is the key to a quick adaptation, as it is a fundamental requirement to keep our governments at the lead of

change processes.

Training leaders who value the professional development of civil servants is a key requirement for consolidating an efficient, modern, transparent, ethic and democratic State.

Helena Kerr do Amaral

President of ENAPRuth Dantzer

President of CSPS

1. Brazilian public executives The reach of the project

• As a specific objective focused on executives, the project contemplated the following: “Developing and increasing the public policy managing capacities of Brazilian federal executives and managers”

• Between March 2003 and July 2007, the project trained 1,070 executives in learning events, 68% of whom were women.

1. Brazilian public executivesProject strategies / components

• Leadership– Participation in courses delivered at CSPS/Canada, including visits to

institutions of the Canadian public service– Workshops and lectures delivered by Canadian experts in Brazil

(ENAP and partner schools)– Canadian courses adapted to the Brazilian reality and delivered by

ENAP

• Challenges of Public Policy Development: an overview9 editions delivered at ENAP, attended by 484 executives

• Leadership: Reflection and Action 4 editions delivered at ENAP, attended by 76 executives + 4 closed classes for the Brazilian Health Surveillance Agency (ANVISA), attended by 86 executives = Total of 162 executives

• Integrated Resource Management 2 editions delivered at ENAP, attended by 43 executives

• Managing Human Performance 4 editions delivered at ENAP, attended by 71 executives

1. Brazilian public executivesProject strategies / components

• Research– Action-Research Roundtable

• Competency-Based Management 24 experts of federal departments, schools of government, regulatory agencies, legislative and judiciary branches, working together in an innovative applied research methodology

• Distance Learning 23 experts of universities, schools of government and federal agencies

• Exchange– Brazil-Canada International Open Forum “To Learn, To Innovate,

To Cooperate: a conversation among peers” 73 executives from ENAP and CSPS partner schools and from 14 countries

– International Seminar on Distance Learning 72 representatives from 28 Brazilian institutions and from 7 foreign ones

1. Brazilian public executives Project strategies / components

• Publications– Translations of documents published by CSPS

• 11 documents published in 09 Cadernos ENAP available on-line on ENAP's website (over 700 downloads in August 2007 alone)

– Action-Research Roundtable • 2 books published, available on-line: “Gestão por Competências”

(Competency-Based Management, 400 downloads in August 2007) and “Educação a distância em Organizações Públicas” (Distance learning in public organizations, 182 downloads in August 2007)

– Reasearch: Gender, Race and Leadership Competencies in the Federal Civil Service

• 1 Caderno ENAP available on-line (266 downloads in August 2007)– Acquisition of books for ENAP's Library

• 184 books included in its collection

1. Brazilian public executives Outcomes observed

• Participation in all activities contributed, in different ways and degrees, for the following outcomes perceived by the executives who took part in them:

– New competencies in leadership– Better perception of their training needs in leadership and

human resource management– Improvements in their work environment and in their

relationship with their teams

1. Brazilian public executivesNew competencies in leadership

• The following competencies, which are more closely linked to participation in learning events, were repeatedly mentioned in the interviews carried out with beneficiaries and in impact reports of courses delivered in Brazil and in Canada:

– Focus on the client (capacity to hear – exercising how to hear what teams and other actors have to say)

• This is a capacity which has resulted in more effective inter- and intra-ministerial relations and actions, particularly in the process of designing and implementing multi-sectoral public policies involving agreements among the actors involved

– Personal inter-relationships (capacity to hear what teams have to say, coaching –giving and getting positive and corrective feedback on a timely basis – self-evaluation and team evaluation)

• Capacity which has resulted in improvements in the performance of teams –quality and productivity – accountability and confidence-based relations

– Capacity to deal with changes (adaptability to change processes and transitions in the government)

• Building capacity to deal with changes and transitions as opportunities to learn, teach and innovate, mobilizing teams to face new challenges.

Personal Inter-relationship

“ When we hold leadership positions, we tend not to look at ourselves enough because of time restraints, among others. We have little space to evaluate ourselves and to get feedback. For example, holding a meeting and asking: ‘What did you think?’ ‘Was I right or wrong?’

This enhances the motivation and engagement of the team and its satisfaction with working. ”

Executive of the Secretariat for Planning and Strategic Investment of the Ministry of Planning, Budget and Management

Capacity to deal with changes

“ In June, we experienced huge changes. A new Secretary was appointed after ten years, there were changes in the team in the three most busy months of the year, when we were preparing the 2008-2011 Multi-Year Plan to submit it to Congress. The challenge for us was to carry on as if nothing had happened. The course "Leading for Results" helped us a lot to deal with these changes, to see them as natural, to deal with our feelings. I discussed this with the team. I won and the team won.

”Executive of the Secretariat for Planning and

Strategic Investment of the Ministry of Planning, Budget and Management – attended the "Leading for

Results" course at CSPS

Presentation by Mrs. Beatrice Kassar do Valle, Coordinator General for Programs Evaluation of the Secretariat for Planning and Strategic Investment of the Ministry of Planning, Budget and Management.

Responsible for the evaluation of programs included in the Federal Government Multi-Year Plan.

Her presentation will emphasize the application of the knowledge acquired in the Course Leading for Results at CSPS.

1. Brazilian public executivesImpacts on the work environment

• The leadership capacities developed in the learning events were enhanced through research, experience-sharing and publicationcomponents. This fact can be perceived in reports and interviews, where improvements in teamwork are repeatedly mentioned, particularly in relation to:

• Identifying demands and the potential of teams• executives emphasize the importance of coaching

techniques to improve the performance and outcomes of the teams

• executives identify the importance of being able to see different capacities and allocate them to the correct activities

• Team training • executives better prepared to recruit, train and manage

teams with complementary and multidisciplinary profiles

Identify demands of the team

“ The course O Papel do Gerente na Gestão do Desempenho de Equipe (Managing Human Performance) was very useful because, among other things, it helped me to see when we should stop to talk, coach or not, and give feedback at the right time or choose the best opportunity for exchanging ideas.

”Executive of the Presidential Staff Office (Casa Civil) / Presidency of the Republic –attended the Managing Human Performance Course at ENAP

It is a total disaster when an executive is basically a policymaker and is assigned to manage many people. Executives should seek to improve their interpersonal capacities, their emotional intelligence. In this course, ENAP adopted the correct approach.

Setting up teams

“ The Roundtable afforded me an opportunity to review certain necessary team organization concepts, profiles and capacities, particularly when setting up multidisciplinary teams in an area in which everything is quite new, such as distance learning in a virtual environment.

Today I have a very consistent team. We defined the required profiles and the outcomes achieved with this team have been very positive.

”Executive of the Federal Data Processing Service (SERPRO) –participated in the Action-Research Roundtable on Distance Learning in Public Organizations.

1. Brazilian public executivesNew leadership demands

• The executives who were heard unanimously praised the investments made by ENAP in courses designed to develop attitudinal and behavioral aspects of leadership and acknowledge this field as one requiring more training.

Evaluations

“ The main benefit of the course was the opportunity it afforded for us to learn some techniques that people adopt intuitively in the public service; however, intuition is not thesame as documenting and organizing all of this and improving behaviors, fostering a creative work environment.

Executive of the Secretariat for Planning and Strategic Investment – attended the Course "Leading for Results“ at CSPS

”ENAP Instructor of the Course "Leadership: Reflection and Action"

The LRA course arouses one's interest because it deals with something that most courses fail to address: one's “emotions”, but without being slushy. It deals with the human being as a whole and does not focus on rational aspects only, leading people to reflect on another aspect which they don’t usually address.

Reaction Evaluations

“

”

Using a direct approach, the course presented topics of high interest to the Public Administration. The practical activities involved intelligent games which promoted candid debates and a climate favoring interpersonal development. The instructors were highly professional, facilitating the development of the activities.

Executive of ANVISA. Participant in the Course "Leadership: Reflection and Action“ at ENAP.

1. Brazilian public executives Limitations / Attention!

• The outcomes achieved also suggest that we should pay attention to certain foci which are required for us to ensure the continuity of leadership development actions:– Contacts with Canadian experts

• The program of short visits of Brazilians executives to Canada and of Canadian executives to Brazil clearly revealed the need to refine the program by identifying areas of common interest more precisely, anticipating potential further developments.

– Multiplying effect • The strategy of recruiting participants to attend courses at ENAP

was based on selecting high hierarchical level executives and managers who can influence, mobilize and recommend the same training to other executives or to members of their teams

• This strategy was correct and should be continued and strengthened, with the aim of improving cost-benefit aspects.

Outcomes2. ENAP

Institutional Strengthening and Development

Schools such as ENAP and CSPS have a key role to play in promoting cultural changes, in making sure that

civil servants are up to these challenges. Ensuring solid governance arrangements is a key element of this

learning.

I believe that the coincidence in terms of mission and values and principles shared by ENAP and the Canada School of Public Service defines the success of our Cooperation Project, because we have been learning, innovating, and cooperating jointly in practice.

Helena Kerr do Amaral

President of ENAPRuth Dantzer

President of CSPS

2. ENAP The reach of the project

• The specific objective of the project was to build capacities in Brazilian federal executives "by strengthening ENAP as a center of excellence in public management through the transfer of Canadian technologies."

• The project involved all ENAP units directly and ensured positive results for all teams, promoting interaction among them.

2. ENAP Project strategies / components

• Adapting new leadership courses and making them available– Adaptation of 4 CSPS courses and regular offering by ENAP

in different modalities• Incorporation of new methodologies

– Action-Research Roundtable• Competency-based Management• Distance Learning

– Armchair Discussion (Café com Debate)• Model adapted and incorporated into ENAP's program. 26 Cafés

com Debate were held according to the model developed by CSPS and attended by over 1,000 participants since 2003.

• Expanded reach and presence of ENAP in the different Brazilian regions– Expansion of Distance Learning opportunities

2. ENAPInstitutional Strengthening

• The activities which were carried out contributed to the following outcomes in different ways and at varying degrees:

– New competences developed in the teams– Capacity to adapt courses– Incorporation of new contents, methodologies and technologies– Enhanced capacity to attract executives– Greater recognition of the School as a space for developing

public executives – Contributions to the National Policy for Human Resources

Development created by a Presidential Decree in 2006 and coordinated by the Ministry of Planning

2. ENAP New competencies in the teams

• The following competencies are repeatedly mentioned in the interviews held with ENAP executives, technical experts and instructors as competencies developed during the cooperation with CSPS:

– Possibilities for incorporating new education methodologies and technologies to be applied to the development of learning events

• The team feels qualified to think and develop innovative products and services.

– Proactive approach• The team analyzes the demands and problems and responds

to them in a swift, professional, and objective manner.– Results-based management

• Clear goals, clearly allocated resources, clearly defined deadlines, analysis of risks and actors

2. ENAP New competencies in the teams

– Negotiation• Recognition of cultural differences, concertation

(consultation/dialogue), consensus-building– Self-confidence

• Recognition of existing knowledge and competencies and of learning and exchange opportunities with partners

– Ability to work in teams and team building• Recognition of the complementariness of capacities

and integrated mobilization of talents, with the aim of ensuring the best performance possible to the team and achieving actual outcomes

– Sensitivity to diversity

2. ENAP Capacity to adapt and deliver new courses according to the Brazilian reality

• 04 new classroom courses adapted and included in ENAP's catalogue in the areas of public policy development, leadership, integrated resource management, and managing human performance. Until August 2007, 23 classes in these topics had been delivered by ENAP using Brazilian instructors and 760 civil servants had been trained in them.

• The course "Challenges in Public Policy Development: an overview" was incorporated as a module called "Workshop on the development of public policies" in the career training for Specialists in Public Policies and Governmental Management (induction courses and continuous development courses for career progression), and in specializations in public management (latu sensu).

• In the process of adapting and holding pilot courses, ENAP offered a training for trainers, selecting participants with a profile to become instructors. In the following courses, which began to be regularly offered by ENAP, those selected participants began to act as instructors.

• Distance Leaning – 01 E-learning course adapted and redesigned in a more interactive and dynamic way. The Course "Passage to E-Learning" has been completed by over 1,200 Brazilian civil servants already.

“The main gain afforded by the cooperation with CSPS, with CIDA's support, was to enable ENAP to build the capacity to develop contents and new methodologies for executives. Besides bringing these contents to Brazil and adapting them successfully with the support from Canadian professors, we also learned how to best adapt them and their methodology.

”Margaret Baroni – Director of the Directorate for Managerial

Development

Presentation by Mrs. Celina Romagnolli Dantas, Advisor of the Director for Human Resources Management/ Presidential Staff Office (Casa Civil) / Presidency of the Republic.

She attended one of the pilot courses adapted by ENAP and today is the instructor in charge of this course, named O papel do gerente na gestão do desempenho de equipe (Managing Human Performance).

2. ENAP Incorporating new methodologies

• Open Space Technology – Benchmark for the design of events aimed at building knowledge

collectively (Brazil-Canada International Open Forum “To Learn, To Innovate and to Cooperate: a conversation among peers”, International Seminar on Distance Learning and events of the National Network of Schools of Government)

• Action-Research Roundtable – This event strengthened the prior strategy of ENAP of betting on new

research methodologies focused on practical issues of the public sector.– 02 Roundtables were held by ENAP (competency-based management

and distance leaning in public organizations), resulting in 02 publications with 1,480 copies distributed and made available for download onENAP's website. From March to August 2007, there were 2,020 downloads of the two books, which indicates they are of great interest of Brazilian executives.

Simple but well structured approach to achieve a publishable knowledge on a specific topic related to public management.

“”Coordinator General for Distance Learning – ENAP

2. ENAP Incorporation of new methodologies

• Café com debate (Armchair discussion)– Inspired by the Armchair Discussion model adopted by CSPS, this event

has become a regular activity throughout the year which is held at a time that does not interfere with the professional activities of its participants and encourages executives to come to the School. The Armchair Discussions are always held from 8:30 to 10:30 a.m. in an informal atmosphere. The lecturers are experts, academics or governmentalauthorities and the topics to be discussed are selected from thegovernment agenda. Over 1,000 civil servants have participated in 26 Cafés com Debate since 2003.

• New instructional technologies– New instructional design approaches for classroom and e-learning

courses have been incorporated by ENAP and have benefited 20,000Brazilian civil servants in all ministries of the federal publicadministration and partner schools already.

2. ENAP Capacity to attract executives

• The adapted courses brought a new audience (executives) to ENAP by combining state-of-the-art contents, innovative methodologies and renowned experts in a space for peers to learn and exchange ideas.

• The highly positive evaluation of the courses (average approval of 85%) has been consistently attracting new executives to attend them and generating new institutional demands for closed classes, such as:– Course “Leadership: Reflection and Action”: Brazilian Health

Surveillance Agency (4 classes)– Course “Managing Human Performance”: Institute for

Historical and Architectural Properties (IPHAN), Ministry of Education, National Found for the Development of Education (FNDE), Military Public Prosecutor's Office, Federal Court of the Federal District.

2. ENAP A space for developing public executives

• Positive image of ENAP, perceived by the federal administration and partner schools as a locus for learning and exchanging ideas

• Dynamism, proactive approach, innovation, generosity (culture of sharing knowledge)

• Qualification to meet customized demands • Products and services developed in the areas of leadership

and human resources management that can integrate training programs demanded by public institutions (ENAP’s know-how associated to the customer’s needs)

• Segmentation of audiences and differentiated training strategies

Recognition

“ I was not very familiar with ENAP and I must confess that I thought that it was not the dynamic, proactive space that it actually is. I thought that it was a more conservative, bureaucratic institution. I was very surprised with what I saw here in the past three years.

Its concern with organizing spaces for cooperation, sharing knowledge, and integrating both domestic and international partnerships fosters the evolution of our public institutions.

”Executive of the Federal Data Processing Service – SERPRO. Participant in the Action-Research Roundtable on Distance Learning in Public Organizations.

2. ENAP Contributions to the National Policy for Human Resources Development

"The competency-based management policy adopted by the Canadian government was an initial benchmark for our discussions. ENAP's mission already included the imperative of ‘building capacities in civil servants.’ Since the presidential decree for human resources development, a key need for the national policy is using Competence-based Management as a training model."

Helena Kerr do Amaral – President of ENAP

• Competency-based management– ENAP is in charge of developing training programs to qualify civil

servants to hold management and high advisory (DAS) positions and of coordinating managerial training programs for civil servants implemented by the other schools of government of the federal public administration.

– ENAP is a member of the Managing Committee of the National Policy for Human Resources Development and coordinates the system of federal schools of government.

– The cooperation with Canada improved ENAP's qualifications in this area, which today is key for the National Policy for Human Resources Development, ensuring a privileged position for ENAP to discuss training policies.

2. ENAP Limitations / Attention!

• The dynamic of holding an international activity requires complex preparations which involve the intense participation of the teams of both schools in the design of actions and adaptations to the needs of the Brazilian public administration. Therefore, in order to make the best use possible of the teams, it is necessary to avoid any agenda overlapping among ENAP's different units.

• The calendar of ENAP's international actions should continue to integrate the strategic planning process of the technical teams, so as to add value to the activities being developed by them and to avoid unforeseen activities or undue burdens.

• ENAP should be better prepared to take on administrative activities which are also being carried out by the technical teams, so that they may focus exclusively on using new knowledge and methodologies.

Outcomes

3. PARTNERSHIPS

3. Domestic and international partnerships

• Qualifying and strengthening local partners is one of the medium-term outcomes anticipated in the original project, as well as strengthening cooperation relations between Brazil and Canada.

• The activities which have been carried out with domestic and foreign partners contributed in different ways and degrees to the achievement of the following outcomes:

– Strengthening of the Partnership Program and of the National Network of Schools of Government coordinated by ENAP at the federal, state and municipal levels as a result of providing increasing opportunities for learning and exchanging new contents and learning methodologies and technologies, reinforcing ENAP's role to act more and more based on the logic of supporting the institutional strengthening of these organizations. Strategy to be adopted for guiding the new Project with Canada.

– Expansion of the network of ENAP and CSPS foreign partners on the five continents.

3. Domestic partnerships • Domestic partners were exposed to new methodologies particularly through

their participation in learning events held at ENAP – Brazil-Canada International Open Forum, International Seminar on Distance Leaning, meetings of the National Network of Schools of Government and meetings held as part of the ENAP’s Partnership Program. This also happened in research and exchange activities and in visits of Canadian experts to four schools of government in the North, Northeast, Southeast and South regions of Brazil.

• Based on the experience of ENAP in the Partnership Program and in the National Network of Schools of Government, the new cooperation project to be implemented in the 2007-2009 period is both a huge opportunity – for contributing toward the institutional strengthening of partners – and a great challenge – as new conditions must be created for carrying out international activities in different local realities and for building capacities to incorporate and adapt new learning and research contents and methodologies.

• Taking into account the work already carried out with these partners, there is an expectation and a desire to turn these schools into centers for disseminatingnew knowledge and good practices in their regions – “champion schools” –, expanding the reach of the Project and its benefits to other schools.

"We were so enthusiastic that we offered to hold the next International Seminar on Distance Learning in Curitiba – State of Paraná."

Carlos Homero Giacomini – President of the Municipal Institute for Public Administration of Curitiba

3. International partnerships • Contacts established in the many different activities which were carried

out over a six-year period with Canadian institutions, particularly the International Forum “To Learn, To Innovate, To Cooperate: a conversation among peers" and the International Seminar on Distance Learning– 73 executives from schools of government of 12 countries, besides

Brazil and Canada, attended the International Forum: Germany, Argentina, China, Spain, France, Guatemala, Portugal, Kenya, Dominican Republic, Russia, Sweden, and Ukraine.

– 72 participants from 35 different organizations, 28 of which were Brazilian organizations and 07 were international ones – Canada, Spain, France and Panama – attended the International Seminar on Distance Learning.

• ENAP adopted the strategy of fostering, whenever possible, learning events with the presence of different international partners, favoring the exchange of experiences among peers and paving the way for new cooperation possibilities for both ENAP itself and CSPS: – South Africa– Ireland (Global Engagement Network – GEN Program) – France– Spain

Unexpected outcomes

• As outcomes that had not been originally anticipated in the Project, the following ones should be highlighted:– Strengthening of ENAP's Distance Learning Program – Inclusion of new contents and methodologies in courses

of the Directorate for Professional Training– Greater integration among ENAP units and teams

Unexpected outcomes Distance Learning

ENAP's Program highly enhanced by the cooperation with CSPS over the past three years.

The reach of distance learning courses offered by ENAP (free of charge) was expanded:

• 49.5% of all training courses offered by ENAP are delivered in a virtual environment• 11,300 civil servants trained in 2006. In 2007, this figure is expected to increase to 17,000

Presentation by Mrs. Tarcilena Polisseni Cotta Nascimento, ENAP's Coordinator General for Distance Learning.

She will present the main outcomes of the Project in relation to ENAP's Distance Learning branch.

Unexpected outcomes Distance Learning

• New products developed:– 01 E-learning course of CSPS adapted and redesigned in a

more dynamic and interactive way. The new format of the Course “Passage to E-learning” was adopted by the Canadian School. This course was donated to other Brazilian government agencies, with the authorization from CSPS.

– Revision of distance learning courses in the light of the blended learning approach.

– Creation of a community of practice in connection with the ENAP course “Meeting Citizens’ Needs”

– Creation of a virtual community – Praxis, after the International Seminar on Distance Leaning.

– Reference material on distance learning in public organizations (translation of documents and instructional materials and donation of 11 books on the topic to ENAP'sLibrary)

Unexpected outcomes Distance Learning

• Creation of ENAP's Virtual School in 2005, with a strong contribution from the Blended Learning Center of CSPS

• Mapping of the profile of distance learning students • Informal consultations by the CSPS team to ENAP about

the revision made of the blended learning concept. • Holding of the International Seminar on Distance Learning

– Strengthening of the international network of partners– Incorporation of the Open Space Technology methodology

• Optimization of resources – The cost of an e-learning course per civil servant can be up to

¼ lower than the cost a classroom course.

Unexpected outcomes Professional Training

Because of a greater convergence between CSPS learning programs and ENAP’s Directorate for Managerial Development, the project covered continued education programs more intensively. However, Canadian contents and methodologies were seen to be applicable to courses developed within ENAP’s Directorate for Professional Training, resulting in:

• Incorporation of new modules and methodologies– Inclusion of a practical methodology for developing public

policies extracted from the CSPS course "Challenges in Public Policy Development: an overview" as a module in the career training for Specialists in Public Policies and Governmental Management (induction courses and continuous development course for career progression) and in specializations (latusensu) in public management.

Unexpected outcomes Greater integration among ENAP units

The project required an intensification of integrated actions among different ENAP units (international cooperation, managerial development, research, professional training, events, editing, library), which contributed to strengthening the capacity to work as a team and as a learning organization –an inherent challenge for a school of government.

Some figures...Quantitative Outcomes:

1,070 Brazilian civil servants trained, 631 of which were women and 289 were men*.1,034 Brazilian civil servants trained in Brazil and 36 in Canada.31 courses/learning events held at ENAP; 04 lectures delivered in partner schools in the north, northeast, southeast and south regions of Brazil; and 20 missions/courses held in Canada.04 CSPS classroom courses adapted by ENAP to the Brazilian reality and offered regularly by the School. Until August 2007, 23 classes had already completed these courses at ENAP, totaling 760 trained civil servants. 01 CSPS distance course was redesigned by ENAP and donated back to the Canadian School. Over 1,200 Brazilian civil servants have completed this distance learning course so far.New approaches to the instructional design of distance learning courses adopted by ENAP as a result of the cooperation with CSPS have benefited about 20,000 Brazilian civil servants in all ministries of the federal public administration.

*In two events held in 2003, no record by gender was made.

Some figures...Quantitative Outcomes:

05 Canadian executives visited 06 different Brazilian ministries and exchanged experiences with them.02 CSPS activities/methodologies were incorporated by ENAP: Armchair Discussion and Action-Research Roundtable. 26Armchair Discussions were held at ENAP, which were attended by over 1,000 civil servants. 02 Roundtables were held by ENAP, resulting in 02 publications, 1,480 copies of which were distributed and made available for download on ENAP'swebsite.11 CSPS publications were translated into Portuguese and published in 09 Cadernos ENAP on different topics related to public management, such as risk management, innovation, knowledge management, management of transitions in the labor force and distance leaning. In the first half of 2007 alone, 5,272 downloads of the 09Cadernos ENAP and the 02 books on Action-Research Roundtables were made.184 books were donated by Canada and incorporated into ENAP's Library.

Other aspects• Efficiency• Sustainability• Gender & Diversity• Lessons Learned

Efficiency• Management, monitoring and evaluation tools negotiated and agreed between the

Project management teams of ENAP and CSPS since the Project design phase, resulting in optimized resources, swift actions, correction of planned steps/activities, opportunities appropriately used and problem-solving. .

• Evaluation– Annual evaluation meetings to review anticipated activities and results

achieved with the aim of defining the action plan for the following year – Efficacy evaluation tools applied to the learning events (reaction and impact

reports)• Intensive optimization of resources (value for money)

– Non-anticipated activities were carried out without additional resources– Negotiations focused on reducing costs and on enhancing the quality and use of

Project resources in developing new activities– Solutions for transferring funds agreed upon with CSPS and the Embassy of

Canada, ensuring safer payments without delays• Information documentation

– Regular registration of course adaptation processes, facilitating their use by other ENAP units and partner schools: instructional design, methodological approaches, instructor training processes, reference bibliography, learning materials. Initial investments which facilitate the work of the team and improve ENAP actions with its clients

– Standards set for the tools and evaluation reports of the courses, learning events and technical missions to Brazil and to Canada, improving the monitoring of activities in the light of anticipated outcomes

Efficiency

“ The project achieved much more than anticipated with the original resources.

Ian Creery – Former Project Manager – CSPS

CIDA got excellent ‘value for money’ and many results achieved with modest funds.

”

The project experienced a good continuum of persons involved. Turnover was low.

ENAP was a very professional and dedicated partner.

The strong and collaborative relationships built and maintained throughout the project contributed significantly to overall success.

Hanny Toxopeus – General Director –International Cooperation and Strategic

Change Directorate – CSPS

Efficiency

“ The success achieved in the cooperation with Canada is largely due to the Project management tools which were built and agreed upon, to the mobilization and commitment of the teams involved and to the engagement of the partners. Cooperation is only valid when those involved are aware that the international partnership adds value to what we do and to our challenges.

The coincidence of the mission, vision and learning programs of ENAP and CSPS made it possible for the two schools to work and grow together in great understanding.

The opportunity to work with different teams of CSPS was one of the main achievements of the Project, both for their professionalism and respect for diversity and for the affection shown by them throughout these six years. ”

Cláudia Conde – Head of the International Cooperation Office – ENAP

Sustainability• Knowledge documentation and management make it possible to access

and disseminate good practices to other ENAP units and partners.• Regular sharing of knowledge and experiences in courses/visits to

Canada based on the workshops held at ENAP both for the technical staff of the School and for teams of beneficiary agencies.

• Regular offer by ENAP of new courses adapted under the cooperation project.

• Capacity to meet increasing demands for leadership and human resources management courses.

• Strengthening distance learning enhances opportunities for training civil servants – democratization of training opportunities:– In 2007, there are plans to train (in distance learning): 17,000 civil

servants in all Brazilian regions.• Capacity to develop links (Partnership Program and National Network

of Schools of Government) and share the capacities acquired under the cooperation with Canada with domestic partners:– ENAP is recognized as a partner that plays a major role in promoting

improvements in public management practices, particularly as a result of its role in building capacities in executives, decision-makers.

– Continuity of the cooperation with Canada involving schools of government located in the five regions of Brazil and ministries and Special Secretariats of the Presidency of the Republic.

Gender and Diversity• Achievements

– Strong participation of women in the learning events (68%)– Research on gender and racial diversity in the public service carried out, which was

used by ENAP and disseminated to ministries and partner schools.– Meeting to discuss the Research on Gender and Diversity: CSPS, ENAP, Special

Secretariats, Experts and Academics. – Meetings between Canadian and Brazilian executives to discuss gender and diversity

topics: “Meeting of Women in Inclusive Policies” Learning about the Canadian policy for diversity either in courses and specific visits to discuss

this topic or through CSPS instructional models had an impact on different ENAP teams:• Training instructors: "Reflections on Diversity and Inequality" workshop held with

instructors and course coordinators a offered by ENAP for the purpose of including the diversity dimension in courses and learning events held by the School

• E-learning: incorporation of regional diversity elements into the design of the course “Meeting Citizens’ Needs.”

• Specialization: course for the Ministry of Education – Workshop on Indigenous Education; and course for the Ministry of Health – Workshop on Health Care for the African-Brazilian Population.

• Survey: “Gender, race and leadership competencies in the Federal Civil Service” (Research published as a Caderno ENAP) and disseminated to the Federal Government and the National Network of Schools of Government.

• Challenges– Documenting the material produced in the Workshop “Reflections on Diversity and

Inequality” and disseminating it to schools of government.– Inclusion of the diversity dimension in ENAP's pedagogic project.– Offering training courses on the diversity topic on a regular basis for ENAP staff and

instructors.

Lessons Learned Learning

• Practical learning – The contact with the CSPS experience in building leadership in the

public service (through visits and participation in courses) led to new possibilities in terms of learning contents and methodological approaches for ENAP teams.

– Many interviewees praised the appropriateness of the topics addressed in the courses and their format, highlighting the sensitization to “what being a leader is all about, the opportunity to exchange ideas with Brazilian and Canadian peers, the importanceof fostering horizontal relations to mobilize teams around outcomes” as gains.

– As regards the relations between ENAP and CSPS teams, they mentioned the “confidence, clarity and professionalism established between them and the possibility of exchanging ideas and learning jointly” as highly positive factors.

Lessons Learned Learning• Multiplying strategy

Recruiting high level executives and managers to take part in the training activities favors a multiplication of the learning within the teams and the possibility of training new team members (recommendationfrom superiors).Making available more places in courses/events held in Brazil toparticipants from the same institution to favor the introduction of new post-training practices and changes in their organizations (so as to enhance their influence/impact of changes).

• It is important to be aware of differences between the public administrations of Canada and BrazilIt is important to prepare foreign and Brazilian executives for missions/visits by providing them with contextualizing and supporting documents/texts for them to understand how the public administrations and institutions of both countries (specificities and differences) operate and to adjust the development of products and services to the needs and practices of the beneficiaries.

Lessons Learned Learning

• It is important to always send two ENAP civil servants to attendcourses at CSPS, particularly for processes of adapting courses, which require more intense and longer activities. This is a strategy that favors the retention and dissemination of knowledge and a greater integration between ENAP's learning staff and research teams.

Lessons Learned Management

• The project was marked by excellent relations/interaction between ENAP and CSPS management teams in Brazil and in Canada, largely as a result of a clear definition if responsibilities, clear and ongoing communications between the parties, a management approach focused on optimizing outcomes and the joint definition of mechanisms to solve problems.

• Ongoing and swift communications with ABC and with the CIDA office in Brazil were extremely important.

• Focusing on building relations between the key actors is fundamental for ensuring their involvement in the activities from the outset.

• It is important to define demands precisely, so as to better adapt and meet the needs of the beneficiaries.

• The outcomes anticipated in the project should be seen as part of the responsibilities of the partners – making sure that they are included in their planning.

• Projects of this kind would benefit very much from having a steering committee.• The gender and diversity strategies should be defined with the support from key

actors and from an external consultant.• The evaluation component should be designed and have its strategies defined with

the support from an external consultant, who should be subsequently involved in building evaluation tools, monitoring activities and drawing up evaluation reports.

ThanksFor the excellent work carried out and the partnerships

which were established, ENAP would like to thank:

• The President of CSPS, Mrs. Ruth Dantzer, the Project management team, and the teams involved in carrying out its activities. Particularly:– Jocelyne Bourgon – President Emeritus of CSPS, Richard

Rochefort, Dan Burke, Jim Carruthers, Luc Rivard and Mathieu Marsolais, whose support and contribution were essential for implementing the Project in its first phase.

– Hanny Toxopeus, Ian Creery, Fida Geagea, Jeffrey Outram and Cintia Becker.

Thanks

• To CIDA/Canada:– Claude Beausejour, who introduced CSPS to ENAP– Susanne Laporte– J. Sutherland– Eliane Moser– Julie Bédard

• To the Embassy of Canada and to CIDA/Brazil:– Ambassador Guillermo Rishchynski– Louis Verret– Rémy Beaulieu– Particularly Simone Direito, who has worked with us in drawing

up and implementing projects with Canada in the past six years.– Dilene Nunes

Thanks

• To ABC/MRE:– Counselor Antônio Piras– Minister Paulo Wangner– Daniel Brasil– Vidya Moreira

• To the Embassy of Brazil in Canada:– Ambassador Waldemar Carneiro Leão Neto– Secretary Pedro Murilo Terra

Thanks

• To the President of ENAP, Mrs. Helena Kerr, for her belief in the value that international cooperation adds to developing the School and its partners.

• To the Directors of ENAP and their teams for their dedication, work, competence and professionalism.

• To the team of the International Cooperation Office for its commitment, dedication and unconditional efforts. In particular to those involved in the Canada Project: – Bruno Mello, Renata Carvalho, Jeovan Assis, Carolina Pagani

and Carla Miranda.

Thanks

• To the partner institutions, who enthusiastically took advantage of the learning opportunities which ENAP and CSPS offered them and inspired us to work more intensively in building a new Project.

Thanks

• To consultant Daniel Simião and to the BaastelGroup, particularly Philippe Bâcle and Evan Green, who helped us to document this final evaluation.

• To our dear translators, Luiz Vasconcelos and Hilda Lemos, who helped us a lot with their professionalism during these four years.

ENAP National School of Public AdministrationSAIS Área 2-A70610-900 – Brasília – DF, BrazilPhone: ++ 55 (61) 3445 7038/7012Fax.: ++ 55 (61) 3445 7167www.enap.gov.br