Evaluating Learning in the CIL Pre-departure Courses ... Evaluating Learning in the CIL Pre-departure

  • View

  • Download

Embed Size (px)

Text of Evaluating Learning in the CIL Pre-departure Courses ... Evaluating Learning in the CIL...

  • Evaluating Learning in the CIL Pre-departure Courses

    Department of Foreign Affairs and International Trade

    Ministère des Affaires étrangères et du Commerce international

    Centre for Intercultural Learning Canadian Foreign Service Institute

  • 1


    Every year, Canadian government departments and organizations prepare and send thousands of Canadians abroad to work in a wide range of capacities in both developing and developed countries. In this context, it becomes critical for all involved that an appreciation and respect for the many subtle but important differences between cultures be an integral component of such exchanges.

    Through the Centre for Intercultural Learning (CIL), the Canadian Department of Foreign Affairs and International Trade provides a variety of Pre-departure and Intercultural Effectiveness Courses to assist Canadians acquire the necessary knowledge, skills and attitudes to perform effectively while on overseas assignment, and to make the experience a positive and rewarding one for all concerned. These courses are tailored to the specific needs of different client groups, and include a range of short duration courses conducted either in-house or off-site or some combination of both. Courses typically last from between one to five days. Clients are also offered the option of day attendance or residential. Most courses are facilitated by an instructor in a group setting, while others also involve an individual, self-administered on- line learning component in addition to the training provided in the facilitated, group learning environment. Course content is based on the latest theory and research in the field, as well as components designed to address the specific learning needs of the particular client group.

    The Centre is internationally renowned for both its work in intercultural effectiveness training, as well as its significant contributions to ongoing research and theory development in the field. The research reported here builds on the earlier work of CIL researchers (e.g. Vulpe et al, 2001)

    1 and represents a systematic

    evaluation of Participant learning in its three and five day Pre-departure and Intercultural Effectiveness programs. This report also describes the overall evaluation approach being developed by the Centre, and highlights the initial findings of the evaluation research conducted over the past two years.

  • 1








    Evaluation Strategy: Theoretical Underpinnings


    The evaluation research conducted at the Centre uses an approach proposed by Kirkpatrick (1994) 2 for

    evaluating training programs. This model provides a useful conceptual framework for delineating the logic of any evaluation, and for determining which measures should be employed in the assessment of outputs and impacts at different points in time.

    The Kirkpatrick model defines four fundamental levels of evaluation through which we can understand the processes of learning, skill acquisition and attitude awareness and to ascertain whether these have occurred as a result of training. Kirkpatrick's four levels of learning are:

    1 Reaction Level

    2 Learning Level

    3 Transfer of Learning/Behaviour Level and

    4 Impacts/Results Level

    In assessing whether learning has occurred at each of these levels, Kirkpatrick has proposed four essential research questions to guide any formal evaluation exercise. These are:

    1 Were the Participants pleased with the training experience? (Reaction Level)

    2 Has there been a change in Participants' knowledge, skills and attitudes as a result of training? (Learning Level)

    3 Has the Learning that has occurred become manifest at the behavioural level? Has the performance or effectiveness of Participants improved has a result of training? (Transfer/Behaviour Level)

    4 Have the learning and behavioural changes which have occurred as a result of training translated into positive impacts on the sponsor organization? (Impacts/Results Level)

    The evaluation research reported here addresses the second question and level of learning identified in the Kirkpatrick model—i.e. Learning Level—and assesses changes in Respondents' knowledge, skills and attitudes following participation in the Centre's short duration pre-departure training courses. That is:

    Has there been a change in Participants' knowledge, skills and attitudes as a result of training received through the Centre's Pre-departure programs?

    Kirkpatrick Model

  • 1










    Later levels of evaluation described by Kirkpatrick, including Transfer and Impact Levels, were not addressed at this phase of the research strategy, but will be examined in later research designed to assess longer-term impacts.

    The second major theoretical platform used to inform the development of the Centre's evaluation strategy is Vulpe et al.'s research on the knowledge, skills and attitudes said to be characteristic of the interculturally effective person. This model proposes nine essential competencies thought to be critical to intercultural effectiveness, including:

    1 Adaptation Skills

    2 An Attitude of Modesty and Respect

    3 Understanding the Concept of Culture

    4 Knowledge of Host Country and Culture

    5 Relationship Building

    6 Knowledge of Self

    7 Intercultural Communication

    8 Organizational Skills

    9 Personal and Professional Commitment

    These competencies are described in detail in the 2001 publication, "A Profile of the Interculturally Effective Person" referenced earlier. At the Centre for Intercultural Learning, the specific learning points for all pre-departure courses are linked directly to these nine core competencies and proposed determinants of intercultural effectiveness.

    Profile of the Interculturally Effectiveness Person

  • 1





    This research was also informed by models of learning that propose that behaviour change occurs in predictable stages, with different learning outcomes and intervention strategies being associated with each stage. One model that has been used extensively in this regard is the Transtheoretical Model of Behaviour Change (TTM) developed by Prochaska and DiClemente.

    3 Originally developed to better

    understand long term change in health-related behaviours, it is now generally regarded as a useful integrative model by which to understand many forms of behaviour change, including at the individual, community and organizational levels.

    The model proposes that behaviour change is an ongoing cyclical process, which involves movement through a series of stages, including

    1 Precontemplation—a stage in which people have no intention of changing their behaviour in the foreseeable future, often because of an inability to recognize that aspects of their behaviour may be inappropriate or problematic.

    2 Contemplation—characterized by an awareness and recognition that there is a need to change one's behaviour to either avoid negative consequences or to derive some benefit.

    3 Preparation—during this stage individuals engage in some initial preparations for behaviour change (for example, acquiring information, talking with advisors, signing up for a course on the subject).

    4 Maintenance—successful behaviour change is said to have occurred when the personal has learned a new, more appropriate set of responses and behaviours, and this behaviour change has been sustained for a minimum of six months.

    In the transtheoretical model, the change process is seen as cyclical, involving periodic relapse and movement back and forth between stages until longer-term behaviour change is eventually achieved. The model also makes use of Bandura's social learning theory and in particular, Bandura's concept of "self-efficacy", defined as the belief that one has the knowledge and skills (or the ability to acquire the necessary knowledge and skills) to respond effectively to the specific challenges and demands of a given situation. (Bandura, 1977; 1982).


    Stage-based approaches to learning and behaviour change

  • 5

    Research has provided strong support for both the transtheoretical model and the concept of self-efficacy, and has also confirmed the temporal and sequential nature of learning and behaviour change. Stage-based models are now used routinely to inform the design and implementation of a wide range of behaviour change strategies, learning programs and other behavioural interventions, including intercultural interventions (e.g. Bennett, 1993).

    5 Such thinking is also consistent with the Kirkpatrick model of learning and evaluation

    of training, and the Vulpe et al. model of the development of intercultural effectiveness. A stage-based approach was therefore adopted in the assessment of the impacts of the CIL courses on Respondents' learning, including in the design and development of the questionnaire and item scales.

    A pre-test, post-test single group design was employed in this study. A matched control group would have been preferred in this initial phase of the evaluation strategy: However, it was not possible to incorporate a true control group at this point in the strategy because of ethical and practical considerations. To do so would have involved denying access to training to a significant number of Participants slated for overseas assignment and this was not acceptable.

    However, the desig