Learning Effects of Negotiation Simulations: Empirical

1

Learning Effects of Negotiation Simulations: Empirical evidence

from different cohorts

Morten Kallestrup

Department of Political Science and Public Administration, University of Southern Denmark

Email: [email protected]

Preliminary draft – Please do not cite

Draft paper prepared for presentation at the Annual Meeting of the Danish Political Science Association

October 27-28 2016

Abstract

Based on empirical evidence from negotiation simulations across various student populations, the paper

assesses and discusses the horizontal impacts of negotiation simulations across different cohorts of students

as well as cohort-specific impacts and their possible explanations. The paper builds upon assessment data

from a spectrum of students covering full-time university students (B.Sc. and M.Sc.); professionals/public

managers, who study part-time at the university (Professional Master of Public Management and

Professional Master of Public Governance), and; selected high school students (Academy for Particularly

Talented High School Students). The empirical data have been collected by the instructor via Shakespeak

and PollEverywhere (Student Response Systems) and electronically by the Faculty Secretariat during the

final course evaluations. The data comprise quantitative measures (satisfaction levels etc.) as well as

qualitative data on learning effects and the students’ experience of the negotiation simulations. Based on

the analysis of the empirical data, the paper discusses the overall impacts of the negotiation simulations on

different student cohorts’ motivation and learning.

Keywords: Negotiation simulation, cohort variance, learning, substantive knowledge, skill-building

2

1. Introduction

While serving as a government official for the Danish Ministry of Business and Growth from 2006 to 2013,

the author of this paper participated in EU Council negotiation simulations prior to taking part in several

real consultations and negotiations in various European Commission expert committees and Council

working groups. The negotiation simulations and the subsequent “real-world” negotiations in Brussels were

astonishing experiences for a young civil servant with extensive university experience, but much less “real-

world” experience.

The very first EU negotiation simulation in the Ministry’s training programme was personally illuminating. It

uncovered (at least subjectively) the lack of knowledge about practical politics and EU negotiations, despite

recently having obtained a PhD in public policy and administration with a specialization in the

Europeanization of national politics and policies. The discrepancies and the difficulties in coupling

theoretical knowledge of European integration and Europeanization with the actual carrying out of EU

negotiations in practice were suddenly and surprisingly exposed. Certain strengths and advantages were

however also revealed, in particular the ability to keep a good grasp of the situation during an enduring and

complex political decision-making process.

The first simulations that I participated in were a part of the general EU negotiation training program for

government officials (civil servants) in the Ministry. Later on, from 2009-2011, the Danish Government’s

preparations for the Danish EU Council Presidency in 2012 consequentially resulted in that all civil servants,

who were expected to play an important role during the Danish EU Council presidency, were also to receive

specific “presidency”-related negotiation training. These exercises aimed at improving our skills as

negotiators. The training was prepared and executed with a particular view to the specific presidency

obligations; in particular chairing of Council working party negotiations and the role of serving as the

Council’s representative and on-behalf negotiator in trialogue negotiations between the Council, the

European Parliament Committees, and the European Commission.1

During the preparatory negotiation simulations prior to the Danish EU Council presidency in 2012, the focus

was on advancing negotiation skills (and networking approach) based on earlier experience from other

1 Prior to university studies, I experienced the thrill of simulation exercises in military tactical simulations as an

integrated part of army officer training.

3

negotiation simulations as well as from taking part in real negotiations in various Council working groups

and expert committees.2

When returning to academia in late 2013, there was momentum for introducing active learning

approaches, and for the development and appliance of operational teaching tools that correlated with this

paradigm. Based on my experiences from the Danish public administration, I decided to test the

introduction of negotiation simulations in my courses when relevant (according to the learning objectives

of the course).

This paper evaluates the current state of affairs with regards to results, experiences, and outcomes of the

negotiation simulations conducted throughout different courses and with different student cohorts during

the period from the fall semester 2014 to the fall semester 2016.

The paper is structured as follows: the theoretical concepts of learning, motivation, and interest are

presented and discussed in section 2. The characteristics, principles, and aims of the conducted negotiation

simulations are outlined and elaborated in section 3. Analysis and discussion of the empirical data are

provided in section 4. Finally, conclusions are derived in section 5, in particular with regards to the

explanatory value vis-à-vis student’s motivation and learning.

2 During the Danish EU Council Presidency from January to June 2012, the negotiation skills trained throughout

negotiation simulations prior to the presidency period were thoroughly applied in practice and put to a “real-world” test. In my case due to my team’s responsibility for securing progress in the Council working party’s negotiations on the new Regulation on the European Standardisation System (Regulation 1025/2012). The Council and trialogue negotiations resulted in a final approval of the trialogue agreement by the COREPER in June 2012 after less than a year of negotiations following the publication of the European Commission’s proposal for a regulation. For an outline of the negotiations and decision-making processes leading to the standardisation regulation, see Kallestrup (2015).

4

2. Negotiation simulations – a pedagogical tool fostering learning?

Simulations are today a widely applied pedagogical tool in many disciplines, including political science

(Boyer & Smith 2015). The purposes of applying negotiation simulations in political science courses seem

however to vary from university to university and from course to course. Yet, according to a recent meta-

analysis by Baranowski and Weir (2015), there are three fundamental simulation goals. Firstly,

enhancement of student learning. Secondly, enhancement of student engagement. And thirdly, challenging

student attitudes or raising awareness of issues (Baranowski & Weir 2015, p. 393). While measurable

engagement is undoubtedly increased with activation of students through simulations (see empirical data

outlined below), this paper mainly deals with the first goal, namely the element of enhancing learning.

2.1 Learning

According to Raymond and Usherwood (2013), simulations are assumed to foster learning in three ways: 1)

through improvement of students’ motivation (extrinsic and intrinsic); 2) through alteration of the learning

environment; and 3) through “gains in substantive learning about external processes or phenomena

(content knowledge) or the self (affective knowledge)” (Raymond & Usherwood 2013, p. 160).

Several studies point to positive substantive learning through negotiation simulations. “Reviewing social

science research, (…) find that engaging in simulations improves students’ motivation and retention of key

concepts that have already been taught in lectures and classroom discussions” (Schonk 2014, p. 3). And:

“Indeed, empirical studies consistently demonstrate that simulation training is an effective means of

teaching individuals a broad range of motor and cognitive skills. (…) More specifically, research from the

law enforcement domain indicate that the use of force simulators can serve as effective instructional

devices, regardless of the simulator used or type of officer in training. In fact, simulator training often

appears to be more effective than alternative approaches” (Bennell & Jones 2004).

Skill-building is also often emphasized as a main aim of negotiation simulations: “In addition to helping

highlight particular negotiation dynamics or concepts, artifice [through simulation] can help learners isolate

and develop particular behavioural skills…” (Crampton & Manwaring 2014, p. 9). One example of a

deliberate quest to develop skill-building rather than imparting substantive knowledge is Ebner and Efron

(2005), who introduced the “pseudo-reality” method, including a systematic approach towards developing

such simulations based on a seven-step model (see e.g. Asal & Blake (2006) for another systematic

5

approach and framework focusing on the choice for educators and instructors between educating students

about process or content).

According to Jones and Bursens (2015) empirical evidence also proves that participation in simulations

result in student gains in affective learning (effects on emotional and belief systems, e.g. attitudes towards

political systems, appreciation for political actors, motivation, and capability of self-assessment), such as

the ability for accurate self-assessment. Jones and Bursens found, among others, that while older students

“showed a greater ability to accurately judge their own levels of preparation”, youngers students’

“demonstrated an increased ability to judge their level of work” (Jones & Bursens 2015, p. 10).

The learning objectives (see table 1 below) of the courses referred to in this paper, in which negotiation

simulations have been applied as a pedagogical tool, do comprise substantive learning requirements as well

as skill-building requirements. Affective learning is less so assessed, however, nonetheless still relevant in

many instances.

In general, the learning objectives aim at that students are trained in critical reflection on different options

for applying theories and methods onto the analysis of political decision-making processes, and to make a

decision based on their critical reflection. Henceforth, it is not enough for the students that they know the

theories and methods (substantive knowledge) or can apply the theories in an analysis (analytical skills).

They must also show ability to reflect upon the relative strengths and weaknesses of theories and methods

vis-à-vis a given case, and subsequently make an argued selection of the most relevant theory (theories),

models, subjects and method(s). Hence with regards to level of competence, the requirements are rather

high vis-à-vis classical cognitive taxonomies (e.g. Blooms taxonomy or the SOLO-taxonomy). Such advanced

cognitive competences are usually assumed to be best acquired through appliance in practice rather than

through passive learning.

Besides learning effects in students’ substantive knowledge and skill-building, the analysis of the empirical

data from the negotiation simulations outlined below, this paper will also discuss impacts on student

motivation (extrinsic and intrinsic) and issue interest, and to a lesser degree performance, rf. below.

2.2 Methodology in negotiation simulation assessment

Recent assessments of the scientific literature on the application of negotiation simulations as a

pedagogical learning tool in political science point to a need for moving beyond the “show and tell”

approach (Baranowski & Weir 2015). Instead, instructors need to apply a more rigorous approach in the

6

assessment of learning outcomes (Raymond & Usherwood 2013, Asal et al. 2013). Simulations are

frequently claimed to lead to gains in substantive learning, though too often without being sufficiently

investigated (Raymond & Usherwood 2013). Learning objectives and assessment both need to be

considered carefully when planning the simulation. When designing simulations and assessments of the

simulations, the instructor needs to identify whether a simulation should produce positive changes in

students’ substantive knowledge, skills and/or affective characteristics (Raymond & Usherwood 2013, p.

157). Firstly, it has to be made clear what the simulation is intended to achieve (substantive knowledge

acquisition; skills development; group socialization). Secondly, the simulation has to be clearly aligned to

the assessment regime, meaning that it ought to be feasible to achieve both the learning objectives as well

as the proposed assessment.3 Thirdly, a strong element of debriefing/feedback has to be included, in order

to reach the full benefits of post-simulations reflection (Raymond & Usherwood 2013, p. 162-163).

While the intention and aim of the simulation as well as the debriefing and feedback element have been

consistently applied, the character of the assessment has developed throughout the negotiation

simulations that form the empirical basis for this paper. The first simulations lack explicit measurement of

e.g. performance, while the latter simulations are closer to the alignment ideal mentioned above, thus

applying pre- and post-testing of the students. The paper will derive and discuss some of the consequential

inferences in the final and conclusive section.

3 In accordance with the ‘constructive alignment’ approach (Biggs & Tang 2007).

7

3. The applied negotiation simulations and courses

”In almost any class, role plays (brief discussions in which members of the class are asked to

take on the role of a participant in a particular policymaking situation) and simulations (more

extended discussions in which teams of students take on a role and teams interact with each

other) help to make the strategic calculus and the constraints on political and policy

interaction come alive for the students. (…) “Real-time” simulations (…) can be particularly

engaging for students (…). Students have an opportunity to test their strategic decisions

against those of real politicians and policymakers, and developments in real-time can serve

as a common reference point through the semester that links later discussions back to

politicians’ decisions”. (Weaver 2014, p. 2)

The pedagogical argument and point of departure for allocating time to the appliance of negotiation

simulations in the below-mentioned courses was to strengthen the achievement of learning objectives by

a) providing a link to the relevance for practice (”real-world” relevance); b) introducing activating teaching

and active learning; and c) helping students’ improve their substantive learning and skill-building through

appliance of the theoretical models on a ”real-world” case, followed by an oral discussion and reflection

during the debriefing session (in which the students can reflect in a safe learning environment).

The applied negotiation simulations, which form the source for the empirical data of this paper, have all

been very similar and based on a classical “Lewin-inspired” simulation set-up. All simulations have thus

included the following phases:

1) Student preparation prior to the negotiation exercise; either through desk studies of relevant websites

or through the reading of compulsory and supplementary literature.

2) Instruction to the negotiation simulation (by instructor), including a presentation of relevant

negotiation theory and techniques.

3) Distribution of individual role descriptions + individual preparation.

4) Negotiation simulation game.

5) Debriefing and oral evaluation.

The amount of time allocated to each phase has differed from simulation to simulation, due to the great

variation among the related courses with regards to purposes and student cohorts. For a detailed outline

8

and comparison of the different simulations applied throughout courses from 2014-2015, see table 1

below.

As it appears from table 1, negotiation simulations have been applied as a pedagogical tool in 8 different

settings/courses (N=8) with various student cohorts at different career levels. Though similar in their overall

structure, there is notable variance in learning objectives; in time allocated; and in the evaluations of the

simulations. For five simulations (ATU 2015; MPM 2014, 2015, 2016; MOL 2016) the subject and

negotiation theme was “the annual national budget negotiations in Denmark”. For one simulation (M.Sc.

2015) the theme was “national pension reform”, and for the two thematic outliers (B.Sc. 2015, MMA 2016)

the subject was “European Union budget negotiations”, framed as Council working group negotiations, yet

also including representatives from the European Parliament and the European Commission in the

negotiation setting. All negotiation simulations were language-wise in Danish, except for two (M.Sc. 2015,

MMA 2016), which were conducted in English, due to the international cohort of students (see table 2

below for characteristics about the different student cohorts).

The instructor’s explicit success criteria for the negotiation simulations were that at least 66 percent of the

students would describe their general experience as overly positive (criteria based evaluation) and that

individual student statements to open questions from at least two thirds of the students would result in

overly positive statements (recipient based evaluation). For the 2016 simulation, the aim was furthermore

to measure learning effects based on pre- and post-testing of students. For further information on types,

length, and participant and instructor roles in the different simulations, see table 1.

9

Table 1: Negotiation simulations, 2014-2016 – game characteristics

Simulation games (N=8)

ATU 2015

B.Sc. 2015

MMA 2016

M.Sc. 2015

MPM 2014

MPM 2015

MPM 2016

MOL 2016

Learning objectives of negotiation simulation

To improve the students’ understanding of negotiation techniques through (simulated) practical appliance

To improve the students’ understanding of EU institutions and decision-making processes through (simulated) practical appliance

To provide a stimulating learning environment as a supplement to classical lectures in other courses

To improve the students’ understanding of decision-making theories and negotiation techniques through (simulated) practical appliance

Subject/theme National budget for 2016

EU budget (Council negotiations)

National pension reform

National budget for 2015



Type Real-life/ simultaneously

4

Fictional (aimed at improving understanding of multi-complexity in EU negotiations)

Fictional simulation (real-life inspired)

Real-life simulation

Length/time 4 hours (+ individual preparation time prior to the negotiation game

Individual/teambased preparation based on instructions + 2 hours negotiation game/debriefing

2+4 (individual preparation time in between)

4+4 hours (informal negotiation time in between)

3+4 hours

4+4 hours (informal negotiation time in between)

Participant roles

Two ministers, two deputy ministers, party leaders/ budget negotiators, and press and stakeholder representatives

European Commission representatives, European Parliament representatives, and Member-State delegations

Two ministers, two deputy ministers, party leaders/budget negotiators, and press and stakeholder representatives

Two ministers, deputy minister, party leaders/ budget negotiators

Instructor role Political support base (game role) Facilitator Evaluator

Council presidency (game role) Facilitator Evaluator

Political support base (game role) Facilitator Evaluator

Simulation game evaluation

Written survey + ATU lesson evaluation

Oral surveys + final course evaluations

Pre- and post-testing

Oral surveys + final course evaluations

Written and oral survey

Pre- and post-testing

4 The real-life deal on the annual national budget for 2016 was agreed at the Ministry of Finance a couple of hours

prior to the simulation game on 17 November 2015.

10

The various student cohorts (8 different student populations) comprised a) a cohort of selected high school

students from the Academy for Particularly Talented High School Students (ATU 2015)5; three cohorts of

full-time university students, respectively: B.Sc. students in an elective course on “Europeanization and

differentiated integration in the EU” (B.Sc. 2015), M.Sc. students studying Comparative Public Policy and

Welfare Studies (M.Sc. 2015), and B.Sc. students studying Market and Management Antropology (MMA

2016). Finally, the collection comprised four cohorts of professionals/public managers, who study part-time

at the university in the study programme on the Professional Master of Public Management, respectively:

in the fall of 2014 (MPM 2014), the fall of 2015 (MPM 2015), the fall of 2016 (MPM 2016) and in the study

programme on Professional Master of Public Governance in the fall of 2016 (MOL 2016).

Table 2: Student cohort characteristics

Student cohorts (N=5) ATU 2015 B.Sc. 2015

MMA 2016

M.Sc. 2015 MPM 2014

MPM 2015

MPM 2016

MOL 2016

Type of students High School Students

(extracurricular activity)

B.Sc. students (full-time students)

M.Sc. students (full-time students)

Public Sector Managers (part-time professional master students)

Students’ nationality Danish International Danish

No. of simulation participants (total student population in related course)

17 (21) 21 (23)

23 (30) 9 (9) 9 (10) 10 (11) 32 (39)

The courses in which the negotiation simulations were applied as a pedagogical learning tool comprise a) a

single four-hour lesson for the selected high school students from the Academy for Particularly Talented

High School Students, b) an elective course on “Europeanization and differentiated integration in the EU”

for B.Sc. students, c) “Blended lessons” for M.Sc. students as a supplement to the other compulsory

courses, d) an elective course on “Political decision-making processes” for professional master students,

respectively in 2014, 2015 and 2016, and e) an elective course on “Strategic Negotiation” for professional

master students. For further information on the different courses, see table 3 below.

5 The students are selected by the ‘Academy for Particularly Talented High School Students in Southern Denmark’

based on specific selection criteria covering certain horizontal and vertical cognitive skills.

11

Table 3: General course characteristics

ATU 2015

B.Sc. 2015

MMA 2016

M.Sc. 2015

MPM 2014

MPM 2015

MPM 2016

MOL 2016

Course title Lesson for The Academy for Talented High School Students

“Europeanization and differentiated integration (EU)”

“Blended lessons” (for M.Sc. students)

“Decision-making processes in the public sector”

“Strategic negotiation”

Aim and learning objectives

To facilitate room for academic disciplines that helps the students expand their disciplinary horizon and apply their knowledge and competences across a wide spectrum of disciplines.

That the student demonstrates ability a) to present and discuss literature and theories of EU governance and Europeanization, b) to apply relevant theories in analysis of concrete cases, c) to analyse EU governance dynamics and the consequences.

The learning objective is to give the students the opportunity to apply theoretical knowledge to practical situations and combine knowledge from different courses.

The purpose is to provide the students with knowledge and ability to analyse political and administrative decision-making and negotiation processes and their background, development, and consequences. The aim is that the students improve their understanding and ability to participate in political and administrative decision-making processes and negotiations.

Didactical tools

Negotiation simulation

Lectures, student presentations, negotiation simulation

Lectures + compulsory negotiation simulation

Lectures, exercises, negotiation simulations

Course evaluation

6

Written survey and ATU lesson evaluation

Written course evaluation Oral evaluation

Written course evaluation

Oral evaluation and written course evaluation

Exam type No exam (extra-curricular activity)

Written 2-days take-home assignment/ Seminar paper (20 p.)

Written 2-days take-home assignment / Seminar paper (20 p.)

Written 2-days take-home assignment7

Exam subject/ theme

No exam (extra-curricular activity)

Comparative public management / Selective

Course-specific

8

National budget negotiations in 2014

9

National budget negotiations in 2015

Still unknown

6 Final course evaluations are organized and distributed at faculty level (by the Faculty of Business and Social Sciences)

and as such out of reach for instructors with regards to content and structure of the evaluation. 7 Exam theme similar to negotiation simulation theme

8 Causality vis-à-vis simulation is very fragile.

9 Exam results proved significantly above average.

12

4. Empirical data and analysis: learning, motivation and interest

The empirical data from the negotiation simulations have been collected by the instructor via Shakespeak

and PollEverywhere (Student Response Systems) and electronically by the Faculty Secretariat during the

final course evaluations. The data comprise certain quantitative measures (satisfaction levels etc.) as well

as qualitative data on the students’ experiences of the negotiation simulations. As it appears from table 4

below, the success criteria for the simulations – measured on quantitative course satisfaction levels – have

been met as all measurable satisfaction levels reach somewhere in between 71 and 100 percent student

satisfaction depending on the issue in question.10

Table 4: Quantitative data from course evaluations

ATU 2015 B.Sc. 2015 M.Sc. 2015 MPM 2014 MPM 2015

Course title Lesson for The Academy for Talented High School Students

“Europeanization and differentiated integration

(EU)”

“Blended lessons”

(for M.Sc. students)

“Decision-making processes in the public sector”

Type of students High School Students (extracurricular activity)

B.Sc. students (full-time students)

M.Sc. students

(full-time)

Public Sector Managers (part-time students)

Population 17 23 30 9 10

Course evaluation by students (anonymous)

11

100 percent state that their general impression / experience was that it was “great”

71 percent

100 percent 100 percent state that they are satisfied or very satisfied with learning activities and the instructor’s pedagogical and professional level.

…state that tools and learning activities to a large or very large extent have contributed positively to understanding the course substance.

79 percent 84 percent

…state that activities to a large or a very large extent have motivated to active participation.


…state that the instructor to at large or very large extent has been good at supporting and helping develop their understanding of the substance.


…state that they to a large or very large extent have been satisfied with the substance of the course.

Response rate (out of total student population)

100 percent 61 percent 43 percent 100 percent 40 percent

10

Evaluations of the 2016 cohorts are not yet available. 11

Not necessarily a causal relation between course evaluation result and the students’ assessment of the negotiation simulation itself.

13

3.1 Learning

As mentioned above, the learning objectives of the courses in which negotiation simulations have been

applied as a pedagogical learning tool do comprise substantive learning requirements as well as skill-

building requirements. Affective learning is less so assessed, however nonetheless still relevant in many

instances.

According to the students themselves, they generally “learn a lot” during a negotiation simulation. In

particular with regards to elements dealing with how to negotiate in practice as compared to learning

about negotiation theory in the lecture hall and as a general occasion for reflection on their personal

practices. For a number for general statements, see box 2 below.

They also express that they recognize issues they had not anticipated before participating in the

negotiation simulation (similar to the author’s personal experience when he participated in the first

negotiation simulation, rf. the introduction) and they experience how difficult it is to negotiate in the “real-

world”. Please consult box 2-4 below for individual student statements and answers to different evaluation

questions relevant for the learning issue.

Box 1: A selection of general student statements

Cohort

General student statements on the negotiation simulation

The negotiation simulation contributed significantly to my learning about

power relations, alliances, and roles in a negotiation situation.

Very gaining and fun exercise – and food for thoughts with regards to my

own actions.

Interesting and different experience. It gave me time to reflect upon

negotiation in practice and in particular on the fact that very much takes

place outside the meeting room.

I experienced how chaotic and difficult it is to manage a negotiation process.

MPM 2014

The policy simulation was an excellent idea. It was a lot of fun and very

relevant at the same time.

M.Sc. 2015

14

Box 2: Learning

Evaluation question

Student answers Cohort

“If any, what did you learn from participating in the negotiation simulation?”

What happens during a negotiation, better understanding of the power game.

The importance of alliances.

The importance of alliances.

The political process during annual budget negotiations.

The pressure.

How compromises are made/the necessity of compromise.

A lot about real negotiations.

To keep opportunities open and to understand what the decisive mandates

want in order to be able to please them.

How real negotiations take place – practicable and applicable in my future.

Everything can change quickly.

How real negotiations take place.

What is actually required by a deputy minister during negotiations and how

the media are used for different purposes.

To achieve compromise, yet not to reveal your real position.

Negotiation methods and what barriers you experience during negotiations.

ATU 2015

A lot – unpredictability, strategy and communication is very important.

Importance of negotiation fazes.

The importance of being prepared.

The importance of preparation and having cleared relations with others.

How difficult it is to predict and execute negotiations.

Positioning, power, and administration of power.

Everything is not as it immediately appears.

Limited.

MPM 2015

15

Box 3: Difficulties and challenges in the negotiation simulation

Evaluation question Student answers Cohort

“What was particularly difficult or challenging, if anything?”

To keep a good grasp of the situation as it unfolded.

Role preferences.

The lack of knowledge/insecurity.

Difficult due to severity of information.

The risk of disclosure and the importance of mandates.

To be an opposition party representative.

To keep grasp of nuances.

To be put on the side-line during the negotiations.

To negotiate for real.

To convince and to make compromises.

ATU 2015

Preparation and strategizing.

Lack of knowledge, knowledge of party politics.

We needed a timeout during the simulation.

Time pressure and lack of knowledge of other participants’

objectives.

Lack of political knowledge.

Negotiation positions/roles.

MPM 2015

However, the researcher must ask oneself whether “perceived” learning actually corresponds to real

learning? If student statements are assumed to be a somewhat reliable indicator, the statements definitely

indicate a feeling or an experience of learning. But whether this feeling or impression of the student

actually results in real learning can only be assessed and measured through thorough post-simulation

testing. According to preliminary evidence from pre- and post-tests of the 2016 cohorts, this seems to be

verified, at least to a certain extent. It is, however, recognized that the question of whether the students’

own assessments of learning through negotiation simulation is a sufficiently reliable indicator, is a question

that is subject to controversy.

16

3.2 Motivation

As it appears from box 5 below, individual statements definitely prove the prevalence of experiences and

impressions by the students that result in very positive proverbs (“exciting”, “worthwhile”, “fun”), when

they are asked to describe their subjective experience and impression with the negotiation simulation. Due

to the extensiveness – both in amount and substance of the proverbs – there is no reason to doubt the

sincerity and thus reliability of the statements.

Box 4: Individual student statements on their general experience with the negotiation simulation


“Please describe your experience of the negotiation simulation (e.g. 3 keywords)”

Brilliant, immensely interesting, realistic – super experience (felt

almost like real negotiations!).

Fun, worthwhile, alternative.

Fun, inspiring, good learning.

Exciting, dynamic, challenging.

Interesting, entertaining, a worthwhile experience.

Fun, worthwhile experience, very serious.

Fun, worthwhile, exciting.

Challenging, different, exciting.

Exciting, challenging, frustrating but fun.

Speedy, attention, creativity.

Challenging, exciting, enthusiastic.

Inspiring, entertaining, a worthwhile experience.

Fun, worthwhile, new/different.

Very worthwhile, challenging, stimulating.

ATU 2015

Demanding, but super experience and definitely worthwhile.

Exciting, challenging, worthwhile.

Intense, realistic, relevant.

Negotiation is difficult.

Occasion for reflection, fun, applicable.

Realistic, worthwhile, good experience.

Stimulating, worthwhile experience, occasion for reflection.

Exciting to experience how much takes place outside the meeting

room.

MPM 2015

17

So there are definitely convincing indicators of a clear stimulation of the students’ intrinsic motivation. But

once again we must ask the relevant control question: does intrinsic stimulation lead to learning? In this

dataset we do not have sufficiently reliable indicators to be able to answer that question. As stated by one

of the students during the following lecture: ”I was so tired when I got home after the negotiation

simulation last week. It was so intense” (MPM 2015). Yet, as stated by Baranowski and Weir: “While

enthusiasm can only help to engage students, it does not necessarily lead to learning” (Baranowski & Weir

2015, p. 399).

On the other side, a negative side effect of the high level of motivation associated with the negotiation

simulation may be that students may find the negotiation simulation much more fun and stimulating than

reading compulsory literature and attending classical lectures. As indicated by the oral statement by one of

the M.Sc. students: “I actually spent much more time preparing for the simulation game than on reading

the compulsory literature for my courses this week” (M.Sc. 2015). This has of course to be taken into

account, if the overall aim is to improve learning vis-à-vis the learning objectives.

3.3 Issue interest

One of the questions in the written evaluations of the negotiation simulations refers explicitly to whether

the students’ feel an increased interest for the issue in question (‘political decision-making processes’). For

student statements, see box 6 below. In general, a majority of the students answer this question very

affirmative. However, the measures are not as unambiguous as they were in the case of motivation. Once

again it seems that the students “feel” stimulated and that they feel they somehow “gain” from the

negotiation simulation, though, as discussed earlier, this “feeling” may not equal absolute learning.

18

Box 5: Issue interest


“Did the participation in the negotiation simulation increase your interest for understanding political decision-making processes?”

I already have a great passion for politics, but yes, I would like to

learn more.

Yes.

No, my interest in political processes is already high.

Already very interested in political processes.

It has become even greater.

Yes.

Yes!!

Yes.

Yes, but I already have great interest.

Definitely.

ATU 2015

Yes!

Yes, in particular the difference between formal and informal

negotiations.

Yes.

No.

Yes.

Yes, very instructive.

No.

MPM

2015

3.4 Performance

In this collection of simulations, performance has not been measured systematically (except for the

instructor’s observations during the simulation games). The only indicator – which may be spurious – is that

the exam results in MPM 2014 were somewhat above the average. But there is no control group to

compare with, so there is no way to control for spurious effects. Preliminary results from the pre- and post-

testing of the 2016 cohorts indicate that there are performance effects with regards to substantive learning

of the decision-making systems and processes that are objects in the various simulation exercises.

19

5. Concluding remarks

As provisionally stated, the aim of the paper was to assess and discuss the horizontal impacts across

different cohorts of students, as well as the cohort-specific impacts and their possible explanations.

Through the assessment and analysis of the empirical data, the paper also provided further elaboration of

the evidence of overall impacts of the negotiation simulations on student’s learning, motivation, and

interest.

According to the students themselves, they “learn a lot” during negotiation simulations, in particular with

regards to how to negotiate in practice as compared to receiving public lectures about negotiation theory

and practice, but also through the recognition of various issues not anticipated before participating in the

negotiation simulation. The question remains, however, whether student-perceived learning actually

corresponds to achieving the learning objectives of the course?

Stimulation of and effects on the students’ intrinsic motivation appeared high across all cohorts. Extrinsic

motivation is not relevant as such, due to the lack of direct causal links between the students’

achievements in the negotiation simulations and their course exam results, which provide the student’s

with a final mark.

With regards to issue interest, the results were also overly positive, though more mixed. Some students

indicated an increased interest in understanding political decision-making processes, while other students

did not.

On the question of performance, the empirical data available were insufficient to draw reliable conclusions.

There is, however, one indication, namely in the case of the MPM cohort in 2014, where there was

equivalence between the theme of the negotiation simulation and the exam subject – and the result was a

collective exam result above average. But the issue is too complex to assess on the basis of these data and

the risk of spurious effects is too high. Preliminary results from the pre- and post-testing of the 2016

cohorts indicate short-term performance effects, at least with regards to substantive knowledge (needs to

be further elaborated).

With regards to cohort-specific results, the co-instructor with the student cohort of international students

in 2015 took notice of the following “lessons learned” on behalf of the students: “a) the game starts long

before it formally starts, b) the national background of players influences their negotiating style, and c)

playing politics is fun”. While the first and third “lessons learned” are very much in accordance with

statements from students across cohorts (see boxes 2-5 above; and appendences 1 and 2), the second

20

“lesson” is unique to the international cohort of students. This points to another interesting observation,

namely whether students’ learning, motivation, and interest – and in the end, performance – differ due to

different cultural and national backgrounds? This paper does not provide sufficient evidence to this

question, but the question ought to be further explored in future research.

Another interesting observation deriving from the analysis of the cohort-specific issues is: to what extent

does the difference between negotiation simulation as an extracurricular activity versus as a curricular (and

thus exam-related) activity impact on the students’ learning, motivation and interest? It is often claimed

that the establishment of “a safe learning environment” is a prerequisite for learning. Others emphasize

that negotiation simulations are particularly successful, mainly because participants are taken out of their

daily risk-based settings during the simulation: This is the reason why students are willing to take risks

during the simulation exercise; risks they would otherwise not take (at least deliberately), because they

would run a risk of losing career opportunities, if failing in the attempt. Another question that ought to be

further explored in future research is what evidence or indicators do we actually possess with regards to

measuring difference in learning between courses with subsequent exams versus extracurricular activities?

Based on their meta-analysis reviewing relevant articles on simulation exercises Baranowski and Weir

(2015) provide a list of recommendations for simulation research design: apply pre- and post-assessments

(and delayed post-assessments); increase the “N”; create a control group; and avoid a “show and tell”

outline of the simulation (meaning that you have to include all relevant basic information about the

simulation in the evaluation) (Baranowski and Weir 2015, p. 401). All requirements are relevant and will, if

applied, most probably contribute to an increased methodological robustness of negotiation simulation

assessments. It should be noticed, however, that in many environments it is beyond the instructor’s reach

to address these requirements sufficiently in practice, often due to centralized course planning. Yet, the

requirements are preferable as ideals when planning and conducting the simulation. In an ideal world,

delayed post-assessments of how well the students’ deliver vis-à-vis learning objectives in real-world

negotiations after the training would be the all-best indicator.

21

References

Asal, Victor & Elizabeth L. Blake (2006). “Creating Simulations for Political Science Education”, Journal of

Political Science Education 2:1, p. 1-18, doi:10.1080/155112160500484119

Asal, Victor, Nina A. Kollars, Chad Raymond & Amanda M. Rosen (2013). “Editors’ Introduction to the

Thematic Issue: Bringing Interactive Simulations into the Political Science Classroom”, Journal of Political

Science Education 9:2, p. 129-131.

Baranowski, Michael K. & Kimberly A. Weir (2015). “Political Simulations: What We Know, What We Think

We Know, and What We Still Need to Know”, Journal of Political Science Education 11:4, p. 391-403.

Bennel, Craig & Natalie J. Jones (2004). The Effectiveness of Use of Force Simulation Training – Final Report.

Carleton University.

Biggs, J. and C. Tang (2007). Teaching for quality learning at university. Maidenhead Berkshire: Open

University Press.

Boyer, Mark A. & Elizabeth T. Smith (2015) “Developing your own in-class simulations: design advice and a

‘commons’ simulation example”, p. 315-326 in Eszter Simon, William J. Miller and John T. Ishiyama,

Handbook on Teaching and Learning in Political Science and International Relations. Edwar Elgar Publishing.

Crampton, Alexandra & Melissa Manwaring (2014). ”Shaping the context, meaning, and effectiveness of

negotiation simulations: teaching and training insights”, p. 2-10 in Harvard Law School Program on

Negotiation Free Report ‘Teaching Negotiations: Understanding the Impact of Role-Play Simulations’,

Harvard University.

Ebner, Noam & Yael Efron (2005). ”Using Tomorrow’s Headlines for Today’s Training: Creating Pseudo-

reality in Conflict Resolution Simulation Games”, Negotiation Journal doi:10.1111/j.0748-

4526.2005.00070.x

Kallestrup, Morten (2015). “Amidst a Plethora of Interests and Actors. The EU Negotiations of Regulation

1025/2012 on European Standardization”, The 20th EURAS Annual Standardization Conference Proceedings,

22-24 June 2015.

Jones, Rebecca & Peter Bursens (2015). ”The Effects of Active Learning Environments: How Simulations

Trigger Affective Learning”, European Political Science doi: 10.1057/eps.2015.22.

22

Raymond, Chad & Simon Usherwood (2013). ”Assessment in Simulations”, Journal of Political Science

Education, 9:2, p. 157-167.

Schonk, Katherine (2014). ”Make the Most of Your Negotiation Training”, p. 2-4 in Harvard Law School

Program on Negotiation Free Report ‘Negotiation Training’, Harvard University.

Weaver, Kent (2014). ”Using Simulations and Role-Plays in Teaching Public Policy and Public Management”,

unpublished manuscript. Georgetown University and the Brookings Institution.

23

Appendices

Appendix 1: Written student evaluation of negotiation simulation (ATU 2015)


Brilliant, immensely interesting, realistic – super experience (felt almost like real negotiations!). Fun, worthwhile, alternative. Fun, inspiring, good learning. Exciting, dynamic, challenging. Interesting, entertaining, a worthwhile experience. Fun, worthwhile experience, very serious. Fun, worthwhile, exciting. Challenging, different, exciting. Exciting, challenging, frustrating, but fun. Speedy, attention, creativity. Challenging, exciting, enthusiastic. Inspiring, entertaining, a worthwhile experience. Fun, worthwhile, new/different. Very worthwhile, challenging, stimulating.


To keep a good grasp of the situation as it unfolded. Role preferences. The lack of knowledge/insecurity. Difficult due to severity of information. The risk of disclosure and the importance of mandates. To be an opposition party representative. To keep grasp of nuances. To be put on the side-line during the negotiations. To negotiate for real. To convince and to make compromises.


Yes, what happens during a negotiation, better understanding of the power game. The importance of alliances. The importance of alliances. The political process during annual budget negotiations. The pressure. How compromises are made/the necessity of compromise. Yes, a lot about real negotiations. To keep opportunities open and to understand what the decisive mandates want in order to be able to please them. How real negotiations take place – practicable and applicable in my future. Yes. Everything can change quickly. How real negotiations take place. What is actually required by a deputy minister during negotiations, and how the media are used for different purposes. To achieve compromise, yet not to reveal your real position. Negotiation methods and what barriers you experience during negotiations.

24

“Did the participation in the negotiation simulation increase your interest for understanding political processes?”

I already have a great passion for politics, but yes, I would like to learn more. Yes. No, my interest in political processes is already high. Already very interested in political processes. It has become even greater. Yes. Yes!! Yes. Yes, but I already have great interest. Definitely.

“How do you experience a negotiation simulation as a pedagogical learning tool?”

Really great, but one needs to make sure that all participants are interested. Nice difference. Good experience! Very exciting and dynamic. Different view into the substance. Good tool to engage all students. Incredibly instructive and tangible. Good way to try theory in practice. Much better tool, interactive. In many ways better due to familiarization and active participation.

“How do you assess your level of learning through a negotiation simulation as compared to a classical lecture?”

Yes, definitely Different kind of learning experience. Higher! Definitely higher. Learnt about other aspects. Learnt more. Yes. Different. Definitely. Definitely.

25

Appendix 2: Written student evaluation of negotiation simulation (MPM2015)


Demanding, but super experience and definitely a worthwhile experience. Exciting, challenging, worthwhile. Intense, realistic, relevant. Negotiation is difficult. Occasion for reflection, fun, applicable. Realistic, worthwhile, good experience. Stimulating, worthwhile experience, occasion for reflection. Exciting to experience how much takes place outside the meeting room.


Preparation and strategizing. Lack of knowledge, knowledge of party politics. We needed a timeout during the simulation. Time pressure and lack of knowledge of other participants’ objectives. Lack of political knowledge. Negotiation positions/roles.


A lot – unpredictability, strategy and communication is very important. Importance of negotiation fazes. Yes, the importance of being prepared. Yes, the importance of preparation and having cleared relations with others. How difficult it is to predict and execute negotiations. Yes, positioning, power, and administration of power. Everything is not as it immediately appears. Limited.

“Did the participation in the negotiation simulation increase your interest for understanding political processes?”

Yes! Yes, in particular the difference between formal and informal negotiations. Yes. No. Yes. Yes, very instructive. No.

“How do you experience a negotiation simulation as a pedagogical learning tool?”

Very inspiring. Worthwhile, but it takes a lot of time. Incredibly contributing. Exciting and different. Good, vivid. Very instructive to experience theory in practice = learning. I prefer classical lectures.

“How do you assess your level of learning through a negotiation simulation as compared to a classical lecture (more or less)?”

Yes! More, but combination of teaching methods should be applied. Yes! Maybe – more exciting. More – but more time should be allocated during the debriefing to coupling theory with the experience. Yes. Less learning.

Documents

Learning Effects of Negotiation Simulations: Empirical