Rater Training Manual - INCODE

Helmut Helker, Christiane Stange

European Union Project 518132-LLP-1-2011-1-ERASMUS-FEXI

This manual has been funded with support from the European Commission.

INNOVATION COMPETENCIES DEVELOPMENT

Rater Training Manual

2

Rater Training Manual This Rater Training Manual presents an assessment training that was especially developed

for the use of the INCODE barometer, which has been developed in the INCODE project to

assess Innovation Competencies in students. The training aims at familiarizing future raters

with the application of the INCODE assessment tool on a practical level. The INCODE rater

training as a part of the INCODE project therefore assures the quality of the INCODE

barometer.

The manual at hand sums up experiences from rater training pilots with INCODE partner

countries. The main focus of the manual is to offer a step by step procedure to develop the

evaluation skills that are necessary to apply the INCODE barometer.

This first version of the manual should be regarded as a blueprint for a comprehensive

INCODE assessment training. Educators are encouraged to explore and apply the training in

their own work.

3

General Introduction – The INCODE Project

The general aim of INCODE is to facilitate the transfer from innovative ideas into innovative

products

• by integrating pedagogical knowledge into working-life innovation activities

and

• by enhancing innovation potentials in higher education institutions,

in 3 Steps:

1. targeting general features of innovation competencies

2. developing them in different Higher Education Curricula

3. assessing the Learning Outcomes with a special tool.

You find more information on the INCODE-project on:

www. incode- eu.eu

http://www.incode-eu.eu/

4

INCODE-Structure

The overall aim of INCODE is to develop Innovation Competencies in Students by means of

a new pedagogical method, the Research Hatcheries. The INCODE barometer was built to

evaluate the development and the performance of the complex cluster “Innovation

Competence” in students. For adequately using the barometer a Rater Training is an

indispensable component of the overall construction of the assessment design. In order to

apply the INCODE barometer in a way that is fruitful both – for students to have a most

differentiated feedback and for raters to produce most reliable results of their ratings – it is

necessary to practise the use of the instrument in advance. Therefore a special training

method has been developed in INCODE. It builds on findings of several studies, which

suggest that combinations of different training methods tend to yield the most effective

results [1, 2, 3, 4].

5

General Introduction – Rater Training Rater Training - General objectives A rater training should aim at these goals: Familiarization with measurement tool (ICB) Improvement of observation skills. Reduction or Elimination of judgmental biases. Improvement of evaluation skills Improvement of the ability of raters to give constructive feedback1 Screening of qualified raters2.

Character of the Training For our purpose we combined three kinds of rater training approaches: Behavioural Observation Training (BOT)

Raters get familiar with the complexity of observation processes Rater Error Training (RET)

Raters learn to avoid common rater errors Frame of Reference Training (FOR).

Raters get acquainted with relevant behavioural indicators of Innovation Competence Duration of the training Due to the combination of the different training models this rater training needs more than five hours time. Target Group Trainees are:

Teachers who will assess their students in research hatcheries and project groups. Students who will participate to be better prepared for self-assessment and peer-

assessment of their innovation competence behaviour in research hatcheries and project groups.3

External raters Group size: 6 to 10 participants.

1 Will not be part of this blueprint but will be added in follow-up versions 2 Will not be part of this blueprint but will be added in follow-up versions 3 A special training for self- and peer-assessment will be added in follow-up versions

6

Training Structure Time Focus and Content Learning Objectives /

Information Method

20 Introduction and Overview • Purpose of the project

• General Description of ICB

Participants get to know the necessary and basic facts about the project and the ICB

Lecture

20 General Information on Rater Training • Purpose

• Objectives, Elements, Target Group

• Agenda and methods of this training

• How results will be used

Participants get basic information about the Training

Lecture

5

Behavioural Observation Training (BOT 1)

• Differences in individual observation

Participants recognize differences in individual observation.

Short “spontaneous” role-play situation


• Psychometric Criteria of Assessment

Participants get acquainted with the concepts of objectivity, accuracy, reliability and validity of social perception

Lecture, hand-out, and Self-Evaluation of trainees` assessments

20


• First experience with rating and being rated

.

Participants experience the rating situation from the different perspectives of rater and ratee. Participants: - get to know the influences on

raters and ratees. - get to know the necessity of

unambiguous definitions

Two role-play-situations (Subject: Problem solving)

30 Behavioural Observation Training (BOT 4)

• Practice of observing and recording evidence

Participants differentiate between relevant and irrelevant information.

Short video scene: Video 4 (T-Shirt) Trainees write down relevant behavioural clues for one ICB-item, Comparison in mini-groups, Trainer feedback,

10 Behavioural Observation Training (BOT 5)

• Sources of possible observation problems

Participants get an overview of possible problems in observing behaviour.

Lecture / wrap up

15 Rater Error Training (RET 1) • Explanation of common

rater errors

Participants know the most common rater errors and can explain how these errors influence ratings.

Lecture, PPTs (attachment)

10 Rater Error Training (RET 2)

• Identification of common rater errors

Participants analyse their own rating behaviour and become aware of their own error tendencies.

Participants try to identify their own typical rater errors

7

Time Focus and Content Learning Objectives /

Information Method

15 Rater Error Training (RET 3)

• Possible error sources and typical biases and rating tips (do /don´t)

Participants know how to avoid rater errors

Wrap-up, Lecture. Hand-out

20 Frame of Reference (FOR 1) training

• Performance dimension training.

• Documentation Training

Participants have a good understanding of central concepts, indicators and dimensions of innovation competence. They are able to give examples of innovation competence behaviour. They know different aspects of documentation

Discussion of competences, indicators and dimensions in the ICB Focusing on behavioural examples of innovation competence


• Training with representative samples, practice, and feedback.

• Classification

Participants are able to assess short behaviour sequences correctly.

Short video scene: Video 5 (Cave) Identification of relevant behaviour (three ICB-items). individual ratings, group discussion, trainer feedback



• Classification, Weighting, Evaluation

Critical incidents representing good, average, and poor performance. Participants are able to assess critical incidents correctly.

Short video scene: Video 1 (Apple) individual ratings, group discussion, trainer feedback



• Classification, Weighting, Evaluation

Participants are able to assess short behaviour sequences correctly. They use group discussion and trainer feedback to reach a consensus.

Short video scene: Video 2 (Backpack) individual ratings, group discussion, trainer feedback


Norming

Participants are able to assess more complex behaviour sequences correctly. They use group discussion and trainer feedback to reach a consensus.

Longer video scene: Video 3 (T-Shirt) individual ratings, group discussion, trainer feedback

15 Certification Exercise 4 Raters must match the ratings of videos, observations, or artifacts done by expert jury in order to be allowed to do assessment in the field

individual ratings, trainer feedback

15 Evaluation and closing Flashlight / feedback

4 A Certification Exercise is necessary and will be added in follow-up versions

8

1. Introduction INCODE Participants are informed on the relation between Innovation Competencies, Research Hatcheries and INCODE-Barometer (see General Introduction this Manual (p.2f)) 2. Overview and General Information on RT Content: Purpose, objectives, elements, target group, agenda and methods of this training; how results should be used (feedback to ratees, coaching for performance improvement) Method: Lecture, Questions & Answers

9

3. Behavioural Observation Training In this part of the training the focus is on the process of observation, on the detection,

perception and recognition of certain relevant aspects of behaviour. Participants become

aware of sources of error that are detrimental for observation accuracy, such as loss of detail

because of simplification, categorization and contextual errors, prejudice etc. [4]. By this

training unit trainees should learn to exchange automatic categorisation processes by

controlled cognitive processes.

Behavioural Observation Training starts in an unheralded way with a (staged) conflict

situation between the trainers and some external person in order to base the learning

process on direct experience. After writing down their observations the trainees are asked to

compare their individual observations and to discuss possible reasons for differences.

3.1. Behavioural Observation Training (BOT - 1)

Content: Differences in individual observation Learning objectives: Participants recognize differences in individual observation.

They experience to what extent social perception is prone to distortions.

Method: Observation Task Task content: Trainees witness an unexpected conflict situation (spontaneous role-play) between trainers and one external person. (Alternative: short movie scene.)

1. Task for individual trainees: “Write down what you saw! Don´t talk, just write!” What happened?

Describe the acting persons

How did the external person look like?

After participants have recorded their observations they form mini-groups.

2. Task for mini-groups:

Compare your individual observations

Discuss possible reasons for differences

3. Sharing of experiences

10

3.2. Behavioural Observation Training (BOT - 2) Content: Presentation of different psychometric criteria Learning objectives:

Participants get acquainted with the concepts of objectivity, accuracy, reliability and

validity of social perception

Participants apply the criteria on the recording of their own perceptions (see task BOT-1)

Method: Lecture

self-evaluation of trainees` observations.

Handout (see attachment 1)

Task: Discuss how objective, reliable and valid your observation of the conflict situation has been.

11


Trainees should be able to experience the rating situation from the different perspectives of rater and ratee and recognize their mutual influences. Content First experience with rating and being rated Learning Objectives Participants experience the rating situation from the different perspectives of rater and ratee. Participants get to know the influences on raters and ratees. Participants get to know the necessity of unambiguous definitions of central concepts (here: “problem solving competence”) and the necessity of behavioural description of evaluation dimensions. Method Participants are given small problem solving tasks (see attachments 2 and 3). While two trainees work on the task their two partners respectively observe and assess their “problem solving competence” on a 10-point scale (see attachment 4). Subsequently they compare their individual ratings, reflect on possible reasons for differences and share their experiences of rating and being rated. Two role-play situations (“Problem-Solving”): 2 participants are problem solvers, the other two raters, changing of roles, change of problem. Time: 5 minutes each. It is not important that the problems are solved in this time. Task: Assessment, self-assessment, partner-assessment Comparison of individual ratings Comparison of rationale for self- assessment and partner-assessment Short reflection on reasons for differences

Assess the general problem solving competence in this special situation of yourself, your partner and the other two problem solvers on a scale from 1 to 10.

12


The focus of the next assignment for the trainees is on recognition of relevant aspects of

behaviour. While watching a short video sequence of three students working on a creativity

task, trainees have to write down relevant behavioural clues of one ICB-feature (out of 25) for

each of the three students in the video. Participants compare their results with each other

and with the results of expert raters.

Content Practice of observing and recording evidence. Identification of relevant information. Learning objectives: Participants differentiate between relevant and irrelevant information. They are able to take notes about relevant behaviour Material: Video 4 (T-Shirt) Task: Write down the relevant behavioural clues of the feature “takes initiatives” Participants write down short descriptions of relevant behaviour as objectively as

possible. Discussion of results in mini-groups Trainer feedback and discussion

Examples for trainer feedback: Relevant behaviour “takes initiatives”

Person A asks questions, criticises, makes proposals, implements own ideas, modifies ideas of others

Person B no own initiatives, only support by physical activity

Person C starts working process, initiates activities, implements own ideas, accepts the modifying ideas of others, criticises

13


Summary of Part 3 and transition to Rater-Error-Training Content: Sources of possible observation problems:

Loss of detail through simplification,

Selective perception,

Contamination from prior information,

Prejudice and stereotyping,

Observer influence.

Learning objectives Participants get an overview of possible problems in observing behaviour. Method: Wrap-up lecture (Part 3: Behavioural Observation Training)

14

4. Rater Error Training (RET) The Rater Error Training begins with a lecture on perception as a process by which humans

interpret their sensory impressions in order to give meaning to the environment [6]. This is

illustrated by visual illusions and pictures with ambivalent content. Subsequentlly the most

common rater errors are described, the trainees are confronted with the types of errors they

committed in the tasks before and they are instructed on how to avoid them.

4.1. Rater Error Training (RET - 1) Content: Explanation of common rater errors

Similarity

Leniency

Halo

Central tendency

Primacy

Context-/ Contrast effects.

Learning objective: Participants know the most common rater errors and can explain how these errors influence ratings. Method: Lecture, PPTs with interactive parts (see attachment 5).

15

4.2. Rater Error Training (RET - 2)

Content: Identification of common rater errors Learning objective: Participants analyse their own rating behaviour and become aware of their own error tendencies. Task: Try to identify your own typical rater errors using your first observation accounts and your first ratings (in pairs or individually).

4.3. Rater Error Training (RET - 3)

Content: Possible error sources and typical biases and rating tips (do /don´t) Learning objective: Participants know how to avoid rater errors Method: Wrap-up lecture, Questions & Answers Flipcharts/ pinboard or handout “Golden Rules” (see attachment 6).

16

5. Frame of Reference Training (FOR) This part of the training provides raters with a frame of reference for making evaluations of

the ratee's performance. The goal is to reduce arbitrary performance standards. Raters

discuss their own standards and implicit theories of performance in comparison with

normative standards developed by experts. The overall aim is to share a common

understanding of performance standards. Studies of FOR have shown significant

improvements in rating accuracy [1, 7, 8].

In the INCODE training concept FOR starts with a description of indicators and dimensions of

the ICB focusing on behavioural examples of innovation competence. In less complex

settings behaviourally anchored rating scales would be an ideal way of specifying

performance standards. Due to the complexity of the concept “innovation competence”, only

some representative examples for high and low performance on a few items are presented.

Raters are trained with video-vignettes that convey performance realistically [9]). Four

different videos are used, which present examples for different kinds of performance with an

increasing degree of difficulty. Raters are trained to document their observations

systematically (classifying, weighting) before evaluating. Raters receive feedback on the

discrepancies between their ratings and the target scores.

General remark: Depending on the assessment skills of the trainee group trainers should

choose appropriate grades of difficulty: e.g. certain parts of the ICB with the scale “not

observed, poor, needs to improve, pass, good, excellent” or the complete ICB with the 10-

point-scale. This FOR-training proceeds from an easy to an intermediate level and has to be

adapted according to the needs of the trainees.

17

5.1. Frame of Reference Training (FOR - 1) Content: Description of competences, indicators and dimensions of the INCODE-Barometer focusing on behavioural examples of innovation competence Learning objective: Participants have a general understanding of central concepts, indicators and dimensions of innovation competence. They are able to give examples of innovation competence behaviour. Method: Discussion of competences, indicators and dimensions in the INCODE-Barometer Focusing on behavioural examples of innovation competence Material: INNOVATION COMPETENCIES DEVELOPMENT – INCODE barometer and User Guide (see attachment 7) [10].

18

5.2. Frame of Reference Training (FOR - 2) Content: Classification: Raters allocate descriptions of behaviour to specific dimensions and items of the INCODE-Barometer. Learning objective: Participants are able to identify relevant behavioural clues in short sequences correctly. They use group discussion and trainer feedback to reach a consensus. Method: Individual ratings, group discussion, trainer feedback Task: Collect descriptions of verbal behaviour and assess the behaviour of each of the three persons in the video on the following items: „Faces conflicts with flexibility to reach agreements“ (item 20) „Applies ethical values“ (item 21) „Takes into account the implications of the task for society“ (item 22)

Task for mini-groups: Form mini-groups and discuss the results and compare them to expert solutions. Material: For assessing those items of the interpersonal and networking dimension a special setting is required. Video 5 (Cave) In the video three students are in the following role-play situation (see attachment 7): You belong to an advisory board, which is responsible for different social science research projects. Today you are called to an emergency meeting. One research project is in severe danger of ending in a catastrophe. Five persons live together for four days in a cave. Falling stones have blocked the exit, water is flooding into the cave and the water level is rising continually. Your Task: You have to decide in which order these people should be saved. The people in the cave are described as follows: Kate (34), married, 4 children, quit her studies in psychology for getting married, she has a secret affair with John Filiz (19), single, student, very attractive. Speaks English, French, Arabic and Turkish fluently. Very rich parents in Istanbul, has friends in the high society. John (35), single, teacher. Before that for 8 years officer in the army, brutal leadership style, posttraumatic stress disorder, drinks too much and likes to gamble.

19

Boris (56), divorced, internationally renowned scientist, works on radically new ways to cure breast cancer, started frequenting gay bars after his wife left him. Jussi (62), married, owner of a small IT-Company, holds a patent for a new kind of server, will bring 120 new and highly qualified jobs, donates money for charitable and cultural purposes.

Item Examples for descriptions of verbal behaviour relevant to items 20, 21, 22

ICB 20 (“Faces conflicts with flexibility to reach agreements”): Student C: “You and I had that, and she has it different.”

ICB 21 (“Applies ethical values”): Student B: “At the beginning we had Kate. She is a mother so we thought she kind of deserves it.”

ICB 22 (“Takes into account the implications of the task for society”): Student A: “So I think he can save many thousands or millions of lives of people.”

Expert solution: Student 1 Student 2 Student 3 ICB 20 8 7 8 ICB 21 6 5 6 ICB 22 8 6 7

20

5.3. Frame of Reference Training (FOR - 3) Content: Participants are able to assess more complex behaviour sequences correctly. They use group discussion and trainer feedback to reach a consensus. Classification

Raters allocate behaviour descriptions to specific dimensions and items of the INCODE-Barometer

Weighting Raters decide on frequency and /or intensity of the relevant behaviour. Evaluation Results are compared with the results of a second rater.

Learning objective: Participants are able to assess short behaviour sequences correctly. They are able to discriminate between the individual and the interpersonal dimension of the ICB. They use group discussion and trainer feedback to reach a consensus. Method: Individual ratings, group discussion, trainer feedback Task: Evaluate the video sequences on the ICB scale and record your justification for your

ratings, use:

ICB 2 (“presents creative ideas”) and

ICB 13 (“transmits ideas effectively”)

Discuss your results and compare them with expert solutions

Material: Video 1 (Apple) Scene can be shown twice Task for the student group on the video: “Do something inspiring with an apple, a lamp and post-its.” Expert solution: Student 1 Student 2 Student 3 ICB 2 4 9 5 ICB 13 2 4 3

21

5.4. Frame of Reference Training (FOR - 4) Content: Participants are able to assess more complex behaviour sequences correctly. They use group discussion and trainer feedback to reach a consensus. Classification

Raters allocate behaviour descriptions to specific dimensions and items of the INCODE-Barometer

Weighting Raters decide on frequency and /or intensity of the relevant behaviour. Evaluation Results are compared with the results of a second rater.

Learning objective: Participants are able to assess short behaviour sequences correctly. They are able to discriminate between the individual and the interpersonal dimension of the ICB. They use group discussion and trainer feedback to reach a consensus. Method: Individual ratings, group discussion, trainer feedback Task: Evaluate the video sequences on the ICB scale and record your justification for your

ratings, use:

ICB 6 (“Approaches the task from different perspectives”) and

ICB 9 (“Shows enthusiasm”)


Material: Video 2 (Backpack) Scene can be shown twice Task for the student group on the video: “Do something inspiring with a backpack, a binocular and a wristwatch.” Expert solution: Student 1 Student 2 Student 3 ICB 6 7 8 6 ICB 9 7 9 6

22

5.5. Frame of Reference Training (FOR – 5) Learning objective: Participants are able to assess short behaviour sequences correctly. They use group discussion and trainer feedback to reach a consensus. Task for the student group shown in the video: Do something inspiring with a t-shirt, 30 wine bottle corks and adhesive tape. Method: Individual ratings, group discussion, trainer feedback Task: Assess the behaviour in the video-sequence,

Compare your ratings with the ratings of your neighbours

Disscuss possible differences


Material: Video 3 (T-Shirt) Expert solution: Items which are not applicable: 11, 20 - 25 Student 1 Student 2 Student 3 ICB 4 7 9 7 ICB 8 6 8 6 ICB 10 10 10 10 ICB 14 10 10 10 ICB 16 10 10 10 ICB 19 9 7 5

23

6. Certification Exercise As Myford & Wolfe [11] state, even an extensive training will not guarantee that all

participants understand and agree with the standards. Consequently the INCODE rater

training concept should end with a certification exercise in which participants must match the

ratings of videos, awarded by a panel of experts in order to receive certification. In this exercise trainees must match the ratings of videos done by an expert jury in order to

be allowed to do assessments in the field.

7. Re-Training

The effects of training do not last over time [11] a re-training or at least periodic re-

calibrations, using e-learning modules e.g., are necessary.

24

8. References

1. Roch, S. G., Woehr, D. J., Mishra, V., Kieszczynska, U.(2012). Rater training revisited: An updated meta-analytic review of frame-of-reference training. Journal of Occupational and Organizational Psychology 85 (2), 370–395.

2. Bernardin, H. J., & Pence, E. C. (1980). Effects of rater training: Creating new response sets and decreasing accuracy and diary-keeping on psychometric error in ratings. Journal of Applied Psychology, 65(1), 60-66.

3. Sulsky, L.M. & Day, D.V. (1994) Effects of frame-of-reference training on rater accuracy under alternative time delays. Journal of Applied Psychology, 79(4), 535-543.

4. Thornton, G. C., & Zorich, S. (1980). Training to improve observer accuracy. Journal of Applied Psychology, 65, 351-354.

5. Foster, P. (1996) Observing Schools: a methodological guide. London: Paul Chapman. (p 89-91)

6. Robbins, S. (2005) Organizational Behavior, Eleventh Edition. Prentice Hall: Pearson Education

7. Bernardin, H.J. & Buckley, M.R. (1981) Strategies in Rater Training. The Academy of Management Review, Vol. 6, No. 2 (Apr., 1981), pp. 205-212.

8. Athey, T. R., & McIntyre, R. M. (1987). Effect of rater training on rater accuracy: Levels-of-processing theory and social facilitation theory perspectives. Journal of Applied Psychology, 72(4), 567-572.

9. Ryan, A.M., Daum, D., Bauman, T., Grisez, M., Mattimore, K., Nalodka, T., & McCormick, S. (1995). Direct, indirect, and controlled observation and rating accuracy. Journal of Applied Psychology, 80 (6), 664-670.

10. Watts F., Garcia-Carbonell A., Andreu-Andrés, M. (2013) INNOVATION COMPETENCIES DEVELOPMENT – INCODE barometer and User Guide [online] Available at <http://julkaisut.turkuamk.fi/isbn9789522164254.pdf< [Accessed at 12 December 2013]

11. Myford, C. M., & Wolfe, E. W. (2009). Monitoring rater performance over time: A framework for detecting differential accuracy and differential scale use. Journal of Educational Measurement, 46(4), 371–389.

25

9. Attachments Attachment 1: Handout Information about objectivity, reliability, validity (5)

26

Attachment 2 : Tasks for problem solving (Examples) Problems (examples) 1. The Bridge Four men want to cross a bridge. They all begin on the same side. It is night, and they have only one flashlight with them. At most two men can cross the bridge at a time, and any party who crosses, either one or two people, must have the flashlight with them. The flashlight must be walked back and forth: it cannot be thrown, etc. Each man walks at a different speed. A pair must walk together at the speed of the slower man. Abel needs 1 minute to cross the bridge, Arno needs 2 minutes, Alex needs 5 minutes, and Arthur needs 10 minutes. For example, if Abel and Alex walk across together, they need 5 minutes. How can all four men cross the bridge in 17 minutes? 2. Zebra There are 5 adjacent houses. Each house has a unique colour, and each owner has a different nationality. Each owner keeps a different pet, drinks a different type of beverage, and has a different occupation. The Brit lives in the red house, the Swede keeps a dog, and the Dane drinks tea. The green house is on the immediate left of the white house. In the green house they drink coffee. The postman has birds. The fireman lives in the yellow house. In the middle house they drink milk. The Norwegian lives in the leftmost house. The baker lives in the house next to the house with the cats. The fireman lives in the house next to the house with the horse. The bus driver drinks beer. The German is plumber. The Norwegian lives next to the blue house. They drink water in the house that lies next to the house where the baker lives. One of the owners keeps a zebra. Who owns the zebra? 3. When the Duchess of Worcestershire died she left a property of 3 million pounds. She bequeathed it all to three married couples, which had cared for her in her last years. Hanni got 50.000 pounds more than Nanni and Fanny got 50.000 pounds more than Hanni. All together the wives got 1.350.000 pounds. The husbands were also bequeathed unequally by the late Duchess of Worcestershire: John got the same amount as his wife. Jan got only half of the sum his wife got. And Jake got twice the amount of his wife. Who is married to whom? 4. The five pieces shown below must be put together to a square. How should this be done?

27

Attachment 3: Problem solving tasks - Solutions Solutions: 1. Bridge First Abel and Arno walk across the bridge. This takes 2 minutes. After this, Abel walks back with the flashlight. This takes 1 minute. Then Alex and Arthur walk across the bridge. This takes 10 minutes. After this, Arno walks back with the flashlight. This takes 2 minutes. Then Abel and Arno walk across the bridge. This takes 2 minutes as before. In total: 2+1+10+2+2=17 minutes. 2. Zebra House 1 House 2 House 3 House 4 House 5 Colour Yellow Blue Red Green White Owner Norwegian Dane Brit German Swede Pet Cats Horse Birds Zebra dog Beverage Water Tea Milk Coffee Beer Occupation Fireman Baker Postman Plumber Bus driver

3. Duchess

Hanni 450.000

Nanni 400.000

Fanny 500.000

John 450.000 Jan 200.000 Jake 1.000.000

4. Square

28

Attachment 4: Scales for the assessment of problem solving competence Self assessment

1 2 3 4 5 6 7 8 9 10 Partner assessment

1 2 3 4 5 6 7 8 9 10 Assessment of problem solver A

1 2 3 4 5 6 7 8 9 10 Assessment of problem solver B

1 2 3 4 5 6 7 8 9 10

29

Attachment 5: PPT-slides „Common Rater Errors“

30

31

32

33

34

Slides: Common Rater Errors - Commentary

Perception, observation, rating of behaviour is a challenging task. Raters should realise that they will never be completely neutral and objective observers.

The belief that what we see is real is challenged dramatically by visual illusions. They show us that in a sense all our perception is illusion. As Kant argued we are not able to see the thing in itself, we are only able to see the world through our perceptions of it.

On slide 3 you see black dots, which in reality do not exist. On slide 4 you see movement that is not there, and on slides 6 and 9 you see different shades of colour, but of course, it is the identical colour in different contexts. On slide 10 it depends on the direction from which you start looking, whether you perceive a duck or a bunny. For more examples one could show sequences of the TED-talks given by Beau Lotto and Al Seckel (URL on slide 5)

Visual illusions demonstrate that we cannot rely on our perception of designs, drawings, and objects. In behaviour rating we perceive other human beings. As a rater we have some kind of relationship to these persons. Our perception will be influenced by the social situation, by our own personality, by all our experiences with ourselves and with other people, by our needs and motivations, by our language, our culture, by conscious and unconscious processes, etc. Slide 12 could be an example for this: Young children will perhaps have no difficulties to see all these dolphins.

Slides 30f give an overview on common errors in person perception and behaviour rating. They should be talked about with reference to participant´s own experiences with rating in the previous Behavioural Observation Training (BOT) and with reference to the ICB.

Possible reasons e.g. for Leniency- Errors or Central Tendency-Errors should be discussed. Participants should talk about their own experiences with projection and stereotyping when judged by others.

Additional examples are given for the Halo-Effect (slides 28 and 29), the Primacy Effect (slides 25 and 26). The basketball-task as an striking example for Selective Perception (slides 19 and 20) is recommended, although it is already widely known.

35

Attachment 6: „Golden Rules“ to avoid common rater errors

Be aware of possible biases.

Address each behavior assessed as independent

from all other items.

Separate observation from judgment.

When taking notes, record your observations in behavioural terms.

Compare the performance

observed to the rating scale, not to other performers.

Evaluate only on the basis of evidence.

Do not try to compensate for a near miss on one

dimension with a generous rating on another.

Describe specific behaviour rather than describing an individual's personality.

Do not assign average ratings only and avoid using

high or low ratings.

Do not rate people more favourably who you

perceive as similar to yourself.

Consider the whole range of

behaviour, not the first or the last.

Don´t rely on your gut feelings.

Respect the scale.

Rate performance on each dimension separately.

Do not use just a limited range of scores, do use all

the scale.

36

Attachment 7: INNOVATION COMPETENCE DEVELOPMENT BAROMETER-GUIDE [10] http://incode-eu.eu/en/innovation-competencies-barometer/

http://incode-eu.eu/en/innovation-competencies-barometer/

37

Attachment 8: Description of the Crisis Situation: You belong to an advisory board, which is responsible for different social science research projects. Today you are called to an emergency meeting. One research project is in severe danger of ending in a catastrophe. This project studies human behaviour in constricted rooms. Five persons live together for four days in a cave more than 50 meters deep under the surface of the earth. Their only connection to the outside world is radio communication to the research station at the entrance of the cave. The research station has received an emergency call: Falling stones have blocked the exit, water is flooding into the cave and the water level is rising continually. A rescue team has already arrived at the entrance of the cave. With the available equipment it is possible to rescue only one person every hour. So it is to be feared that some of the people in the cave are going to drown before they can be brought safely to the surface. The people in the cave have been informed about this situation. They have declined to decide on the order in which they are to be saved. In 50 minutes the equipment will arrive at the entrance of the cave. Then the rescue team will need instructions. They will need a rescue plan. Now the responsibility is with you and the advisory board. Your Task: You have to decide in which order these people should be saved.

Order of evacuation Name 1. 2. 3. 4. 5.

In the documentation of the research project there are some biographical data that may help you to come to a decision: Kate (34), married, 4 children from 7 months to 8 years old. She studied psychology but quit her studies for getting married. Her husband is a municipal councillor. She takes part in the experiment because of John with whom she has a secret affair, which began two years ago. Filiz (19), single, student of sociology. Speaks English, French, Arabic and Turkish fluently. Very rich parents, who live in Istanbul. Her father is the owner of a shipping company and of a chain of first class tourist hotels near Antalya. Filiz is a very attractive woman, she has friends in the high society. She is invited by a Russian business-man to spend the next week on a yacht cruising around Sardinia. John (35), single, teacher. Before that for 8 years officer in the army. Was notorious for his brutal leadership style. Two soldiers died when he gave the order to cross a torrential river. For want of evidence he was not convicted. Perhaps due to a posttraumatic stress disorder he now leads an unsteady life, drinks too much and likes to gamble.

38

Boris (56), divorced, internationally renowned scientist, has found radically new ways to cure breast cancer, will publish his findings in the near future, if he succeeds in compiling his chaotic research records. After his wife left him he started frequenting gay bars and is often seen in male company. Jussi (62), married, two adult children. He is the owner of a small IT-company. He holds a patent for a new kind of server, and next week he will sign a contract with a Chinese state-owned company. That will be a big step forward and will bring 120 new and highly qualified jobs. Jussi is member of the Rotary Club and donates regularly large sums of money for charitable and cultural purposes. He is easily irritable and choleric. And people say that he beats his wife.

Documents

Rater Training Manual - INCODE