Building a Peer Evaluation Program

Building a Peer Evaluation Program

2

Housekeeping

S O ME G E N E R A L H O U S E K E E P I N G R U L E S

o A copy of the slides is available in the resources panel

o The webinar will be recorded and made available ON-Demand

o To keep the webinar short questions will be answered via email later today.

Speaker

Sarah MarrsPRINCIPAL CONSULTANT, PEOPLE INSIGHTS

Sarah has dedicated her career to designing and running feedback programs for over 7 years. She joined Qualtrics in April 2015, after working in-house at Tesco overseeing their global colleague insights survey. At Qualtrics, Sarah works with clients to design feedback solutions that drive real value, through assessment design, sampling methodologies, communication programs, or interpreting results.

She’s a big believer in the power of listening to and acting on feedback, and always energised to talk about new and innovative ways to collect and act on that information.

4

Agenda

TO D AY W E W I L L B E C O V E R I N G

o What is a peer evaluation programo How to create a peer evaluation program

1. Designing a competency framework and items

2. Designing your reports3. Designing your workflow

5

What is peer evaluation? A peer evaluation program collects,

quantifies, and reports observations of a student by their classmates or faculty.

C O MM O N LY I N D I V I D U A L S A R E R AT E D B Y:

o Themselveso Cohort / classmateso Tutor / Professoro Potentially course directoro Potentially academic advisor

6



7



8



9



Peer evaluation is ultimately designed to drive

lasting behaviour change

10

What is peer evaluation? W HAT D O S T U D E N T S G E T F R O M P E E R E VA L U AT I O N?

o Self-awareness (no.1 leadership trait!)o Improvement in learningo Teaching how to give feedbacko Teaching how to take feedback

W H AT D O E S A T U TO R G E T F R O M P E E R E VA L U AT I O N

o A feedback instrument that works for different student styleso An instrument that allows them to get a gauge on student

dynamicso An instrument that allows them to get data-driven feedback

aggregation

11

What is peer evaluation? T H R E E C O M MO N U S E S F O R P E E R E VA L U AT I O N

1. Measure contributions and teamwork at the end of a specific project to feed into project grades

2. Mid-year measurement of a student’s behaviours to provide developmental feedback

3. End-of-placement review of a student work placement to get feedback from both professional and academic sides

12

What is peer evaluation?

YOU MUST KNOW YOUR PEER EVALUATION PURPOSE BEFORE MOVING ON TO DESIGN YOUR PROGRAM

13

How to create a peer evaluation program

T H R E E S T E P S TO C R E AT E Y O U R P E E R E VA L U AT I O N

1. Design your competency framework and items2. Design your reports3. Design your workflow

14

C O MP E T E N C Y A

I T E M ( B E H AV I O U R ) 1



C O MP E T E N C Y B

C O MP E T E N C Y C







STEP 1 - Design a competency framework & items

15

T E AM W O R KI N G S K I L L S

L i s t e n s t o c l a s s m a t e s

’ i d e a s

Ta k e s o n o t h e r ’s

f e e d b a c k

W o r k s w e l l w i t h

c l a s s m a t e s

C O N T R I B U T I O N TO W O R K L O A D

A t t e n d s a l l w o r k i n g s e s s i o n s

P a r t i c i p a t e s a c t i v e l y i n

w o r k i n g s e s s i o n s

I s a b l e t o c a r r y o u t t a s k s i n d e p e n d e n t l y

P L AN N I N G & O R G A N I S I N G

A r r i v e s o n t i m e t o w o r k i n g s e s s i o n s

M a n a g e s t h e i r s c h e d u l e w e l l

M e e t s p r o j e c t -r e l a t e d

d e a d l i n e s

W o r k s w e l l w i t h

c l a s s m a t e s

P R O - T I PFrequency scales are the best way to focus on the consistency of behaviours, e.g:

• Never• Rarely• Sometimes• Often• All the time


16

G O O D I T E M S . . .

o Capture an important aspect of a key competency

o Focus on a single behaviouro Start with an action verb (e.g., motivates,

listens, etc.)o Are written in simple plain languageo Can be acted upon if they are identified as

weaknesses/ opportunities

B AD I T E MS . . .

o Loosely or fail to measure an important competencyo Focus on multiple behaviours (double, triple barreled)o Include too many unnecessary adverbs (e.g.,

efficiently, effectively)o Are too technical or hard to understando Do not match the rating scaleo Are culturally biased


17


O T H E R C O MP E T E N C Y A N D I T E M T I P S

o Avoid including too many items (generally 50-60 total items max)o Include a few (up to 3) open-ended items that ask students to provide

constructive and actionable feedback to their cohorto Remind evaluators not to include identifying information in open-ended

items to protect confidentiality

18

STEP 2 - Design your reports

T W O M A I N T Y P E S O F R E P O R T

1. The individual report (to go back to the student)

2. The aggregated report (to be seen by the tutor / professor / course director only)

19


G O O D R E P O R T S :

AGGREGATE RESULTS WITHIN SOURCE – Aggregate ratings within source for each competency and item

20



PROVIDE MEANINGFUL COMPARISONS – Compare source-level ratings to internal benchmarks


21



PROVIDE MEANINGFUL COMPARISONS – Compare source-level ratings to internal benchmarks

INCLUDE OPEN-ENDED COMMENTS – Present both positive and constructive comments


22

STEP 3 - Design your workflow C O MM O N W O RK F L O W D E C I S I O N S

o Who chooses who should evaluate an individual?o Should the tutor / professor approve nominations for

evaluators?o Should evaluators have an opportunity to opt out?o Should the individual receive their report directly?

23

STEP 3 - Design your workflow S C E N A R I O 1 . MEASURE CONTR IBUT IONS AND

TEAMWORK AT THE END OF A SPEC IF IC PROJECT TO FEED INTO PROJECT GRADES

Tutor / professor adds all students in project groups

– and asks all students in group

to rate each other

All students need to complete their

project group evaluations

(no opting out)

Report goes back to tutor / professor

directly – for each individual. Aggregate report also created allowing them to

compare individuals.

Tutor discusses report with student and

feeds into project grades

24

STEP 3 - Design your workflow S C E N A R I O 2 . MID-YEAR MEASUREMENT OF AN

INDIV IDUAL ’S BEHAVIOURS TO PROV IDE DEVELOPMENTAL FEEDBACK

Individual is asked to

select 3-5 peers and up

to 3 tutors who they

would like to receive

feedback from

Individual will automatically

receive evaluations from their

course director and academic advisor

Peer evaluations

are opt-in – so not

mandatory for all classmates to complete

Report goes back to

individual student first, for them to digest and

later discuss with academic

advisor or tutor.

Course directors get aggregated feedback reports

25

STEP 3 - Design your workflow

Course director adds

placement students -

asking tutor, classmates

and professional coworkers to

evaluate them

Placement evaluations are ‘nudged’

but not mandatory

Individual student

report goes to each tutor for feedback discussion

Aggregate report goes to course director to

review placement

program as a whole

Course directors

reach out to company

placement teams to discuss

feedback / any red flags

S C E N A R I O 3 . END- OF-PL ACEMENT REV IEW OF A STUDENT WORK PL ACEMENT TO GET FEEDBACK FROM BOTH PROFESS IONAL AND ACADEMIC S IDES

In Summary

1. Define Purpose2. Define Competencies & Behaviours3. Define Report4. Define Workflow5. Launch evaluation6. Follow-up with students

27

Thank you

Technology

Building a Peer Evaluation Program