Peer-Instruction Unveiled: Measuring Self-Assessment Skills and Learning Gains in a Large Flipped Learning Environment

Fabio R. AricòDuncan Watson

APT – July 2015

2

ACKNOWLEDGEMENTS

Chris Thomson – UEA alumnus and Research Assistant

UEA-HEFCE Widening Participation Teaching Fellowship

HEA – Teaching Development Grant Scheme

3

OUTLINE

1. Peer-Instruction in a Large Flipped Learning Environment 2. Measuring Self-Assessment attainment and confidence levels objective and subjective measures of confidence

3. The effectiveness of the Peer-Instruction pedagogy

4. Student appraisal of the Peer-Instruction pedagogy

5. Summary of empirical findings.

4

ETHICAL REMARK

You will be presented with data collected during teaching sessions.

Students involved have given informed consent for me to analyse their responses and present the results of this analysis.

I can assist with ethical queries as well, please ask me.

5

1. Peer-Instruction in A Large Class Flipped Learning Environment

6

TEACHING PROTOCOL – the module

Introductory Macroeconomics Level 1 – compulsory year-long module - 170 students

Lectures traditional frontal-teaching (10 per sem.)

Seminars small group, pre-assigned problem sets (4 per sem.)

Workshops large group, problem-solving sessions (4 per sem.)

Support Sessions non-compulsory drop-in sessions (4 per sem.)

7

FLIPPED CLASS and PEER-INSTRUCTION

• Flipped classroom & Peer-Instruction pre-reading + student interaction Mazur (1997) Henderson and Dancy (2009)

well-developed research in Physics and STEM

• Learning analytics for Peer-Instruction Learning gains: Mazour Group - Bates & Galloway (2012)

Student satisfaction: Hernandez Nanclares & Cerezo Menendez (2014)

• There is no literature on the links with self-assessment skillsOpen field, with many unanswered questions e.g. role of demographics, language, previous background

Pedagogically: self-assessment blends with flipping and Peer-instruction.

8

WORKSHOPS – learning environment

Round 1- formative question- 4 choices- no information- no answer

Self-Assessment- confidence question- 4 level Likert-scale- information shared

Peer-Instruction- students talk- compare answers- explain each other

Round 2- formative question- Identical to R1- information shared- correct answer

Are students correctly self-assessing?

Is Peer-Instruction working?

9

2. Self-Assessment Skills PART A – attainment and confidence

10

WORKSHOPS – data coding

For each session (7 in a year in 2013-14):

• Code 1st response: 1 = correct 0 = incorrect

• Code confidence in response: 1 = strongly/agree0 = strongly/disagree

• Compute average score and average confidence per student.

attainment self-assessment

measure measure

11

WORKSHOPS – data coding

For each student (in each session):

If student average score > session average score high-attainment otherwise low-attainment

If student average conf. > session average conf. high-confidence otherwise low-confidence

12

WORKSHOPS – data analysis

Cross-tabulate results:

13

WORKSHOPS - results

What is the relationship between attainment and confidence?

14

WORKSHOP - results

What is the relationship between attainment and confidence?

15

2. Self-Assessment Skills PART B – confidence and entropy

16

WORKSHOPS – data coding and modelling

For each 1st response question (4-10) across all sessions (7 in a year in 2013-14):

• Compute entropy index across responses (A,B,C,D) objective measure of confidence

• Compute average confidence per question subjective measure of confidence

Model: confidence = f ( entropy ) + session-dummies

17

WORKSHOPS – results

confidence = f ( entropy ) + session-dummies

18

3. Peer-Instruction and Learning Gains

19

WORKSHOPS – learning gains

For each 1st and 2nd response question (4-10) across all sessions (7 in a year in 2013-14):

Learning Gain = % correct R2 % correct R1

effectiveness of Peer-Instruction

Model: Learning Gain = f ( % correct R1, confidence ) + session-dummies

20

WORKSHOPS – results

LearGains = f ( % R1, Conf ) + session-dummies

21

4. Student appraisal of Peer-Instruction

22

WHAT DO STUDENT THINK?

• The literature on Peer-Instruction is far too focused on whetherstudents ‘enjoy’ their experience (student satisfaction)

typical of academic practice literature.

• I tried to give more focus on the perception of learning: 1st lecture: introduced the concept of Peer-Instruction asked the students to share what they think of it. each workshop: asked students to share their view on the session and whether they felt they learnt from each other. informal end-of-module feedback: what was the most effective component of the blended learning environment mix within the module.

23

1st lecture: “‘Peer-instruction’ sessions (students teaching each other) are more effective than lectures (teacher teaching students)”

Series10%

5%

10%

15%

20%

25%

30%

35%

40%

45%

strongly agree

agree

disagree

strongly disagree

No. Respondents = 82

24

Wk 4 Wk 6 Wk 8 Wk14 Wk16 Wk18 Wk200%

10%

20%

30%

40%

50%

60%

70%

80%

strongly agree agree disagree strongly disagree

No. 89 38 72 69 52 39 40

Workshop feedback statement: “I have learnt more Economics by discussing answers with my classmates”

25

Comparing student opinion about Peer-Instruction as an effective pedagogy before and after exposure

1st Lecture (N=82) Avg Workshop (N=57)0%

10%

20%

30%

40%

50%

60%

70%

strongly agree

agree

disagree

strongly disagree

26

6%

20%

51%

4%

6%

13%

Lectures

Seminars

Workshops

Support

VLE

NA

End-of-module Feedback: What is the component of the Macro module which had the strongest impact on your learning?

27

5. Summary of Empirical Findings

28

SUMMARY of FINDINGS

• Positive association between attainment and confidence in performance this formative assessment design elicits good self-assessment outcomes

• Negative association between entropy and confidence levels objective and subjective measures of confidence align

• Negative association between learning gains and % correct R1 Peer-Instruction supports low-performers – ‘catching up effect’

• No association between learning gains and confidence confidence levels do not influence the effectiveness of Peer-Instruction

• Students seem to recognise the power of Peer-Instruction consistent opinions across different sources of feedback.

29

CURRENT and FURTHER RESEARCH

• Corroborate and strengthen empirical analysis and methodology. homogenise methodologies used for self-assessment and learning gains further robustness checks on empirical findings.

• “Assessing Self-Assessment” compare and contrasts 2 different learning environments assess the relationship between attainment and confidence.

• Learning analytics at student-level investigate the role of demographics investigate the impact of formative assessment on summative assessment.