Abelardo Pardo (@abelardopardo)Faculty of Engineering and IT slideshare.net/abelardo_pardo

Queen’s U

niversity flickr.com

Feedback at scale with a little help from my algorithms

Susana Fernandez flickr.com

Research Seminar Universitat Pompeu Fabra

Barcelona Spain 5 May 2016

Feedback at scale with a little help from my algorithmsAbelardo Pardo 2

Rishi S flickr.com

About me

Teaching

First year engineering course

Computer systems

350 students

Plenary 2 hour lecture + 2 hour tutorial + 3 hour laboratory session

Active learning

Heavy use of technology

Research

Educational technology

Learning analytics

Data-guided feedback

Naturalistic experiments

Feedback at scale with a little help from my algorithmsAbelardo Pardo 3

gingiber flickr.com

Feedback and Data

Decision Trees

Data driven rubric

Design and production

Feedback at scale with a little help from my algorithmsAbelardo Pardo 4

gingiber flickr.com

Feedback and Data

Feedback at scale with a little help from my algorithmsAbelardo Pardo 5

Eric flickr.com Systematically low ratings of

the feedback provided to the students

Feedback at scale with a little help from my algorithmsAbelardo Pardo 6

Krause, K.-L., Hartley, R., James, R., & McInnis, C. (2005). The First Year Experience in Australian Universities: Findings from a decade of National Studies. University of Melbourne: Centre for the Study of Higher Education.

Eleaf flickr.com The feedback question gets systematically lower values in student surveys

Feedback at scale with a little help from my algorithmsAbelardo Pardo 7

Huxham, M. (2007). Fast and effective feedback: are model answers the answer? Assessment & Evaluation in Higher Education, 32(6), 601-611. doi:10.1080/02602930601116946

• Lateness

• Uncertainty about criteria and contexts

Loonyhiker flickr.com

• Ambiguity or opacity (What do you mean?)

• Negativity

Loonyhiker flickr.com

Feedback at scale with a little help from my algorithmsAbelardo Pardo 8

Hounsell, D. (2007). Toward more sustainable feedback to students. In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in Higher Education: Learning for the Longer Term. London and New York: Routledge.

Perceived as an administrative chore instead of a pedagogical necessity

Marcin W

ichary flickr.com

Feedback at scale with a little help from my algorithmsAbelardo Pardo 9

Joel Sage flickr.com“sustainability of feedback is under threat… pervasive student concerns about the provision of feedback in

an era of mass education”

Hounsell, D. (2007). Toward more sustainable feedback to students. In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in Higher Education: Learning for the Longer Term. London and New York: Routledge.

Feedback at scale with a little help from my algorithmsAbelardo Pardo 10

Hounsell, D. (2008). The Trouble with Feedback. New Challenges, Emerging Strategies. Interchange(2).

Feedback has a Cinderella statusBitslam

mer flickr.com

Feedback at scale with a little help from my algorithmsAbelardo Pardo 11

Active Learning Works

Feedback at scale with a little help from my algorithmsAbelardo Pardo 12

Allen Lai flickr.com

Hounsell, D. (2007). Toward more sustainable feedback to students. In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in Higher Education: Learning for the Longer Term. London and New York: Routledge.

Downward spiral: Student disenchantment mounts when feedback is uninformative and still,

less feedback is provided

Solution: provide high-value feedback, rethink the role of student, enhance relation between guidance and feedback

Feedback at scale with a little help from my algorithmsAbelardo Pardo 13

Brandon Martin-Anderson flickr.com

• Can data guide the provision of effective feedback

• Can this provision be done at scale?

Feedback at scale with a little help from my algorithmsAbelardo Pardo 14

Dm

itry Grigoriev flickr.com

Learning Analytics: measure, collect, analyse data about learners to understand and improve their learning and the environment in which it occurs

Feedback at scale with a little help from my algorithmsAbelardo Pardo 15

Eustaquio Santimano flickr.com

Collect

Report

Analyze

Act

Refine

Feedback at scale with a little help from my algorithmsAbelardo Pardo 16

Feedback at scale with a little help from my algorithmsAbelardo Pardo 17

Eustaquio Santimano flickr.com

Collect

Report

Analyze

Act

Refine

Feedback at scale with a little help from my algorithmsAbelardo Pardo 18

gingiber flickr.com

Feedback and Data

Decision Trees

Feedback at scale with a little help from my algorithmsAbelardo Pardo 19

William

Murphy flickr.com

• 13 Week first year Engineering • Weekly activities (formative/summative) • Videos, MCQ, Exercises, dashboard • n = 272, Weeks 2-5 and 7-13

Pardo, A., Mirriahi, N., Martínez-Maldonado, R., Jovanović, J., Dawson, S., & Gasevic, D. (2016). Generating Actionable Predictive Models of Academic Performance. Paper presented at the International Conference on Learning Analytics and Knowledge, Edinburgh. doi:10.1145/2883851.2883870

Feedback at scale with a little help from my algorithmsAbelardo Pardo 20

Oliver Braubach flickr.comObjective

1. Data indicators close to learning design

2. Predictive model

3. Bridge between model and application

4. Straightforward delivery method

Feedback at scale with a little help from my algorithmsAbelardo Pardo 21

Louish Pixel flickr.com

• Event counts from interactive course material

• Midterm/final exam scores

• Recursive partitioning

• Divide cohort into performance categories

Feedback at scale with a little help from my algorithmsAbelardo Pardo 22

Data collected

• Indicators are directly connected with learning design

• Data structure shaped by the schedule (weeks)

• Data available in a per-week basis

• What is the expected midterm/final score in week n?

Feedback at scale with a little help from my algorithmsAbelardo Pardo 23

Result Example• Week 10

• Predicted score at leaves (out of 40)

• Conditions at nodes

• If (EXC.in >=22) and (VID.PL < 8.5) then score = 6

Feedback at scale with a little help from my algorithmsAbelardo Pardo 24

• Each leaf node represents a group of students with their estimated score.

• Example: 6, 8.3, 8.4, 9.4, 9.9, 10, 15 (out of 40)

• Intervention: suggested work before exam

Result interpretation

Feedback at scale with a little help from my algorithmsAbelardo Pardo 25

shabnam m

ayet Flickr.com

RMSE: Root mean square error, MAE: Mean absolute error

Performance

Feedback at scale with a little help from my algorithmsAbelardo Pardo 26

gingiber flickr.com

Feedback and Data

Decision Trees

Data driven rubric

Feedback at scale with a little help from my algorithmsAbelardo Pardo 27

2 Hours

2 Hours

3 Hours

Pardo, A. & Mirriahi N. (in press). Design, Deployment and Evaluation of a Flipped Learning First Year Engineering Course. In C. Reidsema, L. Kavanagh, R. Hadgraft & N. Smith (Eds.), Flipping the Classroom: Practice and Practices. Singapore: Springer.

2014 Edition

Feedback at scale with a little help from my algorithmsAbelardo Pardo 28

Feedback at scale with a little help from my algorithmsAbelardo Pardo 29

1. realtime feedback

2. convey engagement

3. compare with rest of cohort

4. reset weekly

5. one click away from notes

Approach 1 (2014)Sean D

reilinger flickr.com

Feedback at scale with a little help from my algorithmsAbelardo Pardo 30

Tracking

Feedback at scale with a little help from my algorithmsAbelardo Pardo 31

Technology

Technology

Week N

Feedback at scale with a little help from my algorithmsAbelardo Pardo 32

Weekly real time feedback

Feedback at scale with a little help from my algorithmsAbelardo Pardo 33

No statistically significant difference in the rating of feedback (2013 edition, M=3.25, SD=0.97; 2014 edition, M=3.35, SD=1.03); t(389.78) = -0.97, p <0.17

Feedback at scale with a little help from my algorithmsAbelardo Pardo 34

1. encourage contact between student and faculty (forum)

2. uses active learning techniques

3. gives prompt feedback

4. emphasizes time on task

5. kind, specific, helpful

Approach 2 (2015)Ben Manson flickr.com

Ben Manson flickr.com

Feedback at scale with a little help from my algorithmsAbelardo Pardo 35

Week NTechnology

Human

Feedback at scale with a little help from my algorithmsAbelardo Pardo 36

You should take a more careful look at how symbols are encoded in the video. Would you be able to encode/decode UAL symbols without looking at the video?

Good initial work. However, did you understand the trick to handle encoding with a variable number of bits? Would you be able to provide an example?

Good work. Would you be able to come up with your own machine language and your encoding scheme? Remember that it has to be unambiguous.

Thorough work with the task about machine language encoding. Give it a quick review before the midterm.

Q1 Q2 Q3 Q4

Instructor

Feedback at scale with a little help from my algorithmsAbelardo Pardo 37

Algorithm

Feedback at scale with a little help from my algorithmsAbelardo Pardo 38

AutomaticEmail

Feedback at scale with a little help from my algorithmsAbelardo Pardo 39

1. Data collected weekly

2. Email sent at end of week

3. 4 weeks before the midterm

nateOne flickr.com

Feedback at scale with a little help from my algorithmsAbelardo Pardo 40

Significant difference in the rating of feedback (2014 edition, M=3.35, SD=1.03; 2015 edition, M=3.82, SD=0.90); t(389.78) = -4.88, p <10-6 Effect size (Cohen’s d) = 0.49. Medium positive effect

Feedback at scale with a little help from my algorithmsAbelardo Pardo 41

Significant difference in the midterm score (2014 edition, M=12.80, SD=4.79; 2015 edition, M=13.83, SD=4.89); t(684.5) = -2.86, p < 0.002 Effect size (Cohen’s d) = 0.21. Small positive effect

Feedback at scale with a little help from my algorithmsAbelardo Pardo 42

gingiber flickr.com

Feedback and Data

Decision Trees

Data driven rubric

Design and production

Feedback at scale with a little help from my algorithmsAbelardo Pardo 43

Departm

ent for Business Innovation & Skill flickr.com • Content creation

• Interactive

• Multi-view (tutor, bilingual)

• Focus on content (not style)

• Platform agnostic

• Design-embedded analytics

Feedback at scale with a little help from my algorithmsAbelardo Pardo 44

David M

ichalczuk flickr.com

• Markup language

• Batch processing

• Sphinx-doc

• Webdav gateway for publishing

• Self-contained web site

• Extensible language: macros for design elements

Feedback at scale with a little help from my algorithmsAbelardo Pardo 45

Feedback at scale with a little help from my algorithmsAbelardo Pardo 46

Feedback at scale with a little help from my algorithmsAbelardo Pardo 47

Single source approach

• Plain text

• Version control

• Distributed production

• Suitable for ed designers

Feedback at scale with a little help from my algorithmsAbelardo Pardo 48

Back annotation

Feedback at scale with a little help from my algorithmsAbelardo Pardo 49

“The interactive quizzes during class are excellent” “I love the quizzes”

‘Learning was supported by useful electronic learning resources’ Agreement increased from 78% to 98%

Professional Practice of Radiography (PG). A/Prof. Mark McEntee

Feedback at scale with a little help from my algorithmsAbelardo Pardo 50

https://bitbucket.org/abelardopardo/reauthoring

•Activity duration

•MCQ

•Video embedding

•Difficulty/Usefulness 2D grid

•Tutor view

•Tutor feedback

•Coupled with tracking

• Automatic back-annotation

Feedback at scale with a little help from my algorithmsAbelardo Pardo 51

Fabien CAM

BI flickr.com

• Feedback is delicate and messy

• Data can help scaling

• Good focal point for analytics

• Provide intuitive tools to instructors

• Authoring is a challenge

Abelardo Pardo (@abelardopardo)Faculty of Engineering and IT slideshare.net/abelardo_pardo

Queen’s U

niversity flickr.com

Feedback at scale with a little help from my algorithms

Susana Fernandez flickr.com

Research Seminar Universitat Pompeu Fabra

Barcelona Spain 5 May 2016