Provision of personalized feedback at scale using learning analytics

Centre for the Enhancement of Teaching and Learning Faculty of Education Hong Kong University 19/May/2017

Abelardo Pardo (@abelardopardo)Faculty of Engineering and IT slideshare.net/abelardo_pardo

Antonietta flickr.com

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 2

Maureen Barlin flickr.com

NewDesignMindset

The role of data

and feedback

Personalised feedback at scale

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 3

Maureen Barlin flickr.com

New Design Mindset

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 4

Simple information transfer is not working

Mazur, E. (2009). Farewell, lecture. Science, 323(5910), 50-51.

Krugazor flickr.com

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 5

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 6

Active Learning Works

Engage students in the learning process

Bonwell, C. C., & Eison, J. A. (1991). Active Learning: Creating Excitement in the Classroom ASHEERIC Higher Education Report No. 1. Washington, DC, USA: George Washington University.

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 7

“… robust correlations between student involvement in a subset of ‘educationally

purposive activities’, and positive outcomes of student success and development, including

satisfaction, persistence, academic achievement and social engagement”

Trowler, V. (2010). Student engagement literature review. York, UK: The Higher Education Academy.

Joshua Ganderson flickr.com

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 8

Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: beliefs, techniques, and illusions. Annu Rev Psychol, 64, 417-444. doi:10.1146/annurev-psych-113011-143823

Christian W

eidinger flickr.com

We, as learners may

• Not know how to promote comprehension, retention, transfer.

• Not assess properly our own learning

• Be biased when judging our learning

• Rely too much on social beliefs

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 9

Janson Hew

s flickr.com

Frontier between physical and virtual spaces is blurring

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 10

Beware of technology creating the illusion of rational thinking

Jen R flickr.com

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 11

“… teaching in higher education will necessarily shift the balance of its efforts towards a greater investment in design

as a way of coping with otherwise intolerable pressures on staff and

resources.”

Goodyear, P. (2015). Teaching as Design. HERDSA Review of Higher Education, 2, 27-50.

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 12

"There is no such thing as a neutral design"

Jeremy Brooks flickr.com

Thaler, R. H., & Sunstein, C. R. (2008). Nudge. Great Britain: Yale University Press.

“People make good choices in contexts in which they have experience, good information, and

prompt feedback"

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 13

“38 meta-analyses investigating 105 correlates of achievement, based on 3,330 effect sizes from almost 2 million students”

Schneider, M., & Preckel, F. (2017). Variables Associated With Achievement in Higher Education: A Systematic Review of Meta-Analyses. Psychological Bulletin. doi:10.1037/bul0000098

• The effectivity of courses is strongly related to what teachers do.

• The effectivity of teaching methods depends on how are implemented

• Teachers can improve the instructional quality of their courses by making a number of small changes

- providing detailed task-focused and improvement-oriented feedback

• The combination of teacher-cantered and student-cantered instructional elements is more effective than either form of instruction alone

Variables associated with achievement

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 14

Maureen Barlin flickr.com

NewDesignMindset

The roleof data

and feedback

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 15

Students are less likely to engage in pre-class activities if they are not interactive, do not provide formative feedback, and

not coherently linked with the face-to-face activities

O'Flaherty, J., & Phillips, C. (2015). The use of flipped classrooms in higher education: A scoping review. The Internet and Higher Education, 25, 85-95. doi:10.1016/j.iheduc.2015.02.002

Dan Klim

ke flickr.com

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics

If you could choose one…

• Over 800 meta-analyses of student achievements

• 100 factors with potential influence

• Feedback in top five

• (74 meta-analyses) Most effective form: video, audio, computer-assisted instructional feedback, and/or related goals

16Hattie, J. (2008). Visible learning: A synthesis of over 800 meta-analyses related to achievement. New York: Routledge.

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 17

Boud, D., & Molloy, E. (Eds.). (2013). Feedback in Higher and Professional Education: Understanding it and doing it well. London and New York: Routledge.

Faruk Ateş flickr.com

“Feedback is a process whereby learners obtain information about their work in order to appreciate the similarities and differences between the appropriate standards for any given work, and the qualities of the work itself, in order to generate improved work”

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 18

Black, P., & Wiliam, D. (1998). Assessment and Classroom Learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7-74. doi:10.1080/0969595980050102

Gareth C

hristopher flickr.com

Innovations designed to strengthen the frequent feedback that students receive about their learning yield substantial learning gains

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 19

Hounsell, D. (2007). Toward more sustainable feedback to students. In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in Higher Education: Learning for the Longer Term. London and New York: Routledge.

Perceived as an administrative chore instead of a pedagogical necessity

Marcin W

ichary flickr.com

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 20

Ian Ransley flickr.com

How to scale sustainable practices?

Carless, D., Salter, D., Yang, M., & Lam, J. (2011). Developing sustainable feedback practices. Studies in Higher Education, 36(4), 395-407. doi:10.1080/03075071003642449

• Multi-stage assignments

• Dialogic feedback

• Technology supported

• Self-evaluation

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 21

Baker, R. S. (2016). Stupid Tutoring Systems, Intelligent Humans. International Journal of Artificial Intelligence in Education. doi:10.1007/s40593-016-0105-0

From artificial intelligence to intelligence amplification

Instructors are informed about student engagement but it is up to them to decide if/when/how to act

Tecnalia flickr.com

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 22

Dm

itry Grigoriev flickr.com

Learning Analytics: measure, collect, analyse data about learners to understand and improve their learning and the environment in which it occurs

1st International Conference on Learning Analytics and Knowledge [Online]. https://tekri.athabascau.ca/analytics. [Accessed 2017].

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 23

• Collect data about how students engage in a learning experience

• Interpret the observations in the context of the instructional design

• Translate knowledge into personalised student support actions

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 24

Bartimote-Aufflick, K., Reimann P., Pardo, A. The perspective theory brings to learning analytics in the classroom: A realist approach Manuscript in preparation.

Learning Analytics Model

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 25

Maureen Barlin flickr.com

NewDesignMindset

The role of data

and feedback

Personalised feedbackat scale

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 26

Example of Highly Instrumentalized Learning Design

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 27

Weekly real time feedback

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 28

No statistically significant difference in the rating of feedback (2013 edition, M=3.25, SD=0.97; 2014 edition, M=3.35, SD=1.03); t(389.78) = -0.97, p <0.17

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 29

You should take a more careful look at how symbols are encoded in the video. Would you be able to encode/decode UAL symbols without looking at the video?

Good initial work. However, did you understand the trick to handle encoding with a variable number of bits? Would you be able to provide an example?

Good work. Would you be able to come up with your own machine language and your encoding scheme? Remember that it has to be unambiguous.

Thorough work with the task about machine language encoding. Give it a quick review before the midterm.

Q1 Q2 Q3 Q4

Instructor

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 30

Algorithm

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 31

AutomaticEmail

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 32

Helpful feedback

Effect size (Cohen’s d) = 0.49. Medium positive effect

Midterm Scores

Effect size (Cohen’s d) = 0.21. Small positive effect

Pardo, A., Jovanovic, J., Dawson, S., Gašević, D. & Mirriahi, N. (2017). Using Learning Analytics to Scale the Provision of Personalised Feedback. Manuscript in preparation

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 33

ontasklearning.org

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 34

• Support instructors to create personalised feedback

• Simple rule-base knowledge encoding

• Provide appropriate view of data sources

• Scale to large and highly diverse cohorts

• Will be released as open-source project Q3/4 2017

• First pilots in Q1/2 2017

• Tutorial in LAK 2017

• Contact us if interested

ontasklearning.org

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 35

MTC

0316 flickr.com

Conclusions• New L&T design mindset required • Feedback is effective to promote

student engagement • Learning analytics has the

potential to support students at scale

• Use data to provide personalised student support

Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 36

Provision of personalized feedback at scale using learning analytics

Abelardo Pardo (@abelardopardo)Faculty of Engineering and IT slideshare.net/abelardo_pardo

Antonietta flickr.com

Centre for the Enhancement of Teaching and Learning Faculty of Education Hong Kong University 19/May/2017