Learning Analytics and Student Feedback

Professor Denise Whitelock

The Open University, Walton Hall,

Milton Keynes MK7 6AA, UK

denise.whitelock@open.ac.uk

Learning Analytics and Student Feedback

• What is Learning Analytics?

• Origins

• Early work

• Learning Analytics and Assessment

DMW UOC May 2013

Definition

“Learning Analytics are concerned with the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs”.

Reference

SoLAR, Open Learning Analytics: An Integrated & Modularised Platform, White Paper, Society for Learning Analytics Research, 2011.

DMW UOC May 2013

Interdisciplinary Research

• Computer Sciences

• Learning Sciences

• AIED

• Data Mining specialists

DMW UOC May 2013

Middle Space

• 3rd LAK Conference

• Learning explicit

• New analytic methods• Computational

• Representational

• Statistical

• Visualisation

DMW UOC May 2013

Can LAK hold together for long?

• Challenges

• Different methodologies

• Different theories

• Different predjucies

• Agreement on topics 3rd LAK 2013

• Visualisation, social network analysis, communication and collaboration, discourse analytics, predictive analytics, sequence analytics, assessment

After Suthers & Verbert (2013)

DMW UOC May 2013

Political and economic drivers

• Educause Review (2007)

• Academic Analytics, Campbell & Oblinger (2007)

• Large data and stats = predictive modelling

• Improve number of graduates in US

• US finding now with school exam data

• Hand code

• M.L.

• Apply whote set

• Make predictions

DMW UOC May 2013

Formative Research towards separate field

• Open Learner models (Bull & Kay, 2007)

• Social Network analysis (De Laat et al, 2007)

• Networks Adapting Pedagogical Practice (SNAPP), (Dawson et al, 2010)

• Visualisation of large data sets, Honeycomb (van Ham et al, 2009)

• Gephi: open source tool (Bastian et al, 2009)

• Signals (Arnold, 2010)

DMW UOC May 2013

Signals: Flagship Project

• Moves data from VLE

• Combines with prediction models

• Real time red/amber/green traffic lights

• Pilot study (Arnold, 2010) showed

• Students sought help earlier

• 12% more B/C grades

• 14% less D/F grades

DMW UOC May 2013

Formative Assessment and Learning Analytics (1) Tempelaar et al, 2013

• 1st year Math & Stats undergraduates, Maastricht

• Reason text book

• Online questions

• Practice and performance tests

• 92% higher practice pass

• 51% lower practice pass

DMW UOC May 2013

Formative Assessment and Learning Analytics (2)Important for SAFeSEA?

• Learning styles (Vermunt, 1996)

• Self regulation for deep learning

• Practice for stepwise learning

• Motivation and engagement wheel (Martin, 2007)

• Learning emotions

• Pekrun’s control-value theory of learning emotions

DMW UOC May 2013

Performance of video lectures

• Findings from Mirriahi & Dawson (2013)

• Correlations between quizzes and lectures

• Shows misalignment between Assessment and Teaching materials?

DMW UOC May 2013

HOU2LEARN

• PLE from Hellenic Open University

• Social network analysis and final grades

• Online collaboration is not a predictor for final grade

• Koulocheri & Xenos, 2013

DMW UOC May 2013

OpenEssayist: SAFeSEA Web application for summarisation-based formative feedback

DMW UOC May 2013

localhost:8065

phaeros.open.ac.uk:80

openEssayist

PHP, Epiphany[Symfony2]

User

openEssayist RESTful API

PHP, Epiphany

User

User

pyEARESTful API

Python, Flask

localhost:8064

AfterTheDeadlineSpell/Grammar

checker

Java

User

localhost:9998

Apache TikaText Extractor

Java

Orchestrator

(Open)Learner Model

pyEssayAnalyser

Python, NLTK

localhost:8065

phaeros.open.ac.uk:80

openEssayist

PHP, Epiphany[Symfony2]

User

openEssayist RESTful API

PHP, Epiphany

User

User

pyEARESTful API

Python, Flask

localhost:8064

AfterTheDeadlineSpell/Grammar

checker

Java

User

localhost:9998

Apache TikaText Extractor

Java

Orchestrator

(Open)Learner Model

pyEssayAnalyser

Python, NLTK

Key words and phrases visualized in the essay context. Sentences in light-grey (green) background are key sentences as extracted by the EssayAnalyser (the number at the start of the sentence indicates its key-ness ranking);

bigrams are indicated in bold (red) and boxed.

DMW UOC May 2013

The structural elements of the essay can be used jointly with the key word extraction to highlight relevant information within specific parts of the essay, here the introduction (and the assignment question)

DMW UOC May 2013

Key words and phrases as separate lists

DMW UOC May 2013

Dispersion of key words across the essay http://www.open.ac.uk/iet/main/research-scholarship/research-projects/supportive-automated-feedback-short-essay-answers

DMW UOC May 2013

Can we find ways of using graph visualization techniques on the key words and key sentences, to make them helpful and meaningful to students?

DMW UOC May 2013

Final Thoughts

• Instant machine feedback not prevalent

• Artificial Intelligence analysis to tutors leading to Wizard of Oz responses (Shaffer, 2013)

• Just in time feedback is the ultimate goal

DMW UOC May 2013

References (1)Arnold, K.E. (2010). Signals: applying academic analytics, Educause Quarterly, 33(1), p10. http://www.educause.edu/ero/article/signals-applying-academic-analytics (Accessed 30 April 2013)

Bastien, M., Heymann, S. & Jacomy, M. (2009). Gephi: an open source software for exploring and manipulating networks. Paper presented at the International AAAI Conference on Weblogs and Social Media.

Bull, S & Kay, J. (2007). Student models that invite the learner in: the SMILI:-) open learner modelling framework, International Journal of Artificial Intelligence in Education, 17(2).

Campbell, J.P. & Oblinger, D.G. (2007). Academic Analytics, Educause.

Dawson, S., Bakharia, A. & Heathcote, E. (2010). Snapp: Realising the affordances of real-time SNA within networked learning environments. Paper presented at The 7th International Conference on Networked Learning, Aalborg, Denmark (3-4 May).

DMW UOC May 2013

References (2)

De Laat, M., Lally, V., Lipponen, L. & Simons, R.-J. (2007). Investigating patterns of interaction in networked learning and computer-supported collaborative learning: a role for social network analysis. International Journal of Computer Supported Collaborative Learning, 2, pp.87-103.

Koulocheri, E. & Xenos, M. (2013). Considering Formal Assessment in Learning Analytics within a PLE: The HOU2LEARN Case. Paper presented at The International Learning Analytics & Knowledge (LAK) Conference, Leuven, Belgium, (8-12 April 2013) © 2013 ACM 978-1-4503-1785-6/13/04

Martin, A.J. (2007). Examining a multidimensional model of student motivation and engagement using a construct validation approach. British Journal of Educational Psychology, 77, 413-440.

DMW UOC May 2013

References (3)

Mirriahi, N. & Dawson, S. (2013) The pairing of lecture recording data with assessment scores: A method of discovering pedagogical impact. Paper presented at The International Learning Analytics & Knowledge (LAK) Conference, Leuven, Belgium, (8-12 April 2013) © 2013 ACM 978-1-4503-1785-6/13/04

Pekrun, R. (2006). The control-value theory of achievement emotions: assumptions, corollaries and implications for educational research and practice. Educational Psychology Review, 18, 315-34.

SoLAR, Open Learning Analytics: An Integrated & Modularised Platform, White Paper, Society for Learning Analytics Research, 2011.

Suthers, D. & Verbert, K. (2013) Learning Analytics as a “Middle Space”. Paper presented at The International Learning Analytics & Knowledge (LAK) Conference, Leuven, Belgium, (8-12 April 2013) © 2013 ACM 978-1-4503-1785-6/13/04

DMW UOC May 2013

References (4)

Tempelaar, D.T., Cuypers, H., van de Vrie, E., Heck, A. & van der Kooij, H. (2013). Formative Assessment and Learning Analytics. Paper presented at The International Learning Analytics & Knowledge (LAK) Conference, Leuven, Belgium, (8-12 April 2013) © 2013 ACM 978-1-4503-1785-6/13/04

Van Ham, F., Schulz, H.-J. & Dimicco, J.M. (2009). Honeycomb: visual analysis of large scale social networks, Ifip International Federation for Information Processing, 429-442.

Van Labeke, N., Whitelock, D., Field, D., Pulman, S. & Richardson, J. (2013) ‘OpenEssayist: Extractive Summarisation & Formative Assessment of Free-Text Essays’. Workshop on Discourse-Centric Learning Analytics, 3rd Conference on Learning Analytics and Knowledge (LAK 2013), Leuven, Belgium

Vermunt, J.D. (1996). Leerstijlen en sturen van leerprocessen in het Hoger Onderwijs. Amsterdam/Lisse: Swets & Zeitlinger.

DMW UOC May 2013