Learning Analytics and Student Feedback
Professor Denise Whitelock
The Open University, Walton Hall,
Milton Keynes MK7 6AA, UK
Learning Analytics and Student Feedback
• What is Learning Analytics?
• Origins
• Early work
• Learning Analytics and Assessment
DMW UOC May 2013
Definition
“Learning Analytics are concerned with the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs”.
Reference
SoLAR, Open Learning Analytics: An Integrated & Modularised Platform, White Paper, Society for Learning Analytics Research, 2011.
DMW UOC May 2013
Interdisciplinary Research
• Computer Sciences
• Learning Sciences
• AIED
• Data Mining specialists
DMW UOC May 2013
Middle Space
• 3rd LAK Conference
• Learning explicit
• New analytic methods• Computational
• Representational
• Statistical
• Visualisation
DMW UOC May 2013
Can LAK hold together for long?
• Challenges
• Different methodologies
• Different theories
• Different predjucies
• Agreement on topics 3rd LAK 2013
• Visualisation, social network analysis, communication and collaboration, discourse analytics, predictive analytics, sequence analytics, assessment
After Suthers & Verbert (2013)
DMW UOC May 2013
Political and economic drivers
• Educause Review (2007)
• Academic Analytics, Campbell & Oblinger (2007)
• Large data and stats = predictive modelling
• Improve number of graduates in US
• US finding now with school exam data
• Hand code
• M.L.
• Apply whote set
• Make predictions
DMW UOC May 2013
Formative Research towards separate field
• Open Learner models (Bull & Kay, 2007)
• Social Network analysis (De Laat et al, 2007)
• Networks Adapting Pedagogical Practice (SNAPP), (Dawson et al, 2010)
• Visualisation of large data sets, Honeycomb (van Ham et al, 2009)
• Gephi: open source tool (Bastian et al, 2009)
• Signals (Arnold, 2010)
DMW UOC May 2013
Signals: Flagship Project
• Moves data from VLE
• Combines with prediction models
• Real time red/amber/green traffic lights
• Pilot study (Arnold, 2010) showed
• Students sought help earlier
• 12% more B/C grades
• 14% less D/F grades
DMW UOC May 2013
Formative Assessment and Learning Analytics (1) Tempelaar et al, 2013
• 1st year Math & Stats undergraduates, Maastricht
• Reason text book
• Online questions
• Practice and performance tests
• 92% higher practice pass
• 51% lower practice pass
DMW UOC May 2013
Formative Assessment and Learning Analytics (2)Important for SAFeSEA?
• Learning styles (Vermunt, 1996)
• Self regulation for deep learning
• Practice for stepwise learning
• Motivation and engagement wheel (Martin, 2007)
• Learning emotions
• Pekrun’s control-value theory of learning emotions
DMW UOC May 2013
Performance of video lectures
• Findings from Mirriahi & Dawson (2013)
• Correlations between quizzes and lectures
• Shows misalignment between Assessment and Teaching materials?
DMW UOC May 2013
HOU2LEARN
• PLE from Hellenic Open University
• Social network analysis and final grades
• Online collaboration is not a predictor for final grade
• Koulocheri & Xenos, 2013
DMW UOC May 2013
OpenEssayist: SAFeSEA Web application for summarisation-based formative feedback
DMW UOC May 2013
localhost:8065
phaeros.open.ac.uk:80
openEssayist
PHP, Epiphany[Symfony2]
User
openEssayist RESTful API
PHP, Epiphany
User
User
pyEARESTful API
Python, Flask
localhost:8064
AfterTheDeadlineSpell/Grammar
checker
Java
User
localhost:9998
Apache TikaText Extractor
Java
Orchestrator
(Open)Learner Model
pyEssayAnalyser
Python, NLTK
localhost:8065
phaeros.open.ac.uk:80
openEssayist
PHP, Epiphany[Symfony2]
User
openEssayist RESTful API
PHP, Epiphany
User
User
pyEARESTful API
Python, Flask
localhost:8064
AfterTheDeadlineSpell/Grammar
checker
Java
User
localhost:9998
Apache TikaText Extractor
Java
Orchestrator
(Open)Learner Model
pyEssayAnalyser
Python, NLTK
Key words and phrases visualized in the essay context. Sentences in light-grey (green) background are key sentences as extracted by the EssayAnalyser (the number at the start of the sentence indicates its key-ness ranking);
bigrams are indicated in bold (red) and boxed.
DMW UOC May 2013
The structural elements of the essay can be used jointly with the key word extraction to highlight relevant information within specific parts of the essay, here the introduction (and the assignment question)
DMW UOC May 2013
Key words and phrases as separate lists
DMW UOC May 2013
Dispersion of key words across the essay http://www.open.ac.uk/iet/main/research-scholarship/research-projects/supportive-automated-feedback-short-essay-answers
DMW UOC May 2013
Can we find ways of using graph visualization techniques on the key words and key sentences, to make them helpful and meaningful to students?
DMW UOC May 2013
Final Thoughts
• Instant machine feedback not prevalent
• Artificial Intelligence analysis to tutors leading to Wizard of Oz responses (Shaffer, 2013)
• Just in time feedback is the ultimate goal
DMW UOC May 2013
References (1)Arnold, K.E. (2010). Signals: applying academic analytics, Educause Quarterly, 33(1), p10. http://www.educause.edu/ero/article/signals-applying-academic-analytics (Accessed 30 April 2013)
Bastien, M., Heymann, S. & Jacomy, M. (2009). Gephi: an open source software for exploring and manipulating networks. Paper presented at the International AAAI Conference on Weblogs and Social Media.
Bull, S & Kay, J. (2007). Student models that invite the learner in: the SMILI:-) open learner modelling framework, International Journal of Artificial Intelligence in Education, 17(2).
Campbell, J.P. & Oblinger, D.G. (2007). Academic Analytics, Educause.
Dawson, S., Bakharia, A. & Heathcote, E. (2010). Snapp: Realising the affordances of real-time SNA within networked learning environments. Paper presented at The 7th International Conference on Networked Learning, Aalborg, Denmark (3-4 May).
DMW UOC May 2013
References (2)
De Laat, M., Lally, V., Lipponen, L. & Simons, R.-J. (2007). Investigating patterns of interaction in networked learning and computer-supported collaborative learning: a role for social network analysis. International Journal of Computer Supported Collaborative Learning, 2, pp.87-103.
Koulocheri, E. & Xenos, M. (2013). Considering Formal Assessment in Learning Analytics within a PLE: The HOU2LEARN Case. Paper presented at The International Learning Analytics & Knowledge (LAK) Conference, Leuven, Belgium, (8-12 April 2013) © 2013 ACM 978-1-4503-1785-6/13/04
Martin, A.J. (2007). Examining a multidimensional model of student motivation and engagement using a construct validation approach. British Journal of Educational Psychology, 77, 413-440.
DMW UOC May 2013
References (3)
Mirriahi, N. & Dawson, S. (2013) The pairing of lecture recording data with assessment scores: A method of discovering pedagogical impact. Paper presented at The International Learning Analytics & Knowledge (LAK) Conference, Leuven, Belgium, (8-12 April 2013) © 2013 ACM 978-1-4503-1785-6/13/04
Pekrun, R. (2006). The control-value theory of achievement emotions: assumptions, corollaries and implications for educational research and practice. Educational Psychology Review, 18, 315-34.
SoLAR, Open Learning Analytics: An Integrated & Modularised Platform, White Paper, Society for Learning Analytics Research, 2011.
Suthers, D. & Verbert, K. (2013) Learning Analytics as a “Middle Space”. Paper presented at The International Learning Analytics & Knowledge (LAK) Conference, Leuven, Belgium, (8-12 April 2013) © 2013 ACM 978-1-4503-1785-6/13/04
DMW UOC May 2013
References (4)
Tempelaar, D.T., Cuypers, H., van de Vrie, E., Heck, A. & van der Kooij, H. (2013). Formative Assessment and Learning Analytics. Paper presented at The International Learning Analytics & Knowledge (LAK) Conference, Leuven, Belgium, (8-12 April 2013) © 2013 ACM 978-1-4503-1785-6/13/04
Van Ham, F., Schulz, H.-J. & Dimicco, J.M. (2009). Honeycomb: visual analysis of large scale social networks, Ifip International Federation for Information Processing, 429-442.
Van Labeke, N., Whitelock, D., Field, D., Pulman, S. & Richardson, J. (2013) ‘OpenEssayist: Extractive Summarisation & Formative Assessment of Free-Text Essays’. Workshop on Discourse-Centric Learning Analytics, 3rd Conference on Learning Analytics and Knowledge (LAK 2013), Leuven, Belgium
Vermunt, J.D. (1996). Leerstijlen en sturen van leerprocessen in het Hoger Onderwijs. Amsterdam/Lisse: Swets & Zeitlinger.
DMW UOC May 2013