JISC RSC London Workshop - Learner analytics

Learning AnalyticsWhat? How? Why?

James Ballard

jameslballard

JamesBallard2

@jameslballard

Overview

What are Learning Analytics?

Learner Engagement - a metric for learning

Preparing institutions – tools and skills

Infinite Rooms

Open Discussion

What are learning analytics?Who are they for?

Activity 1 - Introduction

4

Learning AnalyticsWhat are they?

Not quite big dataIn 2012 we created 2,500,000,000,000,000,000 (2.5

quintillion) bytes of data every day

Annual Moodle log data 5Gb

Learning AnalyticsType of Analytics Level or Object of Analysis Who benefits

Learning Analytics Course-level: social networks, conceptual development, discourse analysis, intelligent curriculum

Learner, faculty

Departmental: predictive modelling, patterns of success/failure

Learners, faculty

Academic Analytics Institutional: learner profiles, performance of academics, knowledge flow

Administrators, funders, marketing

Regional (state/provincial): comparisons between systems

Funders, administrators

National and International National governments, education authorities

Siemens and Long (2011)

Common focus• Identifying learners at-risk of drop-out from

the course• Identifying momentum/crisis points

Retention• Predicting final exam success• Predicting future performance (e.g. school ->

university)Performance

• Quantitative views of activity• What are learners doingActivity• Usually linked to a bench-marking of staff

performance• Learning design patterns

Course• What types of things are learners doing• Learner engagement as a metric/proxyEngagement

Small groupsList some examples of what learning providers are measuring or might want to measure.

Activity 2 - Examples

Retention Performance Activity Course Engagem

ent

9

Learner EngagementA metric for learning

EngagementCommon activities such as checking announcements, viewing grades and uploading assignments represent little time investment from the user and may not be useful indicators of engagement.

MacFadyen and Dawson (2012)

Engagement

Engagement has emerged as an alternative view of the learner experience that can enrich the often reductionist language of performance, skills and competence.

HEA, Trowler and Trowler (2010)

Engagement ProcessEngagement is the new metric that supersedes previous linear metaphors, through a developmental process of discovery, evaluation, use, and affinity.

Haven (2007)

Small groupsTag previous examples within the engagement process.

Activity 2 - Examples

Involvement

• The presence of a learner within the institution including data such as physical or virtual visits

Interaction

• Provides a depth of understanding: where involvement measures touches, interaction measures actions.

Intimacy

• Helps understand sentiment and affection; the most common way to collect this type of data is through interviews or surveys.

Influence

• Determines the likelihood of the individual recommending learning to others and contributing to local culture(s).

InvolvementThe presence of a learner within the institution including data such as physical or virtual visits.

Overall Activity

Locations

Time of day

InvolvementThe presence of a learner within the institution including data such as physical or virtual visits.

Overall Activity

Locations

Time of day

InteractionProvides a depth of understanding: where involvement measures touches, interaction measures actions.

Activity types

Action analysis

Connectivity maps

Conole (2007)

InteractionProvides a depth of understanding: where involvement measures touches, interaction measures actions.

Activity types

Action analysis

Connectivity maps

IntimacyHelps understand sentiment and or affection; the most common way to collect this type of data is through interviews or surveys.

Learning Power

Self-theory

Motivated Strategies for Learning Questionnaire, MSLQ

Self-determination theory

Deakin Crick, Broadfoot, and Claxton (2004)

IntimacyHelps understand sentiment and or affection; the most common way to collect this type of data is through interviews or surveys.

Learning Power

Self-theory

Motivated Strategies for Learning Questionnaire, MSLQ

Self-determination theoryRehearsal Elaboration Organisation Self-Regulation Critical Thinking

0

1

2

3

4

5

6

MSLQ

Pre Post

Pintrich (1990)

InfluenceDetermines the likelihood of the individual recommending learning to others and contributing to local culture(s).

Social Network Analysis

Distributed Cognition

Collective Intelligence

Pathway of ParticipationDawson (2010)

InfluenceDetermines the likelihood of the individual recommending learning to others and contributing to local culture(s).

Social Network Analysis

Distributed Cognition

Collective Intelligence

Pathway of Participation

School Leader Network

Harré (1983)

22

Preparing InstitutionsEmpowering environments for learning

You are what you measure

Measure what you get

Metrics are based on the data that is easiest to extract/access, and what you don’t measure is lost.Get what you measure

Anything you measure will impel a person to optimize his score on that metric. Data is not neutral Don’t be surprised if people find ingenious and destructive ways in how they get there. For example, standardised assessment produce kids who perform well on these tests but can falter when asked to demonstrate their knowledge of the same material in a different way

‘Incremental change is not enough. You have to drive large-scale change by changing the environment in which people work’

– Kevin Bonnett, Deputy Vice Chancellor Student Experience

JISC Report

MMU Review

Open Discussion

What types of skills are required by e-learning teams?Do they already exist?

Activity 1 - Introduction

Analytics Process

Collection Storage Cleaning Integration Analysis Presentati

on

CETIS Analytics Series (2012)

Open Source Tools

Data Storage

MySQL / PostgreSQL Apache Hadoop HP VerticaData Mining

Pentaho Rapid Miner Social Network Analysis Gephi Visualisation Google Visualisation d3.js InfoVis Toolkit

Investing in staff experimentation with low cost components from a range of traditions may be a more prudent initial move, even if the most effective tool subsequently turns out to be a ready-made suite.

Data Mining

Algorithm Usage Purpose

Step regression Used for binary classification (0,1)• Select a

parameter• Assign a weight• Calculate value

Predicts simple binary results such as is a student at-risk?

Logistic regression Used for binary classification (0,1)

Same as above but more conservative

J48/C4.5 Decision trees (Quinlan, 1993)

Tries to find optimal split in variables

Good when data splits into groups

JRip Decision rules Find the “best” path and make this a rule until no sensible paths are left and set these to otherwise.

Good when multi-level interaction are common

K* Instance based classifiers

Predicts data based on neighbouring points.

Good when data is very divergent

Random Forest

Classification is used when one wants to predict something (label) which is categorical and not a number.

28

Infinite RoomsLearner enhanced technology

Research Project

ScopeWeb dashboards based on engagement process accessing a data warehouse model developed from Activity Theory. Utilises new and existing analytics and supports multiple learning design approaches.Objectives1. How can student activity help identify

and promote effective teaching practices?

2. Understand the role that analytics can play in learning design, feedback and assessment.

3. Explore how student contributions can provide dynamic indications of success.

If patterns of nonparticipation (disengagement) are to be disrupted an improved conceptual framework may be necessary.

Activity AnalysisEngeström’s (1987, 1999) approach allows us to overcome oppositions between activity and communication and highlight subject-community relations.

Modelling pedagogy with Activity Theory

Stevenson (2008)

http://goo.gl/vOuiqp

Exposing ActivityThe intention of this is to reveal the nature of the system, allowing designers (e.g. teachers) to evaluate the system in the wider context of their teaching and learning practice.

Data Model

Dimension

Fact

Action Post to forum

Tool Forum

Instance Discussion topic

User Oliver Twist

Role Student

Course Introduction to English

Date 02/10/2013

Time 9:45

System Moodle

Enables multi-dimensional tagging to explore data from different perspectives.

Data Capture

Things a learner doesActions• Submissions• Quiz attempts • Forum posts

Feedback to the learnerIntervention

s• Targets• Grades • Assignment feedback

Recognising learningAchieveme

nts• Course completions• Badges• Certificates

How learning is perceivedSurveys• Attitudes to learning/technology• Satisfaction survey

What types of things can we capture.

CodingOne can then begin to distinguish the possible actions that are generated through the use of tools from the operations needed to access them and code these via learning design theories.

Activity TypesApply learning design models to learner data.

Conole (2007)

VisualisationExplore different visualisations of the same data set for different insights.