Learning Analytics - Improving Student Retention

SITS:Vision Annual Conference

Learning Analytics –Improving Student Retention

Paul Travill - Academic Registrar, University of Wolverhampton

Chris Ballard - Innovation Consultant, Tribal


“Learning analytics is the measurement, collection, analysis and reporting of data about learners and

their contexts, for the purpose of understanding and optimising learning and the environments in which it

occurs.”(George Siemens 2011)

What is Learning Analytics?


What is Learning Analytics?

Learning Analytics

Educational Data

Mining

Academic Analytics

Predictive modellingExtract value from big data sets

Business Intelligence applied to education at an institutional, regional and national level

Understand how students are learning and optimise the learning process


Contrasting Learning and Academic Analytics

Monitoring and benchmarking of university KPIs


Education and Big Data


Characteristics of Big Data

2011 Gartner Report

Variety VelocityVolume


Factors affecting retention

Preparation for higher

education

Academic Integration

Social Integration

Personal circumstancesEngagement

Demographic background

Course


Administrative Data Activity Data

Academic performance at

entrance

UCAS Application

Attendance

Engagement

Contact with support services

VLE Usage

Library Usage

Proximity Door access

Social background Module Grades

Course Enrolment

Fees

Engagement and Academic Integration

Predictive Model

Demographics Contact with tutors

Campus PC Usage

Social interaction

Possible future data source

Student factors


Administrative Data• Student

Administration System

• Known at time of enrolment

Activity Data• User interaction

with a system• Patterns of usage• Real time• Collected at scale• Change over time

Initial assessment of

risk

On going assessment of

risk


The dataset

• Data Warehouse• Data set spanning 2010/11 – 2011/12

academic years• Imbalanced data


Staff

Admin Data

Activity DataPredict likelihood of

withdrawal

Predict module grades

View profile of student interactions

Module Outcome Model

Retention Model

Activity Profile

Stud

ent

Comparison to similar students

Cluster students

Which things can we change that could make a difference?



Module: Basic Broomstick SkillsHarry is likely to achieve a grade D. Issues:VLE use is very low compared with the better performing students,Distance from home suggests he should have a higher VLE use profileSupport Recommendation:Suggest attendance at the additional Broomstick Study Skills sessions (Wednesday at 12.00 in the library) Click to make booking.

Module: Quidditch magic spellsHarry is likely to achieve a grade B. This matches the profile of other students in the profile cluster. Remember to encourage him to keep up the work!

Harry PotterCourse: BWiz QuidditchNOTES FOR PERSONAL TUTOR


ACTIVITY DATA


Activity DataStudent ID Date Time Asset ID

0000001 01/09/2012 12:03:01 1

0000001 01/09/2012 12:05:06 34

0000001 05/09/2012 16:46:23 17

0000005 17/10/2012 19:56:01 73

… … … …

Student ID Date Module Number of Transactions

0000001 01/09/2012 Abc 2

0000001 02/09/2012 Abc 0

0000001 03/09/2012 Abc 0

0000001 04/09/2012 Abc 0

0000001 05/09/2012 Abc 1

0000005 17/09/2012 Bcd 7

… … … …

Transactions Time Series


Activity Data Goals

• Convert to Time Series• Pre-process time series (e.g.

smoothing)• Derive measures which describe the

“shape” of the interactions• Use measures to help understand

whether some patterns of interaction are indicative of poor engagement


Extracting meaningful information from Activity Data

• Need to distinguish between students who are regular users of the service, and those who have sporadic high volumes of access (but aggregate volume may be similar

• Acts as a proxy to how well the student is “engaged” with the service

is better than

But may have similar overall numbers of transactions


Symbolic Aggregate approXimation (SAX)

abcdbbbdeaaddbae

Encodes the shape of the time series as a series of character strings

Enables us to cluster together students with similar interaction patterns, or classify interactions (as indicative of students who ultimately withdrew)

Turns out to work well for high volumes of interactions, but not so well for intermittent time series as there is less “shape” to encode.

Smoothing the time series may help


Derive high level measures

• Proportion of days accessed resource• Average number of transactions per

day accessed• Run Length Distance Ratio

Turns out to work better for Library and VLE activity datawhere interactions are much more intermittent


School resource profiles• Low VLE usage does not mean the same thing for every

student!• Need to weight Library and VLE features to take into

account different resource profiles


Challenges with activity data


BUILDING THE MODEL


SITS VLE Library Active Dir.

Demographics, Enrolment, Activity Data, …

Test examples (30%)

TransformPredictions

New

dat

a

1. Train model

2. Test model

Model

Data Warehouse

Calculated FeaturesDerived Features

Training examples (70%)

Other Data SetsOther Data Sets


Performance of the model

Admission

• Demographics

• Course• Academic

performance on entry

• Distance

Semester 1

• Modules Failed

• Modules Passed

• Credit points

VLE ActivityLibrary Activity

VLE ActivityLibrary Activity

Lower Performance

Higher Performance


Challenges with the model

Imbala

nced classe

s

Imbalanced classes require the need to adjust data to ensure model has adequate performance

Training d

ata

Its all about the training examples and features!

History

Training the model with examples representing the progress the student makes at different times

Perfor

mance

Performance improves as the academic year progresses


Chris BallardInnovation Consultant, Tribal

twitter: @chrisaballardblog: triballabs.net