An enhanced learning analytics plugin for Moodle: Student engagement and personalised intervention

An enhanced learning analytics plugin for Moodle

STUDENT ENGAGEMENT AND PERSONALISED INTERVENTION

Monday 30 November 2015 | ascilite 2015

Learning analytics

in MoodleCONTEXTS AND APPROACHES

Danny Liu

Learning analytics in MoodleWHY?

• Benefits

• Familiarity for staff

• Single point of access

• Student activity already in database – what data are most valuable?

• Open source

http://analytics.jiscinvolve.org/wp/files/2015/10/Learning-analytics-framework.jpg

e

Learning analytics in MoodleBUILT-IN REPORTING

LogsActivity report

Activity completionStatistics report

https://moodle.org/plugins/view/block_analytics_graphs

Learning analytics in MoodleCURRENTLY AVAILABLE PLUGINS - VISUALISATION

Analytics GraphsGISMO

https://moodle.org/plugins/block_gismo

Moodle Activity ViewerMOCLog

Mazza et al. (2012)

Jones & Clark (2014)

Learning analytics in MoodleCURRENTLY AVAILABLE PLUGINS - ANALYSIS

SmartKlass

https://moodle.org/plugins/local_smart_klass

Engagement Analytics

https://moodle.org/plugins/browse.php?list=set&id=20

Learning analytics in MoodleA FRAMEWORK FOR EVALUATION AND IMPROVEMENT

• IRAC framework (Jones, Beer, & Clark, 2013)

Jones, D., Beer, C., & Clark, D. (2013). The IRAC framework: Locating the performance zone for learning analytics. In: Proceedings of ascilite 2013, Sydney.

Only and all relevant information available

Needs to aid the task being undertaken

Providing support for context-appropriate action

Iterative development of other three factors

Information

Representation

Affordances for action

Change

Moodle Engagement AnalyticsPLUGIN OVERVIEW

• Originally developed by Phillip Dawson, Adam Olley, and Ashley Holman

Moodle

Report

Logins Forums Assessments

Parameters

Traffic lights

BYO

Moodle Engagement Analytics Plugin

Indicators

Action

Requirements

gathering and

iterative designANALYSIS AND DESIGN

Chris Froissard

?

Extend capability and functionality of

MEAP

++

1) IV1-Unit convenor survey around expectations for student early alert system

2) IV2-Unit convenor survey around use of MEAP

4) User testing with use case (5 video sessions)

3) Prototyping-development of interface

5) Refinement of design based on usage and feedback

Overview of design process

What information woud you like to see to help you decide what to include in your alert?

What would you find most useful about a student alert system?

Collect information about expectations



Prototyping




Prototyping: concepts

Prototyping: paper designs

Prototyping: mockups

User testing





User testing: scripts

User testing: recordings





5) Refinement of design based on usage and feedback

Refinement of design

Refinement of design based on UAT

Refinement of design based on

usage in pilot

22

Moodle Engagement

Analytics PluginDEVELOPMENT, IMPLEMENTATION,

AND EVALUATION

Danny Liu

EnhancementsMODIFYING AND ADDING INDICATORS

I

R

A

C

Assessments Gradebook

• Submission of quizzes, assignments

• Added Turnitin

EnhancementsALTERNATIVE REPORT

I

R

A

C

• Less abstracted representation of student engagement data

Enhancements

26

AFFORDANCES FOR ACTION

• Customisable, personalised messages directly to students

• Suggested and other snippets (some from PassNote)

• Save messages for re-use

I

R

A

C

MEAP+

27

ENHANCED PLUGIN FOLLOWING IRAC FRAMEWORK

Moodle

Report

Logins Forums Assessments

Parameters

Traffic lights

Gradebook

Moodle Engagement Analytics Plugin

Action

Preliminary outcomes

28

IMPACT OF MEAP AT MACQUARIE

• 3400+ personalised emails sent, average ~46% opened

0

50

100

150

200

250

300

350

01 A

ug

08

Au

g

15 A

ug

22

Au

g

29

Au

g

05

Se

p

12 S

ep

19 S

ep

26

Sep

03

Oct

Nu

mb

er o

f st

ud

ents

0

20

40

60

80

100

Are y

ou

ha

vin

g p

rob

lems a

ccessing

iLea

rn?

AC

CG

25

0 - If yo

u sn

oo

ze yo

u lo

se!

Are y

ou

ha

vin

g tro

ub

le log

gin

g in

to iL

earn

?

Are y

ou

ha

vin

g tro

ub

le log

gin

g in

to iL

earn

in A

CC

G10

0?

ME

CH

20

1: En

gin

eering

Dy

nam

ics - iLe

arn A

ctivity

EN

GG

150

qu

iz feed

ba

ck

EC

ON

110 iL

earn

Lo

gin

AH

IS14

0 M

yth in

the A

ncien

t Wo

rld

AH

IS14

0 M

yth in

the A

ncien

t Wo

rld

Are y

ou

ha

vin

g tro

ub

le log

gin

g in

to E

CO

N111

AH

IS14

0 S

ession

2 P

articip

ation

AH

IS14

0 M

yth in

the A

ncien

t Wo

rld

Qu

iz 1

Qu

iz 1 Resu

lt - Afin

25

3

Ilearn

- Afin

25

3

AC

ST

101 - Q

uiz 1

EC

ON

110 T

uto

rial S

ub

missio

ns

EC

ON

111 - Tu

toria

ls con

tribu

te to y

ou

r fina

l gra

de

EC

ON

111 - Tu

toria

ls con

tribu

te to y

ou

r fina

l gra

de

ME

CH

20

2 P

rog

ress thu

s far

ME

CH

20

3 P

rog

ress th

us fa

r

ISY

S114

AC

ST

101 - L

ast d

ay to

with

draw

with

ou

t fail ch

arg

e is…

ISY

S114

Perfo

rma

nce

ISY

S114

week

ly tuto

rial su

bm

ission

s

ISY

S114

Atten

da

nce

and

Perfo

rma

nce

ISY

S114

Wo

rksh

op

Atten

da

nce a

nd

Pe

rform

an

ce

ISY

S114

Wo

rksh

op

atten

da

nce

Tu

torial feed

ba

ck

Tu

torial p

articip

atio

n

Tu

torial p

articip

atio

n fo

r EU

L10

1 So

cieties of E

uro

pe

AC

ST

101 - M

on

day

is the last d

ay

to w

ithd

raw

with

ou

t…

AC

ST

101 - M

on

day

is the last d

ay

to w

ithd

raw

with

ou

t…

AC

ST

101 - M

on

day

is the last d

ay

to w

ithd

raw

with

ou

t…

Its no

t too

late

to g

et help

!

AC

ST

101 - M

on

day

is the last d

ay

to w

ithd

raw

with

ou

t…

EC

ON

111 - Plea

se check

yo

ur tu

toria

l record

s

EC

ON

111 - plea

se check

you

r tuto

rial reco

rds

Per

cen

tag

e o

pen

ed

1 hour 1 day 1 weekEmail opened within:


29

IMPACT OF MEAP AT MACQUARIE

• Who: unit convenors and student support staff

• What: census, updates, reminders

• Why: predominantly for at-risk students

• How: logins, assessment submissions, grades, attendance

I was surprised someone cared/was actually monitoring, kind of a weird, I don't know totalitarian/'people are watching you' feeling? But in this situation I was happy.

I'm glad I was contacted as it motivated me to complete the work and almost "show" the unit convenor that I could do it.

He gave me specific advice and encouraged me and it made me feel much better.

The email basically kicked me into gear and I completed all my assessments post-email to a high level.

Very useful. I wouldn't have been able to do such a large scale analysis and identify so many students without MEAP. I wouldn't have been able to send them such tailored, structured and consistent messages.


30

THE DARK SIDE

• Issues around

• Message composition

• Suggestions

• Student contexts

• Being labelled

I was contacted but in a manner that suggest I should drop out of the course and not waste the convenor's time. She didn't ask whether I was experiencing any issues outside of university, but simply that I should transfer course if I can't handle the workload.

Being labelled as lazy when you're doing your best and don't have any other choice is quite sad.

Being aware and then being told of my own inadequacies is confronting and, yes, does make me feel worse about life in general. It's something I need to be told and is still that extra bit of motivation.

The email was worded in a way that it the unit [convenor] was trying to tell me I was doing horrible and should drop out and didn't refer any help.

Future directions

31

NEXT STEPS FOR MEAP+

• Further work

• Measuring impact of interventions

• Automated determination of parameters (machine learning)

• Investigating staff use

• Alternative forms of representation

• Opening source

• Institutionalisation

AcknowledgementsIT TAKES A VILLAGE

• Students

• Unit convenors

• Student support staff – Janise Farrell

• Learning systems – James Hamilton, Jaymie Parker, Michael Mendoza

• Reference group

• Team members – Deborah Richards, Amara Atif

• 2015 Macquarie University Teaching Development Grant

Get in touch

@dannydotliu

@jcfroissard

[email protected]

[email protected]

Education

An enhanced learning analytics plugin for Moodle: Student engagement and personalised intervention