Learner Enhanced TechnologyCan activity analytics support understanding engagement a measurable process?

James Ballard – HETL SOTE 2015, Utah Valley University

jameslballard

JamesBallard2

@jameslballard

My Research Questions

What can online learner activity tell us about engagement?

How can this be used to inform learning design?

“As we start a new school year, Mr. Smith, I just want you to know that I’m an Abstract-Sequential learner and trust that you’ll conduct yourself accordingly!”

Overview

Icebreaker activity

Introduction

Activity 1 – Classification

Activity 2 – Engagement Profile

Conclusion/Discussion

Icebreaker ActivityShould engagement constitute a formal assessment of teaching?5 mins – introduce yourselves and give initial reactions to this question

IntroductionPersonalised Learning and Engagement

The challenge

Course Delivery (narrow

curriculum)

Learner Experience

(broad curriculum)

Personalised Learning

Framework

Personalised Learning Framework

Ballard & Butler (2011)

Integrated VLE

Live@EDUTurnitin

Twitter

YouTube SlideShare

Diigo

Facebook

Ning

GoogleDocs

Blogger

Recognised: user’s choice - watch and learn

Recommended: user’s choice – train and integrate

Arranged: institutional provision with training

Core: institutional provision with compulsory use and consistent access

LRT @ MMU

Implementation Approach

Outcomes

JISC Transforming Curriculum Delivery

Retention: 62% to 92%

Achievement: 69% to 73%

Anecdotal evidence that students felt more supported

JISC Distributed Learning Environments

Student satisfaction for course organisation up 1.8%

Student satisfaction for appropriate resource selection up 3.5%

Guardian University Award for Student Experience 2013

EUNIS Elite Award for Excellence 2014

LeSoCo College 2008-2010 MMU 2010-2011

PERFORMATIVITY AND POWER

Constant supervision and correction constrains learners through the need

to achieve conformity

HUMANISM AND DEMOCRATIC EDUCATION

Reflection

Ballard (2014)

Problem solving

Human expression

Lived experience

Can we design learning systems that promote more of the right than the left?

Learner Voice – Design for Engagement

Teacher Learner

Involvement Interaction Intimacy Influence

Engagement as Metric

What to track

• Site visits• Time spent• Pages viewed• Search

keywords• Navigation

paths• Site logins

• Contributed comments to blogs

• Quantity / frequency of written reviews, blog comments, forum posts

• Sentiment tracking on 3rd

party sites• Sentiment

tracking of internal sites

• Opinions expressed incustomer calls

• Net Promoter (NP) score

• Servicesatisfaction ratings

• Brand affinity• Content

forwarded tofriends

How to track

• Web Analytics • eCommerce• Social Media

• Surveys• Brand

monitoring• Service calls

• Surveys• Brand

monitoring

Haven & Vittel (2008)

Haven (2007)

Engagement Profile

Can this be appropriated to educational technologies?

Haven (2007)

Delineation

• Extract logs

• Summary statistics

Activity Analysis

• Activity Structure

• Activity System

Instrument Formation

• Pedagogical Model

• Task types

Application• Engagement profile

Methodology

Further Research

Activity 1 – Instrument FormationWhat might we measure?

Conole’s Task TypesAssimilative

• attending and understanding content;

Information Handling

• gathering and classifying resource or manipulating data;

Adaptive

• use of modelling or simulation software;

Communicative

• dialogic activities (e.g. pair dialogues or group based discussion);

Productive

• construction of an artefact such as a written essay, new chemical compound or a sculpture;

Experiential

• practising skills in a particular context or undertaking an investigation

Extracted Logs SampleDate Course Tool Action

03/01/2014 10:03 CIH L3 Cert in Housing Practice Programme

Course View

03/01/2014 10:03 CIH L3 Cert in Housing Practice Programme

Course View section

03/01/2014 10:03 CIH L3 Cert in Housing Practice Programme

Forum View discussions

03/01/2014 10:03 CIH L3 Cert in Housing Practice Programme

Forum Add post

03/01/2014 10:04 CIH L3 Cert in Housing Practice Programme

Forum View discussion

09/01/2014 08:51 CIH L3 Cert in Housing Practice Programme

Assignment View

09/01/2014 08:52 CIH L3 Cert in Housing Practice Programme

Assignment View confirmation form

09/01/2014 08:53 CIH L3 Cert in Housing Practice Programme

Assignment Submit

Case Study Activity 1

• Extracted log data is provided for a learner;

• For each log decide which of Conole’s task types this represents - keep a tally in the worksheet;

• A log may be none of the types and excluded but cannot more than one type;

• (Optional) Complete the pie and/or column charts

Activity 1 Discussion

• What type of tasks is this learner undertaking?

• How might this be used to inform learning design?

• What other classifications might you use?

In Small Groups: 20 minutes

Activity 2 – Practical ApplicationCreating an Engagement Profile

Involvement Interaction Intimacy Influence

Appropriating Engagement Metrics

What to track

• Site visits• Time spent• Pages viewed• Search

keywords• Navigation

paths• Site logins

• Contributions to class

• Quantity and frequency of content

• Type ofcontent accessed

• Learning dispositions

• Discourse tracking Social network analysis

• Evaluation surveys

• Links to outside world

• Extra-curricular contributions

• Practice based research and reflection

How to track

• System Logs• Web Analytics

• System Logs• Learning

Design

• Surveys• Content

analysis

• StudentDevelopment Model

Case Study Activity 2

• Data for two students on are provided – focus on student 1.

• Use the engagement metric to make notes under each phase of the relevant profile in the worksheet;

• Rate each phase from low to high at the bottom;

• (Optional) Make notes on extra data you would like to have for each phase

Activity 2 Discussion

• What can online learner activity tell us about engagement?

• If we extend the data we collect can this be used to design for engagement?

• What other types of data would be useful for engagement?

In Small Groups: 35 minutes

INVOLVEMENT INTERACTION INTIMACY INFLUENCE

• Above average activity on site, but below in course.

• Regular access throughout year

• Primarily course access

• Accessed throughout day

• Interacts with peers via forums

• Submitted assignments

• Apparent focus on low commitment activities

• Potential strategic learning strategy

• Forum posts directed to or in response to teacher

• Low peer-to-peer posts

• No shared links

High Low-Medium Low-Medium Low

Student Profile 1: Jo March

INVOLVEMENT INTERACTION INTIMACY INFLUENCE

• Anomalously high activity profile

• Intermittentaccess

• Interactspredominantly with course materials

• No assignment submissions

• Potential surfacelearning strategy

• Peer interactions not present

• No shared links

Low Low Low Low

Student Profile 2: Oliver Twist

Learner Enhanced Technology

Understanding engagement as a measurable process

Thank You

James Ballard

jameslballard@outlook.com

JamesBallard2

@jameslballard

jameslballard