16
CRICOS 00111D TOID 3059 Push notifications, digital badges & leaderboards: Evaluating the impact of Quitch mobile application on learning Dr Grainne Oates FLB, Senior Lecturer, Accounting/Finance Dr Katya Pechenkina LTU, Research Fellow Dan Laurence LTU (formerly), Learning Designer Dr Daniel Eldridge FSET, Lecturer in Chemistry Prof Dan Hunter FBL, Foundation Dean, Swinburne Law Presented at Transforming Learning Conference Swinburne University of Technology 13-14 September 2016 Melbourne Australia http://transformconference.com/ #SwinTLC

Push notifications, digital badges & leaderboards: Evaluating the impact of Quitch mobile application on learning

Embed Size (px)

Citation preview

Page 1: Push notifications, digital badges & leaderboards: Evaluating the impact of Quitch mobile application on learning

CRICOS 00111D TOID 3059

Push notifications, digital badges &

leaderboards: Evaluating the impact of

Quitch mobile application on learning

Dr Grainne Oates FLB, Senior Lecturer, Accounting/FinanceDr Katya Pechenkina LTU, Research FellowDan Laurence LTU (formerly), Learning DesignerDr Daniel Eldridge FSET, Lecturer in ChemistryProf Dan Hunter FBL, Foundation Dean, Swinburne Law

Presented at Transforming Learning Conference Swinburne University of Technology13-14 September 2016Melbourne Australia http://transformconference.com/ #SwinTLC

Page 2: Push notifications, digital badges & leaderboards: Evaluating the impact of Quitch mobile application on learning

Outline

• Mobile apps for learning

• Quitch: Rationale & elements

• Quitch: Development process

• Measuring impact

• Findings

• Further steps

Page 3: Push notifications, digital badges & leaderboards: Evaluating the impact of Quitch mobile application on learning

Mobile apps for learning

• Challenges: low retention & high failure rates

• Students likely to access content via mobile

phone (or a similar device) not desktop computer

• Push notifications

• Student expect immediate constructive feedback

• Self-testing has capacity to improve outcomes

• Multitude of mobile apps - rarely tailored to

lecture content

Page 4: Push notifications, digital badges & leaderboards: Evaluating the impact of Quitch mobile application on learning

Quitch: Rationale & elements

• Engagement loops

• Regular ‘push-notifications’

• Dynamic timed quiz

• Points & badges

• Leaderboards

• Progression trees

• Videos & forums

A solution should:

• Be easy for educators to use

• Have real time analytics for early intervention &

adaptive learning

Page 5: Push notifications, digital badges & leaderboards: Evaluating the impact of Quitch mobile application on learning

Quitch: Development process

Page 6: Push notifications, digital badges & leaderboards: Evaluating the impact of Quitch mobile application on learning

Quitch: Development process

Page 7: Push notifications, digital badges & leaderboards: Evaluating the impact of Quitch mobile application on learning

Quitch: Development process

Page 8: Push notifications, digital badges & leaderboards: Evaluating the impact of Quitch mobile application on learning

Quitch: Analytics

Possibilities:

• Reengage ‘at risk’ students

• Test if lecture content is comprehended

• Adapt course material dynamically

• Provide regular personalised feedback

Page 9: Push notifications, digital badges & leaderboards: Evaluating the impact of Quitch mobile application on learning

Measuring impact• Is there a relationship between introducing a

mobile app into the classroom and student

retention rates?

• Is there a relationship between the use of

the app and academic performance?

• Is student performance within the app

related to academic performance?

• Do students respond positively to the

introduction of the app?

Page 10: Push notifications, digital badges & leaderboards: Evaluating the impact of Quitch mobile application on learning

Measuring impact: Methods• Two cohorts taking part in the Quitch trial

(S2, 2015)

Accounting (462 enrolled, 265 app users)

Science (249 enrolled, 129 app users)

• Total sample: N=711, with 55% using the

app (394) + prior cohorts for comparison

• Data collected (with student consent):

Final grades compared to the pre-app cohort

App uptake & engagement analytics

Small-scale qualitative online survey (N=8)

Student feedback from end-of-year surveys

Page 11: Push notifications, digital badges & leaderboards: Evaluating the impact of Quitch mobile application on learning

Findings

Is there a relationship between introducing a

mobile app into the classroom and student

retention rates?

• The app introduction was associated with a

positive improvement in student retention

(calculated at 12.23%) compared to the pre-

app semester

Page 12: Push notifications, digital badges & leaderboards: Evaluating the impact of Quitch mobile application on learning

Findings

Is there a relationship between the use of the

app and academic performance?

• App users averaged a percentage mark of

65.19% compared to students who did not

use the app (who averaged a percentage

mark of 58.16%).

• App users on average achieved marks that

were 7.03% higher than students who did not

use the app

Page 13: Push notifications, digital badges & leaderboards: Evaluating the impact of Quitch mobile application on learning

Findings

Is student performance within the app related to

academic performance?

• For app user there was a positive correlation

of .40 between performing well on the app

and achieving higher academic grades

Page 14: Push notifications, digital badges & leaderboards: Evaluating the impact of Quitch mobile application on learning

Findings

Do students respond positively to the introduction of the

app?

The app “reinforced the content” & “kept [students]

involved and invested” by appealing to their competitive

nature

Leaderboards & badges: allowed students to see how they

placed compared to their peers. However: it is “dangerous

to start ranking people” & some students felt “exposed”

Immediate feedback was desired in addition to right/wrong

answer indicator

Page 15: Push notifications, digital badges & leaderboards: Evaluating the impact of Quitch mobile application on learning

Further steps

• Correlation ≠ causation

• In-depth studies of student &

educator experiences needed

• Implications for universities,

educators & students

• Tailored content, timed

content-aligned quizzes,

immediate feedback, rankings,

flexibility

Page 16: Push notifications, digital badges & leaderboards: Evaluating the impact of Quitch mobile application on learning

Thank youPresented by Dr Katya PechenkinaResearch FellowThe Office of the Senior DVC & ProvostLearning TransformationsEmail: [email protected]: @katya_pechenk