Learning Analytics - Competen-SEAcompeten-sea.eu/wp-content/uploads/2018/02/... · Recommendations for designing dashboards 1. Define the goal of the dashboard and desired impact

LearningAnalytics–

what,how,why

MarenScheffelIoanaJivet

OpenUniversiteitNederland

Outline

•  Whatislearninganalytics?•  Whereareweat?•  Howaredashboardsused?•  Howcanweevaluatelearninganalyticstools?•  Howaboutaninstitutionallearninganalyticspolicy?•  Wheredowegofromhere?

Insight is the buzzword in the data revolution. Peoplewho have insight into their behaviour will be able toworkmoreefficientlyfromnowon.Thisequallyappliesto organisations, higher education institutions andnations. The digital revolution has brought about atremendous increase in the volume of data and it iscomparativelyeasytoaccess.Itwouldseemtobealostopportunitytoignoresuchdata.

MarjoleinvanTrigt(2016)

https://cdn.static-econo

mist.com

/site

s/de

fault/files/im

ages/prin

t-ed

ition

/201

5022

8_LD

P001

_0.jp

g

TheUnitedNationsarecallingforactions:

“dataarethelifebloodofdecision-making”

“the new world of data,a world in which dataare bigger, faster andmore detailed than everbefore”

But what about theeducationaldomain?

Dataanalysisiscommon inbusinessandcommerce.

http://arcbridge.com

/wp-conten

t/up

loads/20

15/07/bu

siness-analytics.pn

g

Collecting,storingandanalysingstudentrecords

hasalwaysbeendone

inonewayoranother.

http://balfour.rb

e.sk.ca/sites/balfour/files/Da

vid-Oscar-Brown-Stud

ent-Re

cord-side-2.jpg

http://abo

vethelaw

.com

/wp-conten

t/up

loads/20

13/03/file-cabine

t1.jp

g

learningacquisitionofknowledgeorskills

throughstudy,experience,orbeingtaught

analyticssystematiccomputationalanalysisofdataor

statisticsor

informationresultingfromthesystematicanalysisofdataorstatistics

Learning analytics is the measurement, collection,analysis and reporting of data about learners andtheir contexts, for purposes of understanding andoptimising learningand theenvironments inwhichitoccurs.

Siemens(2011)

Outline


LevelsofLearningAnalytics

LearningAnalyticsFramework

Greller,W.andDrachsler,H.(2012).TranslatingLearningintoNumbers:AGenericFrameworkforLearningAnalytics.EducationalTechnology&

Society,15(3):42–57.

LAshouldbeforteachersandlearners

LASophisticationModel

Siemens,G.,Dawson,S.,&Lynch,G.(2014).ImprovingtheQualityandProductivityoftheHigherEducationSector–PolicyandStrategyforSystems-LevelDeploymentofLearningAnalytics.Canberra,Australia:OfficeofLearningandTeaching,AustralianGovernment.

Drachsler,H.andGreller,W.(2016).PrivacyandAnalytics–it’saDELICATEissue.AChecklisttoestablishtrustedLearningAnalytics.InProceedingsoftheSixthInternationalConferenceonLearningAnalyticsandKnowledge,LAK’16,pages89–98,NewYork,NY,

USA.ACM.DOI:10.1145/2883851.2883893.

http://www.laceproject.eu/ethics-privacy-learning-analytics/

http://www.eugdpr.org/

Outline


LearningAnalyticsDashboards

http://w

ww.learno

nsolutions.com

/wp-conten

t/up

loads/20

16/10/learning-analytics-learno

n-85

0x40

0.jpg

Structure

Source:https://www.flickr.com/photos/bruneluniversity/12926707974/

Learninganalyticsdashboardsare…

…“singledisplaysthataggregatedifferentindicatorsaboutlearner(s),learningprocess(es)and/orlearningcontext(s)intooneormultiplevisualisations”Targetusers:teachers(75%),learners(51%),administratorsandresearchersLearningscenarios:formal(91%),non-formal,informalPurpose:Supportinformeddecisionmaking

Schwendimann,B.A.,Rodriguez-Triana,M.J.,Vozniuk,A.,Prieto,L.P.,Boroujeni,M.S.,Holzer,A.,...&Dillenbourg,P.(2017).Perceivinglearningataglance:Asystematicliteraturereviewof

learningdashboardresearch.IEEETransactionsonLearningTechnologies,10(1),30-41.

Thelearninganalyticsprocess

Verbert,K.,Duval,E.,Klerkx,J.,Govaerts,S.,&Santos,J.L.(2013).Learninganalyticsdashboardapplications.AmericanBehavioralScientist,57(10),1500-1509.

Visualisation

Indicators

Educationalfoundation

• Barchart• Linegraph• Piechart• Scatterplot• Socialgraphs• Text

• Learnercharacteristics• Contextoflearning• Content• Learneractions• Learningoutcomes• Socialinteractions

• Feedbacktheory• Self-regulatedlearning• Collaborativelearning• 21stcenturyskills• Bloom’staxonomy• Socialcomparison

Learninganalyticsdashboards

Learninganalyticsdashboards

Jivet,I.,Scheffel,M.,Drachsler,H.,andSpecht,M.(2017).Awarenessisnotenough.Pitfallsoflearninganalyticsdashboardsintheeducationalpractice.InProceedingsofthe12thEuropean

ConferenceonTechnologyEnhancedLearning(EC-TEL2017),LNCS,pages82-96.Berlin.Springer.

Learninganalyticsdashboardsforteachers

Monroy,C.,Rangel,V.S.,&Whitaker,R.(2013).STEMScopes:contextualizinglearninganalyticsinaK-12sciencecurriculum.InLAK’13(pp.210-219).ACM.

Learninganalyticsdashboardsforlearners

Corrin,L.,&deBarba,P.(2014).Exploringstudents’interpretationoffeedbackdeliveredthroughlearninganalyticsdashboards.ASCILITE2014.pp.629-633.


Beheshitha,S.S.,Hatala,M.,Gašević,D.,&Joksimović,S.(2016).Theroleofachievementgoalorientationswhenstudyingeffectoflearninganalyticsvisualizations.InProceedingsLAK’16(pp.54-63).

ACM.


Aljohani,N.R.,&Davis,H.C.(2013).Learninganalyticsandformativeassessmenttoprovideimmediatedetailedfeedbackusingastudentcenteredmobiledashboard.NGMAST2013,pp.262-267.IEEE.

Learninganalyticsdashboardsforteachers


TheLearningTracker

Designgoals1.  Traceable2.  Scrutable3.  ActionableIterativedesign1.  Interactive2.  Planningahead3.  Framing

TheActivityWidget

TheLearningPathWidget

Recommendationsfordesigningdashboards

1.  Define the goal of the dashboard and desired impact on the learners.

2.  Research educational concepts as theoretical grounding. 3.  Describe data that supports the goal. 4.  Experiment with different visualisations. 5.   Customise the dashboard to provide the same support to all

users. 6.  Integrate the dashboard into the learning activities. 7.  Focus dashboard evaluation on goal fulfillment, impact on

learners' affect and motivation, and on the usability of the tool. 8.  Tool's usability should also assess how learners interpret

information. 9.  Use multiple data sources to validate dashboard effects.

Jivet,I.,Scheffel,M.,Drachsler,H.&Specht,M.(2017).Awarenessisnotenough.Pitfallsoflearninganalyticsdashboardsintheeducationalpractice.EC-TEL2017.

Outline


Whereistheevidence?

http://www.laceproject.eu/evidence-hub/

Learninganalyticstools

•  Demandsofthestakeholders?•  Empiricalevidence?•  Desiredeffect?•  Indicatorsfortoolevaluation?•  Comparability?•  Standardisedevaluation?•  Impact?•  Benefit?

TheEvaluationFrameworkforLA

Finally, if there isn’t a relevant questionnaire foryoursituation,youcandeviseone.

JohnBrooke(2013)

ObjectivesforEFLAcreation

•  Identifyqualityindicatorsforlearninganalytics

•  Createanapplicableevaluationinstrumentbasedontheseindicators

•  Validatetheevaluationinstrument

EFLA1

ObjectivesAwareness Reflection Motivation Behavioural

Change

Learning SupportPerceived Usefulness Recommendation Activity

ClassificationDetection of

Students at Risk

Learning Measures and OutputComparability Effectiveness Efficiency Helpfulness

Data AspectsTransparency Data Standards Data Ownership Privacy

Organisational AspectsAvailability Implementation Training of Educa-

tional StakeholdersOrganisational

Change

EFLA2

DATA-I know what data is being collected-I have access to my data-I understand the presented results

L E A R N E R S

-I know what data is being collected-I have access to my students’ data-I understand the presented results

AWARENESS-I am aware of my current learning status-I comprehend my current learning status-I can project my future learning status

-I am aware of my students’ current learning status-I comprehend my students’ current learning status-I can project my students’ future learning status

REFLECTION-I reflect on my learning activities-I reflect on alternative learning activities-I know when to change my behaviour

-I reflect on my teaching activities-I reflect on alternative teaching activities-I know when to change my behaviour

T E A C H E R S

IMPACT-I can detect whether I am falling behind-I study more efficiently-I study more effectively

-I can detect whether my students are falling behind-My students learn more efficiently-My students learn more effectively

EFLA3

DATA-For this LA tool it is clear what data is being collected-For this LA tool it is clear why the data is being collected-For this LA tool it is clear who has access to the data

L E A R N E R S

-For this LA tool it is clear what data is being collected-For this LA tool it is clear why the data is being collected-For this LA tool it is clear who has access to the data

AWARENESS-This LA tool makes me aware of my current learning situation-This LA tool makes me forecast a possible future learning situation given my (un)changed behaviour

-This LA tool makes me aware of my students’ current learning situation-This LA tool makes me forecast my students’ possible future learning situation given their (un)changed behaviour

REFLECTION-This LA tool stimulates me to reflect on my past learning behaviour-This LA tool stimulates me to adapt my learning behaviour if necessary

-This LA tool stimulates me to reflect on my past teaching behaviour-This LA tool stimulates me to adapt my teaching behaviour if necessary

T E A C H E R S

IMPACT-This LA tool increases my motivation to study-This LA tool stimulates me to study more efficiently-This LA tool stimulates me to study more effectively

-This LA tool increases my motivation to teach-This LA tool stimulates me to teach more efficiently-This LA tool stimulates me to teach more effectively

EFLA4

DATA-For this LA tool it is clear what data is being collected-For this LA tool it is clear why the data is being collected

L E A R N E R S

-For this LA tool it is clear what data is being collected-For this LA tool it is clear why the data is being collected

AWARENESS & REFLECTION-This LA tool makes me aware of my current learning situation-This LA tool makes me forecast my possible future learning situation given my (un)changed behaviour-This LA tool stimulates me to reflect on my past learning behaviour-This LA tool stimulates me to adapt my learning behaviour if necessary

-This LA tool makes me aware of my students’ current learning situation-This LA tool makes me forecast my students’ possible future learning situation given their (un)changed behaviour-This LA tool stimulates me to reflect on my past teaching behaviour-This LA tool stimulates me to adapt my teaching behaviour if necessary

T E A C H E R S

IMPACT-This LA tool stimulates me to study more efficiently-This LA tool stimulates me to study more effectively

-This LA tool stimulates me to teach more efficiently-This LA tool stimulates me to teach more effectively

EFLA

•  isvalid&reliable,

•  addressesthelackofevaluationinstruments,

•  complementsevaluationonaninstitutionallevel,

•  addressestheneedforevidencethathasbeencalledfor,

•  iseasytousewithtemplates&interactivespreadsheet

DATAFor this LA tool it is clear what data is being collected

The Evaluation Framework for Learning AnalyticsE F L A

AWARENESS & REFLECTION

IMPACT

forL E A R N E R S

For this LA tool it is clear why the data is being collectedstrongly

disagreestrongly agree

1O

2O

3O

4O

5O

6O

7O

8O

9O

10O

This LA tool makes me awareof my current learning situation

This LA tool stimulates me to reflecton my past learning behaviour

This LA tool makes me forecast my possible future learning situation given my (un)changed behaviour

strongly disagree

strongly agree

1O

2O

3O

4O

5O

6O

7O

8O

9O

10O

stronglydisagree

strongly agree

1O

2O

3O

4O

5O

6O

7O

8O

9O

10O

strongly disagree

strongly agree

1O

2O

3O

4O

5O

6O

7O

8O

9O

10O

stronglydisagree

strongly agree

1O

2O

3O

4O

5O

6O

7O

8O

9O

10O

This LA tool stimulates me to adaptmy learning behaviour if necessary

stronglydisagree

strongly agree

1O

2O

3O

4O

5O

6O

7O

8O

9O

10O

This LA tool stimulates me to study more efficientlystrongly


1O

2O

3O

4O

5O

6O

7O

8O

9O

10O

This LA tool stimulates me to study more effectivelystrongly


1O

2O

3O

4O

5O

6O

7O

8O

9O

10O

DATAFor this LA tool it is clear what data is being collected

AWARENESS & REFLECTION

IMPACT

forT E A C H E R S

For this LA tool it is clear why the data is being collectedstrongly


1O

2O

3O

4O

5O

6O

7O

8O

9O

10O

This LA tool makes me awareof my students’ current learning situation

This LA tool stimulates me to reflecton my past teaching behaviour

This LA tool makes me forecast my students’ possible future learning situation given their (un)changed behaviour

stronglydisagree

strongly agree

1O

2O

3O

4O

5O

6O

7O

8O

9O

10O

stronglydisagree

strongly agree

1O

2O

3O

4O

5O

6O

7O

8O

9O

10O

stronglydisagree

strongly agree

1O

2O

3O

4O

5O

6O

7O

8O

9O

10O

stronglydisagree

strongly agree

1O

2O

3O

4O

5O

6O

7O

8O

9O

10O

This LA tool stimulates me to adaptmy teaching behaviour if necessary

stronglydisagree

strongly agree

1O

2O

3O

4O

5O

6O

7O

8O

9O

10O

This LA tool stimulates me to teach more efficientlystrongly


1O

2O

3O

4O

5O

6O

7O

8O

9O

10O

This LA tool stimulates me to teach more effectivelystrongly


1O

2O

3O

4O

5O

6O

7O

8O

9O

10O

EFLA scoring per stakeholder groupStep 1: Calculate the average value for each item based on the answers given for that item.

Step 2: Calculate the average value for each dimension based on the average of its items.

Step 3: Calculate the dimensional scores by rounding the results of ((x-1)/9)*100 where x is the average value of a dimension.

Step 4: Calculate the overall EFLA score by taking the average of the three dimensional scores.

http://www.laceproject.eu/evaluation-framework-for-la/

Outline

•  Whatislearninganalytics?•  Whereareweat?•  Howaredashboardsused?•  Howcanweevaluatelearninganalyticstools?•  HowaboutaninstitutionalLApolicy?•  Wheredowegofromhere?

PolicyFramework

http://sheilaproject.eu/

PolicyFramework

ACTION CHALLENGES POLICY

culture capabilities datamanagement

ethics&privacy culture evaluation

financial&humanresources

ethics&privacy methodology

infrastructure infrastructure policymanagement

internal&externalsupport management purpose

methodology methodology stakeholderengagement

purpose

stakeholderengagement

Outline


LASophisticationModel

Siemens,G.,Dawson,S.,&Lynch,G.(2014).ImprovingtheQualityandProductivityoftheHigherEducationSector–PolicyandStrategyforSystems-LevelDeploymentofLearningAnalytics.Canberra,Australia:OfficeofLearningandTeaching,AustralianGovernment.

ProblemofLAUptake

EducationalScience

ComputerScience

LearningAnalytics

MIDDLESPACE

Solution:UseLAasarealmiddlespace

EducationalScience

ComputerScience

LearningAnalytics

MIDDLESPACE

Learninganalytics…

… is not about the creation of intelligent or stupidsystems…… but about the creation of intelligent and successfulstudents…

…andthepromotionofeducation.

Ifeducationisviewedasavehicleforadvancingtheglobaleconomy,thenitmustbetheNorthStarthatguides societies to the next big thing, illuminatingnew ideas that solve pressing challenges andcreatingopportunitiestoshapeabetterfuture.

SamanthaAdamsBeckeretal.(2017)

Nowit’syourturn…

…tocreateyourowndashboard!

Thankyouforyourattention!

[email protected] [email protected]@m_a_s_c @IoanaJJEFLA:www.laceproject.eu/evaluation-framework-for-la/LAattheOU:www.ou.nl/en/welten-open-learning-analytics

Documents

Learning Analytics - Competen-SEAcompeten-sea.eu/wp-content/uploads/2018/02/... · Recommendations for designing dashboards 1. Define the goal of the dashboard and desired impact