Embed Size (px)
Citation preview
Making Sense of Multimodal Learning Analytics (MMLA) Marcelo Worsley AssistantProfessor,LearningSciences&ComputerScience
- What is MMLA? - Why use MMLA? - How to use MMLA? - Examples of MMLA research - On-going challenges and future directions - Resources
Agenda
What is MMLA?
LearningAnaly9cs—asetofmul9-modalsensoryinputs,thatcanbeusedtopredict,understandandquan9fystudentlearning.
(WorsleyandBlikstein,2011)
What is MMLA?
Mul9modallearninganaly9cs(MMLA)(Blikstein&Worsley,inpress.;Blikstein,2013;Worsley,2012)sitsattheintersec9onofthreeideas:mul9modalteachingandlearning,mul9modaldata,andcomputer-supportedanalysis.Atitsessence,MMLAu9lizesandtriangulatesamongnon-tradi9onalaswellastradi9onalformsofdatainordertocharacterizeormodelstudentlearningincomplexlearningenvironments.However,aswedescribelater,thewaysthatresearchersu9lizemul9modaldatavarywidely.
(Worsley,Abrahamson,Blikstein,Grover,SchneiderandTissenbaum,2016)
What is MMLA?
What is MMLA?
What is MMLA?
VIDEO
Pose
Gaze
Gestures
ObjectManipula9on
FacialExpressions
Proxemics
SceneUnderstanding
Movement
Heartrate
ObjectTracking
AUDIO
SpeakerDiariza9on
Prosody
Spectrum
Emo9on/Affect
Intensity
SpeakingRate
Collabora9on/Turn-Taking
VoiceQuality
TEXT
Emo9on/Affect
Complexity
Cohesion
Seman9cs
Syntac9cs
Content
EYETRACKINGDATA
Fixa9ons
Fixa9onPaeern
PupilDila9on
Eyeloca9on
Aeen9on
HeadPose
DEPTHCAMERA
Gestures
BodyPosi9on
JointTracking
SMARTPHONES
Signg/Standing
Loca9on
NoiseLevel
LEAPMOTION
Gestures
ToolUsage
HandMovement
FingerMovement
EDAWATCH
Arousal
Cogni9veLoad
BodyTemperature
Gestures
Heartrate
Stress
EVENTLOGS
Ac9vity
OTHERMODALITIES
fMRI
ELECTROCARDIOLOGY
ELECTROENCEPHALOGY
ELECTROMYOGRAPHY
ELECTROGASTROGRAPHY
Why use MMLA? - Teachingandlearningaremul9modal- Studyandsupportcomplexlearning
environments- Seethehardtosee
- Informdesignofmul9modaltechnologies- Expandno9onsoflearningtonon-
tradi9onalmodali9es- Improveaccessibilityandinclusivity
- Triangulateacrossmodali9es
Why use MMLA? - Visualizing/Represen9nginforma9onfor
humaninference- Predic9onofindicators- Data-driveninterven9ons- Evalua9ngconjecture-basedlearning
designs
How to use MMLA?
How to use MMLA?
How to use MMLA? DATACAPTURESOFTWARE
OpenSocialSignalInterpreta9on
LabStreamingLayer
OpenPipeKit
OpenBroadcasterSohware
Mul9sense(AVRecorder)
iMo9onsAeen9onTool
How to use MMLA? DATACAPTURESOFTWARE
OpenSocialSignalInterpreta9on
LabStreamingLayer
OpenPipeKit
OpenBroadcasterSohware
Mul9sense(AVRecorder)
iMo9onsAeen9onTool
How to use MMLA? DATACAPTURESOFTWARE
OpenSocialSignalInterpreta9on
LabStreamingLayer
OpenPipeKit
OpenBroadcasterSohware
Mul9sense(AVRecorder)
iMo9onsAeen9onTool
How to use MMLA? DATACAPTURESOFTWARE
OpenSocialSignalInterpreta9on
LabStreamingLayer
OpenPipeKit
OpenBroadcasterSohware
Mul9sense(AVRecorder)
iMo9onsAeen9onTool
How to use MMLA? DATACAPTURESOFTWARE
OpenSocialSignalInterpreta9on
LabStreamingLayer
OpenPipeKit
OpenBroadcasterSohware
Mul9sense(AVRecorder)
iMo9onsAeen9onTool
How to use MMLA?
How to use MMLA?
Pre-processingisimportantfordatasynchroniza9on,accoun9ngforindividualdifferencesbetweenpar9cipants,andgegngdataintheappropriateformatfordataextrac9on.Ittendstovarybydatatype.
How to use MMLA?
VIDEOOpenFace(gaze)OpenFace(facerecogni9on)Emo9entFACETNoldusFaceReaderOpenCVAffec9vaMicrosohFaceAPIMicrosohEmo9onAPIIntrafaceMul9senseEularianMagnifica9onCodeiMo9ons
AUDIOHTKPraatOpenSmileOpenEarCovarepELANICSIDiarizerMatlabLIUMDiarizerCMUSphinxGoogleASRAT&TWatsonASRBingSpeechAPIAudacityEmovoice
TEXT
NaturalLanguageToolkit
Lightside
LIWC
Cohmetrix
ToneAnalyzer
StanfordParser
Wordnet
Sen9wordnet
Mallet
Word2Vec
EYETRACKINGDATA
PyGaze
Ogama
EyeTrackingR
TobiiSDK
SMISDK
iMo9onsAeen9onTool
PupilDila9on
DEPTHCAMERA
OpenNUI
KinectforWindows
QSRLib
Libfreenect
EDAWATCH
Ledalab
DATAEXTRACTIONSOFTWARE
OpenSocialSignalInterpreta9on
How to use MMLA?
How to use MMLA?
HumanCoders
Fusion Let’ssaythatyouwanttostudyengagementandhave
audio,video,bio-physiologicalandgesturedataavailableforanalysis.Howdoyouusethesetocomputea
measureofengagement?
How to use MMLA?
How to use MMLA?
Examples
XboxKinect- Audio(Talk)- Video(Pose/
Gaze)- Gestures(Hand
Movement)
RapidMiner- XMeans
Researcher(s)
XboxKinect- Audio(Talk)- Video(Pose/
Gaze)- Gestures(Hand
Movement)
RapidMiner- XMeans
Researcher(s)Custom
Challenges & Future Directions
Simplifying Data Capture & Analysis
Better Visualization/Inference Tools
Best Practices Around Data Fusion and Data Analysis Pipelines
Applications (especially as it relates to inclusive technology
and providing feedback to learners and teachers)
Resources Mul9modalLearningAnaly9csSpecialInterestGroup-hep://sigmla.orgCrossMMLAWorkshopsEC-TEL2017,LAK2018(tenta9ve)MMLAWorkshops@ICMI,LAK,ICLS(2012–Present)SOLARLASI–Mul9modalLearningAnaly9csTutorialWorkshopsJournalofLearningAnaly9csSpecialSec9ononMul9modalLearningAnaly9csCIRCLCyberlearningReport
Berland,M.,Baker,R.S.,&Blikstein,P.(2014).Educa9onalDataMiningandLearningAnaly9cs:Applica9onstoConstruc9onistResearch.Technology,KnowledgeandLearning,19(1–2),205–220.
Blikstein,P.,&Worsley,M.(2016).Mul9modalLearningAnaly9csandEduca9onDataMining :usingcomputa9onaltechnologiestomeasurecomplexlearningtasks,3(2),220–238.heps://doi.org/hep://dx.doi.org/10.18608/jla.2016.32.11
Fouse,A.S.(2011).ChronoViz :Asystemforsuppor9ngnaviga9onof9me-codeddata.Chi,1–6.heps://doi.org/10.1145/1979742.1979706
Grafsgaard,J.F.(2014).Mul9modalAnalysisandModelingofNonverbalBehaviorsDuringTutoring.InProceedingsofthe16thInterna=onalConferenceonMul=modalInterac=on(pp.404–408).NewYork,NY,USA:ACM.heps://doi.org/10.1145/2663204.2667611
Leong,C.W.,Chen,L.,Feng,G.,Lee,C.M.,&Mulholland,M.(2015).U9lizingdepthsensorsforanalyzingmul9modalpresenta9ons:Hardware,sohwareandtoolkits.ICMI2015-Proceedingsofthe2015ACMInterna=onalConferenceonMul=modalInterac=on,(3),547–556.heps://doi.org/10.1145/2818346.2830605
Luz,S.(2013).Automa9ciden9fica9onofexpertsandperformancepredic9oninthemul9modalmathdatacorpusthroughanalysisofspeechinterac9on.Proceedingsofthe15thACMonInterna=onalConferenceonMul=modalInterac=on-ICMI’13,575–582.heps://doi.org/10.1145/2522848.2533788
Morency,L.-P.,Oviae,S.,Scherer,S.,Weibel,N.,&Worsley,M.(2013).ICMI2013grandchallengeworkshoponmul9modallearninganaly9cs.Proceedingsofthe15thACMonInterna=onalConferenceonMul=modalInterac=on-ICMI’13,373–378.heps://doi.org/10.1145/2522848.2534669
Oviae,S.,&Cohen,A.(2013).Wrieenandmul9modalrepresenta9onsaspredictorsofexper9seandproblem-solvingsuccessinmathema9cs.…15thACMonInterna=onalConferenceonMul=modal…,1–8.Retrievedfromhep://dl.acm.org/cita9on.cfm?id=2533793
Scherer,S.,Worsley,M.,&Morency,L.-P.(2012).1stinterna9onalworkshoponmul9modallearninganaly9cs.InICMI(pp.609–610).
Schneider,B.,&Blikstein,P.(2015).UnravelingStudents’Interac9onAroundaTangibleInterfaceusingMul9modalLearningAnaly9cs.JournalofEduca=onalDataMining.
Spikol,D.(2017).UsingMul9modalLearningAnaly9cstoIden9fyAspectsofCollabora9oninProject-BasedLearningIntroduc9onPELARSsystemandcontext.Cscl,(June),263–270.heps://doi.org/10.22318/cscl2017.37
Thompson,K.(2013).Usingmicro-paeernsofspeechtopredictthecorrectnessofanswerstomathema9csproblems:anexerciseinmul9modallearninganaly9cs.…15thACMonInterna=onalConferenceonMul=modal….Retrievedfromhep://dl.acm.org/cita9on.cfm?id=2533792
Worsley,M.(2012).Mul9modalLearningAnaly9cs-Enablingthefutureoflearningthroughmul9modaldataanalysisandinterfaces.Interna=onalConferenceonMul=modalInterac=on,12–15.heps://doi.org/10.1145/2388676.2388755
Worsley,M.,Scherer,S.,Morency,L.P.,&Blikstein,P.(2016).ExploringBehaviorRepresenta9onforLearningAnaly9cs.Proceedingsofthe2015Interna=onalConferenceonMul=modalInterac=on(ICMI),251–258.
XavierOchoa,MarceloWorsley,KatherineChiluiza,&SaturninoLuz.(2014).MLA’14:ThirdMul9modalLearningAnaly9csWorkshopandGrandChallenges.Proceedingsofthe16thInterna=onalConferenceonMul=modalInterac=on,531–532.heps://doi.org/10.1145/2663204.2668318
39
MarceloWorsleyAssistantProfessorElectricalEngineeringandComputerScience&LearningSciencesmarcelo.worsley@northwestern.eduMarceloworsley.com9ilt.northwestern.edu
