Upload
laila-shoukry
View
34
Download
6
Tags:
Embed Size (px)
Citation preview
© author(s) of these slides including research results from the KOM research network and TU Darmstadt; otherwise it is specified at the respective slide
Multimedia Communications Lab 09.06.2015 | Serious Games – Serious Games Analytics| 9
Serious Games Analytics
Laila Shoukry, M.Sc.
KOM – Multimedia Communications Lab 2
Motivation and Definitions
Learning Analytics
Game Analytics
Serious Games Analytics
Learning Analytics in Serious Games
Modelling for LA in SG
Choosing Data
Capturing Data
Aggregating Data
Analysing Data
Deploying Results
Summary
Agenda
csloh.com/research/v-lab/
KOM – Multimedia Communications Lab 3
Pop Quiz!
Pressenza.com
KOM – Multimedia Communications Lab 4
Pop Quiz!
echtlustig.com
Wikihow.com
KOM – Multimedia Communications Lab 5
Assessment of Learning
http://assessment.uconn.edu/
KOM – Multimedia Communications Lab 6
Assessment for/in/of Learning?
syllabus.bos.nsw.edu.au
KOM – Multimedia Communications Lab 7
Assessment for/in/of Learning?
Open Learner Model (OLM) Presenting to the learner
an understandable visualization of his current knowledge state
Proven to improve learning outcomes
Personalized Learning Tailoring to specific needs
and characteristics of each learner
Learning Style, Strengths, Weaknesses, Pace, Special Needs
KOM – Multimedia Communications Lab 8
What is Learning Analytics (LA)
edtechreview.in/
“Learning Analytics is the measurement,
collection, analysis and reporting of data
about learners and their contexts, for
purposes of understanding and
optimising learning and the environments
in which it occurs.” George Siemens 2011
“Learning Analytics is about collecting traces that learners leave behind and using those traces to improve learning” Eric Duval
KOM – Multimedia Communications Lab 9
More accessible, affordable and functional
Technology
Devices able to collect, synthesize and
analyse massive amounts of data
New Types of Data Available
More insight into learning
Why Learning Analytics
mindedge.com
dailygenius.com
KOM – Multimedia Communications Lab 10
Learning Analytics (LA)
vs Educational Data Mining (EDM)
KOM – Multimedia Communications Lab 11
Learning Analytics Dimensions
(Greller & Drachsler, 2012)
KOM – Multimedia Communications Lab 12
Learning Analytics - Example
Einmaleins.tu-graz.at
KOM – Multimedia Communications Lab 13
Learning Analytics - Example
Einmaleins.tu-graz.at
KOM – Multimedia Communications Lab 14
Learning Analytics - Example
Einmaleins.tu-graz.at
KOM – Multimedia Communications Lab 15
Learning Analytics - Example
Einmaleins.tu-graz.at
KOM – Multimedia Communications Lab 16
Learning Analytics - Example
Einmaleins.tu-graz.at
KOM – Multimedia Communications Lab 17
Game Analytics
northeastern.edu
“All forms of business intelligence data, and any kind of method, in game development and research.” Anders Drachen 2012
Main Goals:
Monetization and Improving Gameplay
gamasutra.com
KOM – Multimedia Communications Lab 18
Game Analytics - Example
steampowered.com/status/ep2/ep2_stats.php
KOM – Multimedia Communications Lab 19
Game Analytics - Example
steampowered.com/status/ep2/ep2_stats.php
KOM – Multimedia Communications Lab 20
Game Analytics - Example
steampowered.com/status/ep2/ep2_stats.php
KOM – Multimedia Communications Lab 21
Game Analytics - Example
steampowered.com/status/ep2/ep2_stats.php
KOM – Multimedia Communications Lab 22
Game Analytics - Example
steampowered.com/status/ep2/ep2_stats.php
KOM – Multimedia Communications Lab 23
Goal of LA in Serious Games
openequalfree.org/gamification-versus-game-based-learning-in-the-classroom/10082
Why Learning Analytics for Serious
Games (Game Based Learning)
Evaluation of Serious Games
Justifying expense in learning contexts
Objective and cost-effective approach
Evaluation with Serious Games
Provide a big amount of gameplay data
Interactive and engaging nature Stealth
Assessment
Enable insight about learner attributes and
learning progress
KOM – Multimedia Communications Lab 24
Serious Games Analytics
Learning Analytics
“The measurement, collection, analysis and
reporting of data about learners and their contexts,
for purposes of understanding and optimising
learning and the environments in which it occurs.” George Siemens 2011
Game Analytics
“All forms of business intelligence data, and any
kind of method, in game development and
research.” Anders Drachen 2012
Serious Games Analytics
“Methods for learners' gameplay data collection,
analysis and visualization to be used in Serious
Games research.” Christian Loh 2014
KOM – Multimedia Communications Lab 25
Outline
KOM – Multimedia Communications Lab 26
Modelling for LA
Baalsrud et al. (2014)
KOM – Multimedia Communications Lab 27
Modelling for Learning Analytics in SG
Competence-Based Knowledge Space Theory (CbKST)
Requires learning domains to be modelled as a prerequisite competency
structure
Inferring competence states Learner Model
Flow Theory
Narrative Game-Based Learning Objects (NGLOB)
Additionally considers player type and narrative aspects
Triple vector: Narrative Context, Gaming Context, Learning Context
Evidence-Centered Design (ECD)
Competency Model, Evidence Model and Action Model
KOM – Multimedia Communications Lab 28
Competence-Based Knowledge Space Theory
(CbKST)
Knowledge Domain: set of skills relevant
for solving problems
Competence State: subset of skills that
a learner has available
Prerequisite Relation: capturing
prerequisites between sets of skills
Competence Structure: collection of
competence states corresponding to the
prerequisite relation
Leas-box.eu Skill Assignments :
associating to each problem those competence states that are sufficient
for solving it
associating to each learning object the skills that it teaches
prerequisite structure on the set of problems/learning objects
Css.uni-graz.at
KOM – Multimedia Communications Lab 29
Flow
Kiili et al. (2014)
Flow Theory , Csikszentmihalyi 1988, Zone of Proximal Development, Vygotsky
KOM – Multimedia Communications Lab 30
Authoring
Annotation of Situations
Narrative, Gaming and Learning Context
Appropriateness for Player Model
Run-Time
Decision: How does a story continue?
Narrative Game-Based Learning Objects
(NGLOB) - Revisited
Göbel et al. (2010)
KOM – Multimedia Communications Lab 31
Evidence-Centered Design (ECD)
KOM – Multimedia Communications Lab 32
Outline
KOM – Multimedia Communications Lab 33
Choosing Data for Learning Analytics in SG
Depends on learning goals, setting, tasks, game
genre, mechanic and platform
• Intensive vs. Extensive Data
• Extensive Data: for Higher Quantity
• Intensive Data: for Higher Quality
• Single-Player vs. Multiplayer
• Multiplayer:
• additional social component
• Data fed into social network analysis to identify aspects
of collaborative learning
• Generic vs. Game-Specific Traces
• Generic:
• Identify strengths and weaknesses of learning games
• Compare different learning games
• Game-Specific:
• Designing games „with analytics in mind“
• More tailored to invidiual games
StoryPlay Learning Analytics Tool
KOM – Multimedia Communications Lab 34
Serious Games Analytics
Serrano-Leguana et al. 2014
Measures to be derived:
Gaming:
general in-game performance, in-game
learning, in-game strategies, player type
Learning:
general traits and abilities of the learner,
general knowledge, situation-specific state,
learning behaviors, learning outcomes
KOM – Multimedia Communications Lab 35
Outline
KOM – Multimedia Communications Lab 36
Capturing Data for Learning Analytics in SG
Depends on data modalities and interactions
• Activity logs
• Widely employed
• Records interaction data in form of log files
• Multimodal Learning Analytics
• Includes biometric data and other multimodal
data for assessing motivation, fun and
collaboration aspects in learning settings
• Introduces its own challenges for aligning data
• Mobile and Ubiquitous Learning Analytics
• Data of mobile learners, suitable for mobile
games
• Interaction with mobile devices
• Considering contextual information
Noldus.com/facereader
KOM – Multimedia Communications Lab 37
Privacy Concerns
Collectiveevolution.com
Nutzungsbedingungen für E-Learnin an der HWR Berlin auf der Lernplattform Moodle: Art der gespeicherten Daten Bei der Nutzung von Moodle werden Ihre Beiträge und Aktivitäten in Protokolldateien des Webservers sowie der Moodle-Software gespeichert, soweit das für die individualisierten Funktionalitäten in Moodle erforderlich ist. Weder Kursverantwortliche (z.B. in der Rolle „Lehrende“), noch andere Kursteilnehmende (z.B. in der Rolle „Studierende“) haben Zugriff auf diese Nutzungsdaten. Kursverantwortliche (in der Rolle „Lehrende“) haben Zugriff auf sogenannte Aktivitätsübersichten zu Zwecken der Lehrvermittlung, der Lehrorganisation und der Lehrerfolgskontrolle im betreffenden Kurs. Dargestellt werden hier persönliche Beiträge zu Aktivitäten wie Foren, Wikis, Blogs oder Aufgaben.
KOM – Multimedia Communications Lab 38
Capturing Data – Example Code
Sample variable traces (Gleaner format)
KOM – Multimedia Communications Lab 39
Capturing Data – Example Code
Logging using Google Analytics
KOM – Multimedia Communications Lab 40
Outline
KOM – Multimedia Communications Lab 41
Aggregating Data for Learning Analytics in SG
Depends on data sources and sample size
• Extensive Data Aggregation accross
Users
• Log data joined into central database after
preprocessing using session identifiers
• Log files generated on all machines should
use same data format
• Need for standardized xml formats
• „Aggregation Model“: using semantic rules
to map game actions or states to meaningful
expressions under which similar events are
grouped
• Intensive Data Aggregation accross
Modalities
• Multimodal Data Synchronization needed for
observing behavior accross MM data
channels
• Some tools exist: Replayer, ChronoVis
ChronoViz.com
KOM – Multimedia Communications Lab 42
Outline
KOM – Multimedia Communications Lab 43
Analyzing Data for Learning Analytics in SG
Depends on learning context and application
• By instructor
• This step is not done by the system but instructor intervenes
according to visualized statistics
• Automatic Analysis
• For intelligent tutoring systems and adaptive Serious Games
• Measures to be derived:
• Gaming: general in-game performance, in-game learning, in-game
strategies, player type
• Learning: general traits and abilities of the learner, general knowledge,
situation-specific state, learning behaviors, learning outcomes
• Rules governing the interpretation of in-game sources of evidence
to infer competencies
• Algorithms applied during learning sessions to update competency
models
• Data Mining and Machine Learning approaches can be used for
identifying solution strategies, error patterns and player goals
onlinelearninginsights.wordpress.com
KOM – Multimedia Communications Lab 44
Game Verbs
Gamedesign.glasslabgames.com
KOM – Multimedia Communications Lab 45
Outline
KOM – Multimedia Communications Lab 46
Deploying Results for Learning Analytics in SG
Depends on learning context and
application
• Visualization
• visualizations of narrative structure,
player model and skill tree
• graphs, Hasse Diagrams, Heat Maps
• for games, a special need for real-time
operation, extensibility and
interoperability
• Adaptation
• macro-adaptivity: system responds by
choosing the appropriate next learning
object or narrative event
• micro-adaptivity: adjusting aspects
within a learning task like task diffculty or
feedback type
KOM – Multimedia Communications Lab 47
Visualization - Dashboards
Teachtown.com
KOM – Multimedia Communications Lab 48
Visualization - Dashboards
StoryPlay Dashboard
KOM – Multimedia Communications Lab 49
Visualization - Dashboards
GLEANER Dashboard
KOM – Multimedia Communications Lab 50
Visualization - Dashboards
GLEANER Dashboard
KOM – Multimedia Communications Lab 51
Visualization - Dashboards
GLEANER Dashboard
KOM – Multimedia Communications Lab 52
Visualization - Dashboards
EngAGe Dashboard
KOM – Multimedia Communications Lab 53
Popular Analytics Tools
Piwik Google Analytics
OpenSim Analytics for Virtual Worlds
KOM – Multimedia Communications Lab 54
Projects (some ongoing)
Lemo-projekt.de
KOM – Multimedia Communications Lab 55
Summary
Laila Shoukry, Stefan Göbel, Ralf Steinmetz:
Learning Analytics and Serious Games: Trends and Considerations. In: SeriousGames '14 Proceedings of the ACM International Workshop on Serious Games,
p. 21-26, ACM MM’14, November 2014. ISBN 978-1-4503-3121-0. http://dl.acm.org/citation.cfm?id=2656729.
KOM – Multimedia Communications Lab 56
Questions & Contact
KOM – Multimedia Communications Lab 57
References
• N. R. Aljohani and H. C. Davis. Learning analytics in mobile and ubiquitous learning environments. In 11th • World Conference on Mobile and Contextual Learning: mLearn 2012, Helsinki, Finland, 2012.
• R. S. J. D. Baker and K. Yacef. The State of Educational Data Mining in 2009 : A Review and Future Visions. 1(1):3-16, 2009. • P. Blikstein. Multimodal learning analytics. Proceedings of the Third International Conference on • Learning Analytics and Knowledge - LAK '13, page 102, 2013. • S. Bull, Y. C. Y. Cui, a.T. McEvoy, E. Reid, and W. Y. W. Yang. Roles for mobile learner models. The 2nd IEEE International
Workshop on Wireless and Mobile Technologies in Education, 2004. Proceedings., pages 124-128, 2004. • S. Bull and J. Kay. Open learner models. In Advances in Intelligent Tutoring Systems, pages 301-322. Springer, 2010. • G. K. Chung and D. S. Kerr. A Primer on Data Logging to Support Extraction of Meaningful • Information from Educational Games: An Example from Save Patch. CRESST Report 814. National • Center for Research on Evaluation, Standards, and Student Testing (CRESST), page 27, 2012. • A. Cooper. Learning Analytics Interoperability – The Big Picture In Brief. Learning Analytics Community Exchange, 2014. • E. Duval. Attention Please!: Learning Analytics for Visualization and Recommendation. LAK '11, pages 9-17, New York, NY, USA,
2011. ACM. • A. Dyckho and D. Zielke. Design and Implementation of a Learning Analytics Toolkit for Teachers. Journal of . . . , 15:58-76, 2012. • G. Dyke, K. Lund, and J.-J. Girardot. Tatiana: An Environment to Support the CSCL Analysis Process. In Proceedings of the 9th
International Conference on Computer Supported Collaborative Learning – Volume 1, CSCL'09, pages 58{67. International Society of the Learning Sciences, 2009.
• B. M. Eun Young Ha Jonathan Rowe and J. Lester. Recognizing Player Goals in Open-Ended Digital Games with Markov Logic Networks. Plan, Activity and Intent Recognition: Theory and Practice, 2014.
• J.-C. Falmagne, D. Albert, C. Doble, D. Eppstein, and X. Hu. Knowledge Spaces: Applications in Education. Springer Science & Business, 2013.
• I. Garcia, A. Duran, and M. Castro. Comparing the eectiveness of evaluating practical capabilities through hands-on on-line exercises versus conventional methods. In Frontiers in Education Conference, 2008.
• FIE 2008. 38th Annual, pages F4H{18{F4H{22, Oct • 2008. • S. Goebel, M. Gutjahr, and S. Hardy. Evaluation of Serious Games. Serious Games and Virtual Worlds in Education, Professional
Development, and Healthcare, page 105, 2013. • Kiili, K., Lainema, T., de Freitas, S., & Arnab, S. (2014). Flow framework for analyzing the quality of educational games.
Entertainment Computing, 5(4), 367-377.
KOM – Multimedia Communications Lab 58
References
• S. Goebel, V. Wendel, C. Ritter, and R. Steinmetz. Personalized, adaptive digital educational games using narrative game-based learning objects. pages 438-445. Springer, 2010.
• S. L. . B. H. Jan Plass. Assessment mechanics: design of learning activities that produce meaningful data.New York University, CREATE Lab, Apr. 2013.
• K. E. B. E. W. Joseph Grafsgaard Joseph Wiggins and J. Lester. Predicting Learning and Aect from Multimodal Data Streams in Task-Oriented Tutorial Dialogue. In Seventh International Conference on Educational Data Mining, 2014.
• D. S. Kerr and G. K. W. K. Chung. Using Cluster Analysis to Extend Usability Testing to Instructional Content. CRESST Report 816. National Center for Research on Evaluation, Standards, and Student Testing (CRESST), May 2012.
• M. D. Kickmeier-Rust and D. Albert. An Alien's Guide to Multi-Adaptive Educational Computer Games. Informing Science, 2012. • M. D. Kickmeier-Rust and D. Albert. Learning analytics to support the use of virtual worlds in the classroom. In Human-Computer
Interaction and Knowledge Discovery in Complex, Unstructured, Big Data, pages 358{365. Springer, 2013. • K. R. Koedinger, R. Baker, K. Cunningham, A. Skogsholm, B. Leber, and J. Stamper. A data repository for the EDM community: The
PSLC DataShop. Handbook of educational data mining, pages 43-55, 2010. • J. Konert, K. Richter, F. Mehm, S. G•obel, R. Bruder, and R. Steinmetz. Pedale - a peer education diagnostic and learning
environment. Journal of Educational Technology & Society, 2012. • K. Korossy. Modeling knowledge as competence and performance. Knowledge spaces: Theories, empirical research, and
applications, pages 103{132, 1999. • J. Lester, B. Mott, J. Rowe, and J. Sabourin. {learner modeling to predict real-time aect in serious games. Design Recommendations
for Intelligent Tutoring Systems, page 201, 2013. • S. M. Letourneau. Intensive data at CREATE -Research approaches & future directions. In Simulations and Games Learning
Analytics and Data Mining Workshop, 2013. • R. Levy. Dynamic Bayesian Network Modeling of Game Based Diagnostic Assessments. CRESST Report, 2014. • C. Martin, O. Galibert, M. Michel, F. Mougin, and V. Stanford. NIST Smart Flow System User's guide. • M. Minovic and M. Milovanovic. Real-time learning analytics in educational games. pages 245{251. ACM, • 2013. • R. J. Mislevy, R. G. Almond, and J. F. Lukas. A brief introduction to evidence-centered design. cse report 632. US Department of
Education, 2004. • A. Mitrovic and B. Martin. Evaluating the eect of open student models on self-assessment. International Journal of Articial
Intelligence in Education, 17(2):121-144, 2007. • A. Morrison, P. Tennent, and M. Chalmers. Coordinated visualisation of video and system log data. In Coordinated and Multiple
Views in Exploratory Visualization, 2006. Proceedings. International Conference on, pages 91-102. IEEE,2006.
KOM – Multimedia Communications Lab 59
References
• K. Nurmela, E. Lehtinen, and T. Palonen. Evaluating CSCL log les by social network analysis. In Proceedings of the 1999 conference on Computer support for collaborative learning, page 54. International Society of the Learning Sciences, 1999.
• H. Ogata, M. Li, B. Hou, N. Uosaki, M. M. El-Bishouty, and Y. Yano. SCROLL: Supporting to share and reuse ubiquitous learning log in the context of language learning. Research & Practice in Technology Enhanced Learning, 6(2), 2011.
• S. Oviatt, A. Cohen, and N. Weibel. Multimodal learning analytics: Description of math data corpus for icmi grand challenge workshop. In Proceedings of the 15th ACM on International conference on multimodal interaction, pages 563{568. ACM, 2013.
• J. L. Plass, B. D. Homer, C. K. Kinzer, Y. K. Chang, J. Frye, W. Kaczetow, K. Isbister, and K. Perlin. Metrics in Simulations and Games for Learning. In Game Analytics, pages 697{729. Springer, 2013.
• M. Prensky. Computer games and learning: Digital game-based learning. Handbook of computer game studies, 18:97-122, 2005.
• C. Reuter, F. Mehm, S. G•obel, and R. Steinmetz. Evaluation of Adaptive Serious Games using Playtraces and Aggregated Play Data. Proceedings of the 7th European Conference on Game Based Learning (ECGBL) 2013, (October):504{511, 2013.
• A. Serrano, E. J. Marchiori, A. del Blanco, J. Torrente, and B. Fernandez-Manjon. A framework to improve evaluation in educational games. In Global Engineering Education Conference (EDUCON), 2012 IEEE, pages 1-8. IEEE, 2012.
• A. Serrano-Laguna, J. Torrente, P. Moreno-Ger, and B. Fernandez-Manjon. Application of learning analytics in educational videogames. Entertainment Computing, 2014.
• V. J. Shute, M. Ventura, M. Bauer, and D. Zapata-Rivera. Melding the power of serious games and embedded assessment to monitor and foster learning. Serious games: Mechanisms and eects,
• pages 295-321, 2009. • G. Siemens and R. S. J. Baker. Learning Analytics and Educational Data Mining : Towards Communication and Collaboration.
2010. • K. K. Tatsuoka. Architecture of knowledge structures and cognitive diagnosis: A statistical pattern recognition and classication
approach. Cognitively diagnostic assessment, pages 327-359, 1995. • T. P. Vendlinski, G. C. Delacruz, R. E. Buschang, G. Chung, and E. L. Baker. Developing high-quality assessments that align
with instructional video games (CRESST Report 774). . National Center for Research on Evaluation, Standards, and Student Testing, 2010.
• A. V. Wallop. Althea Vail Wallop CS 229 Final Project Report Learning to Predict Disengagement in Educational iPad Application. pages 1-5, 2010.
• J.-D. Zapata-Rivera and J. E. Greer. Interacting with inspectable bayesian student models. International Journal of Articial Intelligence in Education, 14(2):127-163, 2004.
KOM – Multimedia Communications Lab 60
References
• Baalsrud Hauge, J., Berta, R., Fiucci, G., Fernandez Manjon, B., Padrón-Nápoles, C., Westra, W., & Nadolski, R. (2014, July). Implications of learning analytics for serious game design. In Advanced Learning Technologies (ICALT), 2014 IEEE 14th International Conference on (pp. 230-232). IEEE.