Click here to load reader

Designing and evaluating incidental learning

  • View
    366

  • Download
    3

Embed Size (px)

Text of Designing and evaluating incidental learning

Designing and evaluating incidental learning

Designing and evaluating incidental learningAndrew Brasher, Ann Jones, Agnes Kukulska-Hulme, Mark Gaved, Eileen Scanlon, Lucy NorrisIET, The Open University

www.maseltov.eu

www.maseltov.euAbstractOver the past three years we have been developing a framework intended to facilitate both the occurrence and evaluation of incidental learning. This work has been part of the MASELTOV project, supported by the European Commission, eInclusion programme FP7-ICT-7. The project intends to exploit the potential of mobile services for promoting integration and cultural diversity in Europe, and is focusing on support for immigrants with particular needs e.g. those who have not learned foreign languages, and who have a cultural background that contrasts with that of their host country.The goal of the Incidental Learning Framework is to facilitate the creation of technology rich learning opportunities which emanate from incidental learning i.e. learning that is spontaneous and unplanned. As a design tool its use should encourage links and triggers to structured and reflective learning to back up and deepen learning that happens incidentally. The framework is a descriptive mechanism that enhances analysis and evaluation of incidental learning, and a generative tool to support discussions around software system design. It facilitates the communication of learning design ideas both visually and textually.Initial work was presented at the CALRG conference in 2012, and the framework has been under development since that time (Brasher et al., 2012, Gaved et al., 2013, Kukulska-Hulme et al., in press). Development of the framework has included expert evaluation, and changes in response to data acquired from field trials of a mobile app intended to support language and cultural learning. In this paper we describe the development, use and evaluation of the framework to date, and reflect on lessons learnt.References Brasher, Andrew; Dunwell, Ian; Akiki, Oula and Gaved, Mark (2012). MASELTOV Deliverable D7.1.1: Incidental Learning Framework. MASELTOV Consortium, Graz, Austria.Gaved, Mark; Kukulska-Hulme, Agnes; Scanlon, Eileen; Jones, Ann; Jones, Janet; Dunwell, Ian and Lameras, Petros (2013). MASELTOV Deliverable Report 7.1.2: Incidental Learning Framework. MASELTOV Consortium, Graz, Austria.Kukulska-Hulme, A., Gaved, M., Paletta, L., Scanlon, E., Jones, A., & Brasher, A. (in press). Mobile Incidental Learning to Support the Inclusion of Recent Immigrants. Journal of Technologies in Education.

1

Maseltov projectMobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services www.maseltov.eu

Joanneum Research, AustriaCURE - Centre for Usability Research & Engineering, AustriaUniversity of Applied Sciences, FH Joanneum, AustriaAthens Information Technology, Greecebusuu.com -, Spain

Universitat Oberta de Catalunya, SpainOpen University, UKCoventry University, UK Czech Technical University, Czech RepublicTelecom Italia SpA, ItalyFluidtime Data Service GmbH, AustriaFundacian Desarrollo Sostenido, SpainVerein Danaida, AustriaMigrants Resource Centre, UKPearson Publishing, UK

www.maseltov.eu2(Background)

ContentsOverview of the Incidental Learning FrameworkEvaluation & use Workshop at plenary meeting expert researchers Iliinskys workChallengesUse in evaluation: analysis of MK trial dataConclusions

www.maseltov.eu

3

Research questions

Research question driving the development of the Incidental Learning Framework

www.maseltov.euGoals of the Incidental Learning Framework to facilitate the creation of technology rich learning opportunities which emanate from incidental learning i.e. learning that is spontaneous and unplanned; to encourage links and triggers to structured and reflective learning to back up and deepen learning that happens incidentally;to enhance analysis and evaluation of incidental learning; to support discussions around software system design;to facilitate the communication of learning design ideas both visually and textually.

www.maseltov.euThe goal of the Incidental Learning Framework is to facilitate the creation of technology rich learning opportunities which emanate from incidental learning i.e. learning that is spontaneous and unplanned. As a design tool its use should encourage links and triggers to structured and reflective learning to back up and deepen learning that happens incidentally. The framework is a descriptive mechanism that enhances analysis and evaluation of incidental learning, and a generative tool to support discussions around software system design. It facilitates the communication of learning design ideas both visually and textually.

5

Review of frameworksDesign frameworksto support the planning and instantiation of new learning events and situationsAnalytical frameworks for analysing, understanding and evaluating learning eventsLearner models to describe the state of the learner so that a system can react appropriately

www.maseltov.euReview of frameworksDesign frameworksAnalytical frameworksLearner models

Design frameworkAnalytical framework

www.maseltov.euExamples of frameworksAnalyticalTheory of Learning for the Mobile Age (Sharples et al., 2007)A Pedagogical Framework for Mobile Learning (Park, 2011)

DesignEcology of Resources design framework (Luckin, 2010)Four dimensional framework (de Freitas et al., 2010)Language learning defined by time and place(Kukulska-Hulme, 2012)Learner modelAdvances in learner & skill modeling in intelligent learning environments (Desmarais and Baker, 2011)

www.maseltov.euKolbs learning cycle

www.maseltov.euILF version 1

OutcomesSocialTime

TasksToolsPlaceLearners journey

Time

Incident: understanding notice board

Language tool

Start

TextLens

Task: finding station

Task: asking directions

Reflect/plan

Structured learning

Language Tool/Social

Task: navigating

Navigation tool

Language ToolLanguage Tool/Social

www.maseltov.eu

10

Example, stage 1

www.maseltov.euExample, stage 2

www.maseltov.euEvaluationWorkshop: adding detail to scenarios Expert focus group

www.maseltov.eu13

Expert focus group: (Professor Patrick McAndrew, Dr. Canan Blake, Dr. Elizabeth Fitzgerald)

Engagement with software developers and others suggested that the framework generates helpful discussions about the learner journeys which immigrants might make The extent to which working with the framework generates a set of guidelines for software programmers to implement is not straightforward. To some extent the framework is better understood as it a tool for generating discussion to aid the planning of learner journeys and better at exposing misunderstandings and different interpretations of the same problem space through different domain experts views (similar to Howes concept of dialogic resolution).

EvaluationThe framework generates helpful discussions about the learner journeys which immigrants might makeExposes misunderstandings and different interpretations of the same problem space through different domain experts views The current representation was sufficient to engage participants and encourage discussion.

(Gaved et al., 2013)

www.maseltov.eu14Engagement with software developers and others suggested that the framework generates helpful discussions about the learner journeys which immigrants might make The extent to which working with the framework generates a set of guidelines for software programmers to implement is not straightforward. To some extent the framework is better understood as it a tool for generating discussion to aid the planning of learner journeys and better at exposing misunderstandings and different interpretations of the same problem space through different domain experts views (similar to Howes concept of dialogic resolution).

EvaluationOrdering of dimensions was questionedOuter dimension of learners journey was debated: - too general? - how to show e.g. social inclusion in to a new community

(Gaved et al., 2013)

www.maseltov.eu15Engagement with software developers and others suggested that the framework generates helpful discussions about the learner journeys which immigrants might make The extent to which working with the framework generates a set of guidelines for software programmers to implement is not straightforward. To some extent the framework is better understood as it a tool for generating discussion to aid the planning of learner journeys and better at exposing misunderstandings and different interpretations of the same problem space through different domain experts views (similar to Howes concept of dialogic resolution).

Extension ideas (1)

Visualisation of specific learner journeys, showing opportunities for planned and incidental language learningGaved et al. (2012)

www.maseltov.euExtension ideas (2)

Visualisation of the range of affordances that may be associated with particular places that a MASELTOV service user is likely to encounter during their daily activities

www.maseltov.euExtension ideas (3)

Visualisation of the range of dimensions associated with motivation and concerns that might be felt by the service user while engaging with mobile situated language learning

www.maseltov.eu

18

Choosing visual properties

Iliinsky, N. (2013)

www.maseltov.euPlaceNot just a location - also contextual information. A Place is a geographical and historical context for Names and Locations (Gillies, 2011).a Place has a name and a location, and each may be applicable for a particular time period.

Andrews workplace currently has Name: Jennie Lee Building Location: map reference SP 88626 37058

www.maseltov.euFor example, the place that may be identified as Andrews workplace currently has the name the Open University and the location map reference SP 88626 37058.

20

Learning outcomes

Categorical

Related

www.maseltov.euKnowledge and skillsCognitive21

Linear view

PlaceTaskTools

OutcomesSocialTime

Evening

Language learning

Practise work related vocabulary

Understand work related vocabularySleep/Break

19.0023.0007.00

Language learning

Practise past tense

Communicate successfully in work related situationBreakfast08.00

www.maseltov.euVary thickness of layers depending on amount of activity at particular places or times.22

Data gatheringPre trial questionnaire:paperPost trial interviews: face-to-face

E.g. Lesson access duration: 239s Synchronous & located with MApp use Event data from the MAppUsage of other appsGeo contextual data

Asynchronous & not located with MApp use Weekly survey: online

www.maseltov.euBot quantitative and qualitative data was collected during the trial, using a variety of methods. Some of the methods of collection enabled data to be collected synchronously (and colocated) with participants use of the MApp, whilst other methods were asynchronous (and not colocated) as illustrated in the slide. The weekly survey is in-between it may have been filled in via the phone as an participant was in the middle of using the Mapp for some other purpose, or it might have been filled in via a PC independently of use of the MApp by the participant.

The combination of these methods should allow us to get a picture of participants behaviour during the trial (e.g. where and when they used different MApp services), and their reflections on that behaviour, both at the time and place of their use and afterwards.

The event data is both quantitative and qualitative MApp as it is used. E.g. forum posts (qualitative), textlens pictures (qualitative??) lesson access duration (quantitative). 23

Participants use of Mapp services

I have really liked the English course, it is very practical for the times when you have to use English for different occasions..it has helped me a lot The second is the translation tool because now I take a photo of everything I don't understand and at that moment I know what it says ... It has helped me a lot. I have learned a lot more English language

I have learned to develop myself in different aspects of speaking English with the help of the course

www.maseltov.euSpeech bubbles show Lucys translations of participants answers to What have you learnt this week? question in the weekly questionnaire. The usage data is real, but it is the maximum usage time per service in each day. So if someone used the language learning for 30 minutes, 27 minutes and 25 minutes during one day, only the 30 minute value would be shown.

CF. Maria example.

Numbers related to FPIs from AIT dataForum stats, No of posts by Mila etc. Recommendations delivered in English24

Use in analysisExample language lesson data keyvaluetimestamplessontransport22/01/2015 13:22publicationmas_trav22/01/2015 13:22score90.9090881322/01/2015 13:22lessonplan_journey24/01/2015 18:41publicationmas_trav24/01/2015 18:41score024/01/2015 18:41lessonplan_journey24/01/2015 18:41publicationmas_trav24/01/2015 18:41score024/01/2015 18:41lessontransport_chaos24/01/2015 20:07publicationmas_trav24/01/2015 20:07score10024/01/2015 20:07lessonbody_and_health24/01/2015 20:59publicationmas_health24/01/2015 20:59score85.714286824/01/2015 20:59

www.maseltov.euhttps://www.google.com/maps/d/edit?mid=zA_x5OA972vE.k86arSPhEhkA

25

ConclusionsDevelopment of ILF as a design tool Work to do is guided by literature and experimental findingsDevelopment of ILF as an analytical toolData available complex and multi-faceted;Initial experiments started; Transfer of design view not straightforward?Learner models?;Suggestions welcome! Levels of abstraction?Level of detail?

www.maseltov.euDevelopment of the framework has included expert evaluation, and changes in response to data acquired from field trials of a mobile app intended to support language and cultural learning. In this paper we describe the development, use and evaluation of the framework to date, and reflect on lessons learnt.

26

To doInclude stuff from conclusions slide from 2012 Calrg presentationfeedback from eval D7.12.Other stuff.

Issues learning outcomes can be shown at gps locations, but need to show learning journey To point to new developments show new version

www.maseltov.euExtensionsD7.1.2 i.e.- SCAMP- Learning journey

www.maseltov.euReferencesde Freitas, S., Rebolledo-Mendez, G., Liarokapis, F., Magoulas, G. & Poulovassilis, A. 2010. Learning as immersive experiences: Using the four-dimensional framework for designing and evaluating immersive learning experiences in a virtual world. British Journal of Educational Technology, 41, 69-85.Desmarais, M. C. & Baker, R. S. J. D. 2011. A review of recent advances in learner and skill modeling in intelligent learning environments. User Modeling and User-Adapted Interaction, 22, 9-38.Kukulska-Hulme, A. 2012. Language learning defined by time and place: A framework for next generation designs. In: DAZ-VERA, J. E. (ed.) Left to My Own Devices: Learner Autonomy and Mobile Assisted Language Learning.Iliinsky, N. (2013). Choosing visual properties for successful visualizations. Retrieved 24/4/2014, from http://bit.ly/successfulvisLuckin, R. 2010. Re-Designing Learning Contexts: Technology-Rich, Learner-Centred Ecologies (Foundations and Futures of Education). 208.Park, Y. 2011. A Pedagogical Framework for Mobile Learning: Categorizing Educational Applications of Mobile Technologies into Four Types. International Review of Research in Open and Distance Learning, 12, 78-102Sharples, M., Taylor, J. & Vavoula, G. 2007. A Theory of Learning for the Mobile Age. In: Andrew, R. & Haythornwaite, C. (eds.) The Sage Handbook of Elearning Research. London: Sage.

www.maseltov.eu

29