Upload
mcheckley
View
53
Download
3
Embed Size (px)
Citation preview
Design method for the evaluation and quality assurance of an online learning
environment.
Marianne Checkley ESAI 2015
Background and Context
• Personalised online learning programme for young early school leavers aged 13-16 years.
• 52 students• Blended centre and at-home• 10 online teachers
• Maximising multiple data streams in examining the online educational experience
• Creating a narrative approach
Presentation Focus
Instruments developed to evaluate, benchmark, and provide quality-assurance processes for online learning environments
are primarily used at Third Level (Oncu & Cakir, 2011).
Research from iNacol designed to measure qualityin K12 online and blended learning environments divides strategies into outcomes and processes (Patricks et al, 2012).
Including proficiency, individual student growth, graduation rate, college and career readiness and closing the achievement gap.
For quality assurance of processes the following headings are assessed under comprehensive criteria: Content, Instructional Design, Student Assessment, Technology, Course Evaluation and Support.
Developing an evaluation framework for the educational experience on iScoil:
• Research suggests incorporating the data streams available from a virtual learning environment (Kennedy and Soifer, 2013) with a student centred qualitative approach to participation from all stakeholders in order to consider outcomes in a contextualized way (Niehaus, 2014).
• Deepwell (2007) recognises the capacity of Stake’s Countenance approach to organise large amounts of diverse data and as such is ideal for the analysis of online learning programmes.
Three interdependent phases Antecedent, Transaction, Outcome.
AntecedentConditions existing prior to instruction that may relate to
outcomes
TransactionThe process of
instruction
OutcomesThe effects of the
programme
What is the profile of iScoil students?
How do the learners engage with the online
platform?
What certification is achieved?
What is the theory underpinning curriculum
design?
What is the learning experience?
What are the progression routes?
What is the teaching experience?
How does student profile impact outcomes?
How does student profile impact engagement?
Are there unintended outcomes?
How is the curriculum delivered?
Is there potential for improvement
Table 1: Research Questions within a countenance framework
Figure 1: Community of Inquiry Model (Swan et al, 2009)
EDUCATIONAL EXPERIENCE
Supporting Discourse
COGNITIVE PRESENCE
Selecting Content
TEACHING PRESENCE
Setting Climate
SOCIAL PRESENCE
Features for Data Collection 1
Features for Data Collection 2
Figure 2: Cognitive Apprenticeship Model (Collins, Brown & Newman, 1989)
AntecedentConditions existing prior to instruction that may
relate to outcomes
TransactionThe process of
instruction
OutcomesThe effects of the
programme
Student Profile: Gender-Reason for Referral-Location of Learning.
Learner engagement. patterns on VLE
Certification achieved according to student
profile
Design of Curriculum. The Learning Experience. Progression Routes according to student
profile.
Learner-Teacher-Content Interaction.
The Teaching Experience
Learner Content Interaction.
Table 2: Features for Data Collection within a countenance framework
Research Question
• How do the students engage with the online platform?
• What is the learning experience?
• What is the teaching experience?
• How is the curriculum delivered?
• How does student profile impact on engagement and learning?
Feature
• Learner engagement patterns on VLE
• The teaching experience
• The learning experience
• Learner/Teacher/Content Interaction
• Learner/Content Interaction
Method
• VLE Log Analysis• Focus Group• Semi-Structured
Interviews• Authoring Tool Open
GLM• Outcomes Matrix
How the Framework works out: Transaction Phase
Figure 3: Corresponding Research Questions, Features and Methods across a countenance approach for the Transaction Phase
Data Collection …
• Document Analysis• Learning Designer• VLE Log Analysis
• Open GLM• Focus Groups• Semi Structured Interviews• Outcomes Matrix
Reflective use of Authoring Tools
A relevant example is a study where multiple design and authoring tools explore one lesson (Prieto et al., 2013). The study ultimately highlighted the complexity involved in the learning design process as different tools and processes were a better fit to different teachers depending on their pedagogical aims or institutional and technological contexts.
A number of computational tools that guide the thinking of education practitioners in pedagogical design have emerged from learning design research over the last ten years (Masterman, Walker & Bower, 2013).
While these design and authoring tools were developed to facilitate the development, adaptation and sharing of teachers’ pedagogical ideas, they are also, according to Laurillard (2012) useful as tools for reflection on practice.
AntecedentStudentProfile
Anxiety
Behavioural
School Refusal
Disaffection
Illness
0 2 4 6 8 10 12 14
FemaleMale
DisaffectionBehavioural
School RefusalAnxiety
Illness
0
2
4
6
8
10
12
Blended Male Blended Female At-Home Female At-Home Male
Gender and Reason for Referral
Gender, Location and Reason for Referral
AntecedentCurriculum Design
RWL4%
Produce18%
Practice42%
Inquire36%
Results from reflective use of the Learning Designer grounds the design in constructivist learning theory with an emphasis on an active model of learning.
However absence of discussion based activities limit the opportunity for interaction and social construction of knowledge.
Figure 5: Results of Learning Designer analysis of Communications Unit 2 Writing
Transaction Phase
Feedback Loops
Positive Affirmation
Personalised
Teaching Teams
Constructivism By Proxy
Modular Assessment
Relevant Content
Open GLM University of Vienna
Results of thematic analysis of variety of data
Outcomes
6 Modules
5 Modules
4 Modules
3 Modules
2 Modules
1 Modules
0 1 2 3 4 5 6 7
At-Home N=19 Blended N=12
Unknown
Second Level School
VEC Second Level School
Youthreach
Solas
Community Training Centres
0 1 2 3 4 5 6 7
Blended At-Home
Certification
Progression
Synchronous Social Presence
Project Based Learning
Student Online Collaboration
A Model for Prevention
Conclusions
The interdependence of sampling across phases used a parallel and simultaneous design (Cresswell & Plano-Clark, 2011) that permitted triangulation of data and ultimately strengthened the research process.
Research design supported a study where data were collected, analysed and interpreted in a way that provided an insight into an online learning journey from induction to exit and also identifying areas for improvement.
ReferencesDeepwell, F. (2007). Embedding Quality in e-Learning Implementation through Evaluation. Educational Technology & Society, 10 (2), 34-33.
Kennedy, S. & Soifer, D. (2013). Why blended learning can’t stand still: A commitment to constant innovation is needed to realise the potential of individualised learning. Lexington Institute.
Laurillard D. & Ljubojevic D. (2011). Evaluating learning designs through the formal representation of pedagogical patterns. In Investigations of E-Learning Patterns: Context Factors, Problems and Solutions. (eds J.W. Kohls & C.Kohls), pp. 86–105. IGI Global, Hershey, PA.
Masterman, L. (2013). The Challenge of Teacher’s Design Practice. In Beetham, H. and Sharpe, R. (Eds.), Rethinking Pedagogy for a Digital Age, (64-77). New York: Routledge.
Oncu, S., & Cakir, H. (2011). Research in online learning environments: Priorities and methodologies. Computers & Education, 57(1), 1098-1108. doi:10.1016/j.compedu.2010.12.009
Patrick, S., Edwards, D., Wicks, M. & Watson, J. (2012). Measuring Quality from Inputs to Outcomes: Creating Student Learning Performance Metrics and Quality Assurance for Online Schools. Vienna (VA): International Association for K-12 Online Learning.
Prieto, L., Dimitriadisa, Y., Craftb, B., Derntlc, M., Katsamanie, M., Laurillard, D., Masterman, E., Retalis, S. & Villasclaras, E. (2013). Learning design Rashomon II: exploring one lesson through multiple tools. Research in Learning Technology, 21.
Stake, R.E. (1967). The countenance of educational evaluation. Teachers College Record, 68 (7), 523-540.
Swan, K., Garrison, D.R., and Richardson, J.C. (2009). A constructivist approach to online learning: The Community of Inquiry framework. Information Technology in Higher Education: Progressive Learning Frameworks. Hershey, PA.