Upload
bart-rienties
View
366
Download
0
Embed Size (px)
Citation preview
The power of learning analytics to measure learning gains: an OU,
Surrey and Oxford Brookes story
https://twitter.com/LearningGains
https://abclearninggains.com/
The results of the ABC project were made possible due to Simon Cross, Ceri Hitching, Ian Kinchin, Simon Lygo-Baker, Allison Littlejohn, Jekaterina Rogaten, Bart Rienties, Ian Scott, Rhona Sharpe, Steve Warburton, and Denise Whitelock. Please contract [email protected] if you want to know more about ABC learning gains
Why measure Learning Gains
Talk to your neighbour:
1. What do you think Learning Gains are?
2. Why should universities worry about Learning Gains?
3. Why should I/we need to think about Learning Gains?
Why measure Learning Gains
• Green paper UK (November 2015) states – learning gain (and student outcomes) is a key aspect to consider when recognizing teaching excellence and quality.
• Learning Gains are key elements of TEF, but how to measure?
• 13 Pilot projects have been awarded a total of 4m to advance our understand of learning gains in HE and develop valid and reliable assessment of learning gains.
• Our project is ABC mixed-method learning gains research and for quantitative part of the study we are looking at what affective, behavioural and cognitive learning gains students show.
Agenda for today
1. What are learning gains?
2. How are we working within ABC project to measure learning gains?
3. How are learning gains measured: a meta-analysis
4. Do students make learning gains, and what is the power of LA to predict learning gains?
5. Policy implications
1 What are learning gains?
In line with Cronbach & Furby, 1970 and Lord, 1956, we define Learning Gains as:
Growth or change in knowledge, skills, and abilities over time that can be linked to the desired learning outcomes or learning goals of the course
Rogaten, J., Rienties, B, Cross, S., Whitelock, D., Sharpe, R., Lygo-Baker, S., Littlejohn, A. (Submitted: 07-07-2016). Reviewing the concept of learning gains in higher education: an affective, behaviour and cognitive perspective. Educational Research Review. Impact factor: 3.860
2. How are we working within ABC project to measure learning gains?Affect
Behaviour
Cognition
Academic performance
VLE
Satisfaction
Quantitative
QualitativeAcademic
performance
VLE
Satisfaction
Interviews
Diary
Module design
Socio-Demographic
Structure of the ABC project
Open University Team• Strength• Learning analytics• Quantitative data analysis• Assessment• Online learning and learning design• Cross-sector thinking and working
• Responsibilities• Overall leading the project• Leading the Phase 1 of the project
(secondary data analysis)• Phase 1 ethics applications• Developing models of learning gains
Dr Bart RientiesDr Jekaterina Rogaten
Prof Denise Whitelock
Dr Simon CrossProf Allison Littlejohn
Open University1. Learning AnalyticsLargest VLE datasets of affect, behaviour and cognition of 200K students, with 2 large scale implementations of predictive modelling
2. Learning DesignExtensive modelling of learning designs (OULDI)
3. Student Experience on a ModuleExtensive datasets of students’ experiences with blended and online learning
Oxford Brooks University Team
Prof Rhona SharpeDr Ian Scott
StrengthExpertise in learners’ experienceMixed-methods approachMeasures of learning and engagementLearning analytics
ResponsibilitiesCo-leading Phase 2 of the projectIn depth-interviews and diary data collectionTesting validity of self report measures of learning gainsTriangulation
Oxford Brookes University1. Academic Performance Tracking Tool Presents staff with data from student information system, NSS and module evaluations in form of dashboards used for module and programme reviews.
2. Grade Point AverageProvides students with information about their progress. Provides staff with a more nuanced indicator of attainment.
3. Survey of Student EngagementDeveloped own scales for monitoring student engagement with Assessment Compact, Academic Advising and Graduate Attributes.
University of Surrey Team
ResponsibilitiesCo-leading Phase 2 of the projectTesting the validity of the self-reported measures of learning gains (+OB)Testing the validity of learning analytics modelling (+OU)Dissemination: website and Twitter
Prof Ian Kinchin Prof Steven Warburton Dr Simon Lygo-Baker
StrengthUniversity pedagogy and concept mapping Educational technologiesExpertise in innovative teaching approaches development
Ceri Hitchings
University of Surrey
• Business Intelligence - Progression Analysis project has built / is building:• Live data on students who are at risk, to act as an early warning indicator so that active support
measures can be put in place;• Visualisation of data via dashboards for academics to put in place appropriate actions;• Extension into WP & Outreach domain with specific data feeds, modelling, and a WP progression
dashboard.
• Institutional success: • Ability to makes links between what gains our students make in correlation with an institution which
has high levels of satisfaction and has enhanced learning and teaching across all Faculty.
3. How are learning gains measured: a meta-analysis
• The concept of learning gain is primarily used to examine the effect of any particular educational ‘intervention’• There is a gradual increase in
studies examining learning gains all across the world• All learning gains can be
classified into ABC53%
16% 21%
10% Behaviour-Cognitive Learning Gains
Affective-Cognitive Learning Gains
Affective-Behaviour-Cognitive Learning Gains
Cognitive Learning Gains
Rogaten, J., Rienties, B, Cross, S., Whitelock, D., Sharpe, R., Lygo-Baker, S., Littlejohn, A. (Submitted: 07-07-2016). Reviewing the concept of learning gains in higher education: an affective, behaviour and cognitive perspective. Educational Research Review. Impact factor: 3.860
A wide divergence in methods and reported effects
Rogaten, J., Rienties, B, Cross, S., Whitelock, D., Sharpe, R., Lygo-Baker, S., Littlejohn, A. (Submitted: 07-07-2016). Reviewing the concept of learning gains in higher education: an affective, behaviour and cognitive perspective. Educational Research Review. Impact factor: 3.860
Results will be shared when published
4. Do students make learning gains, and what is the power of LA to predict learning gains?
• Learning gains are usually measured using pre-post standardised testing • It is resource intensive and becomes even more so if one wants to estimate
learning gains across various disciplines and number of universities • Using assessment results for estimating learning gains has number of
advantages:• Assessment data readily available• Widely recognized as appropriate measure of learning • Relatively free from self-reported biases• Allows a direct comparison of research finding with the results of other studies
Rogaten, J., Rienties, B, Whitelock, D. (2016). Assessing learning gains, TEA Conference, Tallinn, Estonia
Three-level Growth Curve Model
Level 1
Level 2
Level 3
TMA1
Student1
TMA3 TMA1 TMA2 TMA3 TMA1 TMA2 TMA3TMA2
Student2 Student3
Module1 Module2
TMA1 TMA2 TMA3
Student4
TMA1 TMA2 TMA3
Student5
Module3
Rogaten, J., Rienties, B, Whitelock, D. (2016). Assessing learning gains, TEA Conference, Tallinn, Estonia
VLE VLEVLE
Study 1• Participants
• 11,909 Social Science students of whom 72% were females and 28% were males with average age of M = 30.6, SD = 9.9
• 5,791 Science students of whom 58.2% were females and 41.8% were males with average age of M = 29.8, SD = 9.6.
• Measures• Tutor Marked Assessments (TMA)• Socio demographics (gender, ethnicity,
prior educational qualification)• Across 111 modules
Study 1: Rogaten, J., Rienties, B, Whitelock, D. (2016). Assessing learning gains, TEA Conference, Tallinn, EstoniaStudy 2: Rogaten, J., Rienties, B., Whitelock, D. (Submitted: 17-10-2016). Multi-level growth modelling of learning gains: the impact of VLE engagement on learning Arts and Business. Paper submitted to LAK2017.
Study 2• Participants
• 2,875 Arts students of whom 64% were females and 36% were males with average age of M = 34, SD = 13.3
• 5,082 students of whom 54% were females and 46% were males with average age of M = 29.6, SD = 9.3
• Measures• Tutor Marked Assessments (TMA)• Socio demographics (gender, ethnicity, prior
educational qualification, work part-time or full-time, and disability)
• VLE engagement• Across 36 modules
Three level growth model
Study 1: Rogaten, J., Rienties, B, Whitelock, D. (2016). Assessing learning gains, TEA Conference, Tallinn, EstoniaStudy 2: Rogaten, J., Rienties, B., Whitelock, D. (Submitted: 17-10-2016). Multi-level growth modelling of learning gains: the impact of VLE engagement on learning Arts and Business. Paper submitted to LAK2017.
Results will be shared when published
The power of learning analytics to measure learning gains: an OU,
Surrey and Oxford Brookes story
https://twitter.com/LearningGains
https://abclearninggains.com/
The results of the ABC project were made possible due to Simon Cross, Ceri Hitching, Ian Kinchin, Simon Lygo-Baker, Allison Littlejohn, Jekaterina Rogaten, Bart Rienties, Ian Scott, Rhona Sharpe, Steve Warburton, and Denise Whitelock. Please contract [email protected] if you want to know more about ABC learning gains