Embed Size (px)
Citation preview
So, just how much do your students think they know?Using online formative assessment to enhance student learning
Dr Arlëne G. HunterCentre for Open Learning in Maths, Science, Computing and Technology (COLMSCT), The Open University in Ireland
Session overview
• definition and role of formative assessment• overview of OU formative assessment project• initial findings and implications for future use
• discussion threads:– formative versus summative assessment– impact of technology on learning– importance of student self-esteem and motivation to learn
Why ‘formative’ assessment?
• What is ‘formative assessment’?– one line descriptor for students, staff, department etc…– what isn’t formative assessment?
• How is it used?– example(s) from your course/ own learning experience
• Why is it used?– the intended benefits for each of the stakeholders involved
Formative assessment issuesCommon concerns and questions:• students won’t engage with it• they won’t know how to use it to help• it may demotivate some learners• it’s a lot of effort for no return
(from the tutor and student perspective)• students are already assessed too much• apart from the grades, how does it differ to summative
assessment• how does it differ to self-assessment
Should formative assessment be ‘numerically graded’ or have a threshold?
Formative assessment benefitsPotential benefits:• ‘safe environment’ to test the effectiveness of learning• pre-empts difficulties before they ‘count’• directs progression (scaffolding)• promotes better awareness of strengths and weaknesses• boosts confidence and self-esteem• encourages reflection• offers targeted support• motivates learning (and the learner)
What are the benefits for you as the tutor/instructor?
Background to project – the challenge
• new level 2 course in Earth System Science– conceptually challenging course– requires broad range of scientific skills and prior knowledge
(Biology, Chemistry, Earth Sciences, Maths, Physics)– different approach to ‘standard’ UKOU course (co-published text)
Challenge• devise mechanism to support and inform all stakeholders
of academic progress (students, tutors, course teams…)
• SOFA: Student Online Formative Assessment
Original goals/objectives• formative
– ‘safe’ learning environment to test depth of learning and application– enhance student awareness of personal strengths, weaknesses, achievements
and progression
• adaptive – match questions to learner competence [logistically challenging]
• integrated– embed SOFAs into the whole course, using a learning outcomes approach– involve all stakeholders in feedback process on academic progression
• timely – offer ‘just in time’ and self-directed study options
Book 1
seven SOFAs linked to Chpts 1-7
each SOFA – 10 questions, 2 levels
Revision SOFA – random selection – specific outcomes
3-stage feedback
end results learning outcomes based– overall result– per learning outcome
mixed format (different learning styles/ skills)
SOFA framework
complexity of learning increases between SOFAs
accessed from course website
Initial challenges and problems• student-course interaction an unknown entity
– how would they engage with an online formative assessment– what academic/ time demands would affect engagement
• question setting – what represents the ‘norm’ level and what to test– building in adaptivity:
• 10-15 versions per question = question bank
– technical issues/limitations• dealing with non-standard characters• coping with discursive questions and answers
• stepped learning– designing questions not activities
• time and resource for development and use of data
What are your goals/objectives?
• Why are you interested in formative assessment?
• How does/could it fit into your course structure?
• What would it test (and for whom)?
• Unidirectional or iterative exercise (for whom)?
Question Types
Question type Method of interaction
assign scientific term free text; drag and drop
MCQ (1 or more answers) radial buttons
complete a table free text; drag and drop
link geological processes/ interactions
free text; drag and drop
label diagrams drag and drop
simple/complex calculations (1-2 steps; sig. figs; sci. not, units…)
numeric/text entry; use or find equation, graph, info…
interpret graphs, diagrams, equations
varied
multi-format combination
e.g. Assign scientific term
e.g. MCQ
e.g. Complete a table
e.g. Link geological processes
e.g. Label diagram
e.g. Two-step calculation
e.g. Interpret graphs
e.g. Multi-format question
Final formative feedback
Learning outcomes summary
Evaluation of SOFAs (2007, 2008)
• Metadata collected by system (all users)– number of users, timing and frequency of use per student– time spent per question/ per assessment – mode of interaction: ‘click through’– submitted answers (analyse common errors)
• inbuilt questionnaire at the end of each SOFA (all users)– quick feedback questions – perceived usefulness– free-text comment box
• targeted interviews (success case method analysis)– specific subsection of community– exemplar success and non-success cases
SOFA usage: who and when?• ~60% of all students used at least once• usage per SOFA decreased over time (53% - 20%)• used by more female students (67%) than males (59%)• high achievers x5 more likely to use than not use• low achievers/fails were x2-3 less likely to use it at all
• predominantly used at first scheduled opportunity– if >5 questions attempted, likely to reuse for final revision– revision SOFA predominantly used mid-course
• number of unique users typically declined from Q1-10 – number of attempts per question constant at ~x2 number of users– implication for optimum number of questions?
How were the SOFAs used?• click through showed most did questions in order
– very few reviewed questions first– some repeat users targeted specific questions
• ‘plausible answers’ generally submitted in early attempts – change in use by some users– spurious answers entered (x3) to get to final answer– repeat new version, submitting plausible attempts
• repeated attempts at questions in one sitting– correct answers: repeated positive engagement– thoroughly testing understanding– repeated wrong answers: deterioration in engagement– shift from plausible attempts to spurious answers
Perceived value to learning• First attempt/ scheduled study period
test understanding, useful for revision, measure progress
made learning outcomes apparent, pace learning
• Revision period(s) useful for revision, test understanding, measure progress
made learning outcomes apparent, pace learning
• Structured interviews – shift from extrinsic drivers (e.g. got the right answer)…– to intrinsic motivators (e.g. starting to recognise personal strengths and
weaknesses, increased desire to see sustained improvement)
Student feedbackPositive/neutral experiences• reflective, focussed on personal progression, seeking dialogue/reassurance
This is a great way to test what has been learnt and is a great revision tool!! I now know where I have gaps in my knowledge and also learnt some things which I didn't pick up when reading the book. You don't always appreciate what you’re reading and its significance until you try and apply it. [2007 student]
Negative experience• typically emotive, focussed on perceived personal errors/difficulties
Mathematical questions - very frustrating as I have the method right, but may get a significant figure wrong or miss a step - this cannot be seen, so [the quiz] just says wrong…[2008 student]
Suggestions for improvement• instructional, focussed on technical processes and/or perceived errors in
questionsInstead of saying one or more of your answers are incorrect it should tell you exactly which ones are wrong before you have your second attempt. [2008 student]
Structured interviews - positiveWhat worked? – technical/system set-up• access via course website ‘Check your understanding...’• repeat function and ‘changing questions’• instant feedback on effectiveness of their learning approach
What results were achieved? – learning progression• ‘impartial’ insight into personal strengths, weaknesses and misunderstandings• demonstrated ongoing progress• instant pacing mechanism – review learning approach/materials
What was its value? – perception of worth• focussed learning and helped direct areas in need of ‘extra work’• built confidence in personal abilities and progress• provided ‘real’ measure of personal achievement
Structured interviews - negativeBarriers to use• don’t like working online – preference for paper, more flexible, more familiar• time pressures – too much to do, not worth anything
Barriers to learning• academic level – too high, demoralising when wrong• can’t compare Q&A – preference for passive rather than active learning• too many clicks to final answer – answer rather than process driven• expected to get all questions right first time – unrealistic expectations
Barriers to progression• don’t like being told ‘wrong’ or ‘try again’ – unwillingness to accept feedback• should focus on what is known – unwillingness to test learning• ‘Big brother is watching’ – fear of reproach (all self-esteem issues)
Initial outcomes• majority of students who expressed an opinion, liked the
SOFAs and found them helpful (>97%)– but only reaching proportion of the student population (40% never tried)– more confident and able students more likely to use resource– academically able students make most use of resource
• majority of students who did the SOFAs, repeated them– quantitative data and qualitative perceptions indicate general improvement in
application of learning– positive reinforcement for some; negative demotivator for others
• qualitative feedback demonstrates more confident learners – greater personal awareness of efficiency of learning approach– growing confidence in personal skills and sense of academic progress– awareness of personal strengths, weaknesses and learning needs – changing perception from reviewing feedback (i.e. what was wrong) to focussing
on feeding forward (i.e. what needs to be developed)
• but, some still wanted ‘hard copy’ version– issue of technology or due to familiarity?
Points for discussion
• Role of formative (versus summative) assessment– balancing extrinsic/intrinsic needs and expectations– mode and role of formative feedback– what type of student benefits/ who loses out (is gender really an issue)
• Impact of technology on formative assessment– use of technology to maximise potential effectiveness of formative
assessment and/or feedback (benefit or barrier)– time constraints on staff and students– metadata – what should be collected and how should it be used
• Relevance of ‘self-esteem’ on use and learning– using formative assessment to promote positive self-reflection rather than a
negative barrier to progression– impact of self-esteem on ability to use both positive and negative feedback
Concluding points for reflectionUsed effectively, formative assessment can:• provide a scaffolding for learning• enable a greater sense of self-worth • improve awareness of personal competency levels• motivate individuals to aim higher
but, for some students formative assessment can:• act as a barrier to learning• reinforce current deficits in knowledge and application• negatively impact on self-esteem• demotivate and prevent any attempts to improve
Needs to be implemented carefully into the whole learning experience, with appropriate back-up resources to support all learners recognise the value to personal progression.
Dr Arlëne G. HunterTeaching FellowCentre for Open Learning in Maths, Science, Computing and Technology (COLMSCT)The Open University in Ireland40 University Road BelfastNorthern Ireland. BT7 1SU
