Upload
abner-dickerson
View
215
Download
0
Embed Size (px)
Citation preview
Evaluation
PGCert in Learning and Teaching
Module 402:Course Design and
Evaluation
Learning outcomeDesign effective evaluation strategies through a critical comparison of alternative strategiesOrWhat forms of evaluation occur in your
module/ programme nowWhat approaches are used in other
programmesWhat are the strengths/weakness of
‘your’ system◦ In comparison with others and with a model
What might you adapt to improve evaluation practice in your module/ programme
Definitions Evaluation is:
A process whereby a teacher, group of students, or their institution,
find out, on the basis of their own perceptions, the overall quality of a programme, a particular course, or a teacher
generally part of a formative process aimed at improving standards. heacademy.ac.uk
Who
Does what
Why
Some key pointsEvaluation is more than student
feedback◦ But includes student feedback as a key source of
evidence◦ Here student feedback means feedback from
students, about the programme, not feedback to students, about their work
USA uses assessment to mean evaluation (UK)
In general:◦ Module evaluation tends to focus on student
feedback◦ Programme evaluation should have a wider more
systematic approach◦ You need to consider both levels in the assignment
Why find out moreTo find out about:Student learningStudent difficultiesYour teaching
◦ Strengths and weaknesses
To engage students in their learning experience
To provide evidence for:Your professionalismQuality control
◦ Organisational requirement Morss & Murray 2005
Why find out moreWhat systems are there in your
area for finding out about teaching, learning, student perceptions for your module
Articulating your own (departmental) practice and listening to what others do is one step toward the assignment's ‘critical comparison’
Evaluation: The systemEvaluation is more than student
feedbackThere are many sources of
informationIt can be thought of as a process
◦CircularUsually driven by the institution
◦Often applied differently in different departments
◦Regularly applied to modules & programmes as well as to individual lecturers
The evaluation cycle
5 Agree actionNotify students, staff
6 Implement changesReport
3 Gather: Data, feedback
2 Devise: Strategy - method(s), timing
4 Analyse & interpret
1 Decide: Reasons, purpose, level
HEAcademy.ac.uk
1 Purpose of EvaluationMain purpose:
• enhance students’ experience of learning and teaching• contribute to monitoring and review of quality and standardsAlso contributes to:• effectiveness of course design and delivery• dialogue with students• students reflecting on their experiences• enhancing the teaching and learning process• identifying good practice• measuring student satisfaction
which can lead to marketing (recent development)
• staff developmentheacademy.ac.uk
Question:Which of these predominate in your area
‘Level’ of the evaluation- where is the focus?
Individual session(s) – delivery, learningModule – delivery,
administration/resourcesProgramme – delivery, integration,
overallUniversity – services
Critical questionsIn your area: Which of these is evaluated as a ruleHow is it done and who causes it to be doneWho gets any value
2 Devise the strategySources of information, and timing
Sources of information:Students:
◦ Feedback◦ Staff student liaison committee◦ Performance in assessment◦ National Student Survey – increasingly
importantSelf, peers, mentors, external
examiners, reviewsExternal sources:
◦ Employers and Professional bodies◦ Other universities’ courses◦ League tables
Question:Which of these has been useful or impacted on your work
When to evaluate - timingTo benefit current students: After an individual session Within a module - for immediate feedback on
comprehension, interest, perception of relevance, workload
To benefit future students: At the end of a module - to gauge whether
learning outcomes have been met, and potential changes for the next time
At the end of the year? Every 5 years? (UK systems)
Which of these are you aware of/ benefit from?
3 Gather: How and what process?
Method:Questionnaires (hard copy and electronic)Focus group – various approachesStudent assessment resultsPeer reviewCATs - Classroom Assessment TechniquesAny other way?Process:Who will collect this data
◦ When - in a session, after it, ◦ What format – paper forms, summary report,
electronic??
Questionnaires: rationale
Decisions to be made:◦Quantitative
◦ for comparison between modules/teachers and over time
◦Qualitative ◦ to allow more in-depth and tailored responses
◦Open or closed questions
◦Hard copy or VLE/ web based (Survey Monkey)
◦Repeating questions from previous years (for comparison) or changing the questions to give new insights
◦Aligning questions to the NSS
Questionnaires: design
Bear in mind:
◦ Is the purpose clearly stated
◦ Are the questions unambiguous. Are the questions answerable
◦ Are they phrased in a way that is easily understood
◦ Is there a mix of open and closed questions
◦ How many questions are being asked
◦ Is the Likert scale being used appropriately?
(adapted from Brennan and Williams 2004)
NB questions themselves are often not researched or evaluated
See the resource in vital: ‘Questions to consider when critiquing the evaluation questionnaires’
4 Analyse - interpretHow are paper responses
collated – put into electronic format?
How analysed?How are qualitative comments
analysed and reported?Who does all this?
Question:What happens in your area?
5 Agree action
What happens to the results of the evaluation?
How do the students or providers of feedback know what will be changed for next time?
Where is the commitment to change?Who is responsible for ensuring that
change happens?Question:In your area: How does this happen?Does it ‘work’?
Change the deliveryUpdate the curriculumProvide opportunities for staff
developmentClarify expectations
look at the student handbook!Update resourcesTell the Students!!
6 Implement the changes
Question:Who does all this in your area?
How can we get students to respond and take it seriously?
Make time in sessions
Require attendance/ completion
Make it easy (e.g. Vital)
Don’t overload - too many questions, too often
Vary evaluation methods
Written comments rather than tick boxes?
Give examples of constructive feedback
Establish ground rules
Give feedback e.g. on notice boards, Vital
• Demonstrate the effect
Question:Which of these is used in your areaWhich could be?
Coming soon……..University purchase of Evasys
◦Will change evaluation practice
Increasing responsiveness to NSS◦Becoming more thoughtful, less
reactive?
NSS Questions: areas & e.gs.
The teaching on my courseStaff have made the subject interesting
Assessment and FeedbackFeedback on my work has helped me clarify things I did not understand
Academic supportI have received sufficient advice and support with my studies
Organisation & ManagementThe course is well organised and is running smoothly
Learning ResourcesThe library resources and services are good enough for my needs
Personal developmentI feel confident in tackling unfamiliar problems
Overall satisfactionOverall, I am satisfied with the quality of the course
Possible additions to NSSAcademic challenge/reflective and
integrative learning, e.g. ◦My course has challenged me to achieve my best
The learning community/ collaborative learning
◦I have not been encouraged to talk about academic ideas with other students
Student voice ◦Staff appear to value the course feedback given by students
I have been encouraged to use technology to enhance my learning
HEFCE 2014
Alternatives
UK variation on National Survey of Student Engagement; NSSE (US & Aus)
◦ Gibbs (2012): “NSSE provides a more valid predictor of student gains than does NSS”
◦ See HEA website ‘Engagement for enhancement’ & ‘Institutional case studies (NSSE pilot)’
What other processes contribute to evaluation?
Recap: the university has other processes beyond student feedback to evaluate the quality of programmes◦ Annual Subject Review report◦ Internal Periodic Review◦ Professional Body Review◦ External Examiners◦ Institutional Audit – QAA (November 2015)◦ Required responses to NSS
see Student Experience Committees
Have you experience of theseHave you seen value from them
Learning outcomeDesign effective evaluation strategies through a critical comparison of alternative strategiesOrWhat forms of evaluation occur in your
module/ programme nowWhat approaches are used in other
programmesWhat are the strengths/weakness of
‘your’ system◦ In comparison with others and with a model
What might you adapt to improve evaluation practice in your module/ programme
QuestionsIn your
department/programme/team how effectively are evaluation processes leading to change
Consider one major change that you could make to evaluation in your area
What aspects of ‘your’ evaluation systems would you recommend to others
Use the questions on the slides as prompts for your assignment
Resources Angelo, T., & Cross, P. (1993) Classroom Assessment Techniques: A Handbook for
College Teachers, 2nd edition Brennan, J & Williams, R.(2004) Collecting and using student feedback
www.heacademy.ac.uk/ First words on Evaluation- Oxford Brookes University
www.brookes.ac.uk/services/ocsd/firstwords/ Haugen, L, (1999) Classroom Assessment Techniques (CATs), Iowa State
University http://www.celt.iastate.edu/teaching/cat.html HEFCE (2014) NSS Review https://
www.hefce.ac.uk/pubs/rereports/Year/2014/nssreview/Title,92164,en.html Macdonald, R. (2006) The use of evaluation to improve practice in learning and
teaching. Innovations in Education and Teaching International Vol. 43, No. 1, February 2006, pp. 3–13
Morss, K., & Murray, R. (2005) Teaching at University: A Guide for Postgraduates and Researchers
Shevlin M., Banyard P., Davies M., & Griffiths M. (2000) The Validity of Student Evaluation of Teaching in Higher Education: love me, love my lectures? Assessment & Evaluation in Higher Education, Volume 25, Number 4, 1 December 2000, pp. 397-405(9)
Why find out more
Demonstrating evidence based
practice
Improving student experience
Improving your experience!
Raising profile (yours)
Supporting change
One-Minute Paper
Course Knowledge and SkillsDuring last few minutes of class
period, ask students to use a half-sheet of paper and write "Most important thing I learned today and what I understood least."
Review before next class meeting and use to clarify, correct, or elaborate.
Haugen 1999 (from Angelo & Cross 1993)
Muddiest Point
Course Knowledge and SkillsSimilar to One-Minute Paper but
only ask students to describe what they didn't understand and what they think might help.
Same as One-Minute Paper. If many had the same problem, try another approach.
Haugen 1999 (from Angelo & Cross 1993)
How can you gather evidenceAnd what counts as evidence
Your own evidence:Teaching diary: your record,
responses, reflections, rationales, observation of students, informal notes (e.g. on ppt slides)
Video or audio tapingChecklistLearning log
◦Goals and achievements (may be shared)
VariousTeaching methodsQuestionnaires
◦ Standardised◦ Designed by you for specific purposes◦ Input from students
Focus groups◦ Run by neutral person
Staff- student liaison committeesPeer review
◦ Peer observation; mentor; supervisor