Upload
julie-sievers
View
66
Download
2
Tags:
Embed Size (px)
Citation preview
Evaluating Your Students’ Learning & Whether Your
Project Is Succeeding
Innovators’ Toolkit 4Julie Sievers, Center for Teaching Excellence
Why Evaluate Your Project?
To see if it was worth your time and effort
To decide whether your colleagues / department / field should adopt similar strategies in other courses.
To enable you to base your arguments about these issues on evidence, not just anecdote and impression.
Scholarship of Teaching & Learning:
To enable you to present at a conference, or publish in a journal
A Taxonomy of Questions
1. Assessing Effectiveness of Course, Project, Method
• Did I accomplish my goals?
• Does this strategy work?
• Does it work better than other strategies?
“What Works?” Q – in Pat Hutchings taxonomy of questions, Opening Lines (2000)
Assessment Q: – Similar to program assessment for SACS • “Are you
accomplishing what you set out to do?”
• “How do you know?”
Effectiveness Questions
Success or failure question:
Did students meet the SLOs? Yes or no.• Variation: Did students’ meet this or
that SLO?• Collect evidence similar to that used
for program assessment.
Comparative question:
Did the students meet the SLOs BETTER in this version of the class than in another?• Requires comparison data• May compare to your previous
version, or a standard version you are also offering.
• May compare to a colleague’s course.
Improvement question:
Did the students IMPROVE in their ability to meet an SLO over the course of the semester?• Requires baseline data –
where were students on this SLO in week 1? Where are they in week 15?
• Proves success even if SLO is still not met
Yardstick: learning outcomes
More effectiveness questions
Questions about non-SLO goals
For example: Did this project . . .• Increase student
engagement?• Increase student
motivation?• Facilitate deeper learning?• Improve classroom climate?• Create a classroom that
supports productive failure
Yardstick: Your goals (need to be articulated, but does not have to be an official SLO)
A Taxonomy of Questions2. Inquiry Into Student
Learning
Goal: To better understand • learning processes, • classroom social dynamics, • student errors or misconceptions, • the roles played by various factors
in student learning, etc.
• Descriptive / documentarian• Less oriented around practical
questions of a method, more about fundamental learning issues and processes at work in the course / experience.
“What Is (Happening)?” Q – in Pat Hutchings taxonomy of questions, Opening Lines (2000)
SoTL Q• Common in
Scholarship of Teaching & Learning inquiry
• Less similar to assessment questions.
Inquiry into Student Learning Questions
• What thinking processes do students engage in when learning about X?
• What are students’ beliefs about how they best learn Discipline X?
• How do practices of self-reflection change their learning about X?
• What do students find most difficult about learning in Course X?
• How do students prior understanding of Discipline X affect their ability to acquire new understanding in that field?
• How does their attitude towards Y affect their ability to do work on issue X?
“Here the effort is aimed not so much at proving (or disproving) the effectiveness of a particular approach or intervention but at describing what it [learning] looks like, what its constituent features might be.” – Pat Hutchings
A Taxonomy of Questions3. How Are Things Going?
Goal: To better understand • How students are experiencing
the course• How students are doing – before
they submit a major assignment • Student preconceptions,
misconceptsion• Student study and work habits /
processes
May be:• Anonymous• Ungraded • Diagnostic
Designed to enable you to intervene quickly before semester / unit / project – even class! -- is over
How Are Things Going Questions
Simple / quick tools:• Misconception / Preconception
check• minute paper• Muddiest point (minute) paper• Classroom opinion polls• Course-related self-confidence
surveys• Productive study-time logs• Annotated portfolios• Documented problem
solutions• mid-semester evaluation
(These can also be evidence towards answering your other questions)
See Angelo & Cross, Classroom Assessment Techniques
What’s Your Problem, Anyway?
“One telling measure of how differently teaching is regarded from traditional scholarship or research within the academy is what a difference it makes to have a “problem” in one versus the other.
In scholarship and research, having a “problem” is at the heart of the investigation process; it is the compound of the generative questions around which all creative and productive activity revolves.
But in one’s teaching, a “problem” is something you don’t want to have, and if you have one, you probably want to fix it. Asking a colleague about a problem in his or her research is an invitation; asking about a problem in one’s teaching would probably seem like an accusation.
Changing the status of the problem in teaching from terminal remediation to ongoing investigation is precisely what the movement for a scholarship of teaching is all about.”
Randy Bass, “The Scholarship of Teaching: What’s the Problem?” Inventio: Creative Thinking about Learning and Teaching. (1999)
Activity: Draft Your Question(s)
By Yourself -- 10 minutes
1. Brainstorm a list of questions you wish to answer at end of project
2. Narrow down list to the top 1 or 2
3. Revise the questions to 1. make them more specific2. Make them measurable
With your peer -- 5 min each
1. Explain your top 1 or 2 questions.
2. Peer’s job: Help peer revise questions.
Pairs:• Alex & Yuliya• Gary & Jason• Richard & Mary• Kate & Jimmy• Chris & Rachael
Answering Your Question with Evidence
Evidence . . .
AKA• Data• Artifacts• Student
work
Some things to consider:
• Direct vs. indirect evidence
• Qualitative, quantitative, or both?
• Embedded (“normal educational practice”) vs. add-on
• Product vs. process
Two Examples
From the “How to Start” worksheet in the Vanderbilt SoTL Guide
For literature class, in which a “difficulty log” has been introduced, to be completed as students read a text.
• Analyze the logs, looking for themes or patterns in responses. From the logs, document specific types of difficulty.
From calculus class, in which students have been asked to document their problem solving steps as a way of helping them develop metacognitive skills.
• Looking at scores on test prior to and after the new activity.
Activity: What’s Your Evidence?
By Yourself -- 5 min
Brainstorm the types of evidence you will need to collect to answer your question?
Evidence should fit:• Your questions• Your discipline• Your course• Your timeline
With your peer -- 5 min each
1. Explain your evidence choices.
2. Peer’s job: Help peer refine plan.
Pairs:• Alex & Yuliya• Gary & Jason• Richard & Mary• Kate & Jimmy• Chris & Rachael