Click here to load reader

Toolkit 4 evaluating students learning

  • View

  • Download

Embed Size (px)

Text of Toolkit 4 evaluating students learning

1. Evaluating Your Students Learning & Whether Your Project Is Succeeding Innovators Toolkit 4 Julie Sievers, Center for Teaching Excellence 2. Why Evaluate Your Project? To see if it was worth your time and effort To decide whether your colleagues / department / field should adopt similar strategies in other courses. To enable you to base your arguments about these issues on evidence, not just anecdote and impression. Scholarship of Teaching & Learning: To enable you to present at a conference, or publish in a journal 3. A Taxonomy of Questions 1. Assessing Effectiveness of Course, Project, Method Did I accomplish my goals? Does this strategy work? Does it work better than other strategies? What Works? Q in Pat Hutchings taxonomy of questions, Opening Lines (2000) Assessment Q: Similar to program assessment for SACS Are you accomplishing what you set out to do? How do you know? 4. Effectiveness Questions Success or failure question: Did students meet the SLOs? Yes or no. Variation: Did students meet this or that SLO? Collect evidence similar to that used for program assessment. Comparative question: Did the students meet the SLOs BETTER in this version of the class than in another? Requires comparison data May compare to your previous version, or a standard version you are also offering. May compare to a colleagues course. Improvement question: Did the students IMPROVE in their ability to meet an SLO over the course of the semester? Requires baseline data where were students on this SLO in week 1? Where are they in week 15? Proves success even if SLO is still not met Yardstick: learning outcomes 5. More effectiveness questions Questions about non-SLO goals For example: Did this project . . . Increase student engagement? Increase student motivation? Facilitate deeper learning? Improve classroom climate? Create a classroom that supports productive failure Yardstick: Your goals (need to be articulated, but does not have to be an official SLO) 6. A Taxonomy of Questions 2. Inquiry Into Student Learning Goal: To better understand learning processes, classroom social dynamics, student errors or misconceptions, the roles played by various factors in student learning, etc. Descriptive / documentarian Less oriented around practical questions of a method, more about fundamental learning issues and processes at work in the course / experience. What Is (Happening)? Q in Pat Hutchings taxonomy of questions, Opening Lines (2000) SoTL Q Common in Scholarship of Teaching & Learning inquiry Less similar to assessment questions. 7. Inquiry into Student Learning Questions What thinking processes do students engage in when learning about X? What are students beliefs about how they best learn Discipline X? How do practices of self- reflection change their learning about X? What do students find most difficult about learning in Course X? How do students prior understanding of Discipline X affect their ability to acquire new understanding in that field? How does their attitude towards Y affect their ability to do work on issue X? Here the effort is aimed not so much at proving (or disproving) the effectiveness of a particular approach or intervention but at describing what it [learning] looks like, what its constituent features might be. Pat Hutchings 8. A Taxonomy of Questions 3. How Are Things Going? Goal: To better understand How students are experiencing the course How students are doing before they submit a major assignment Student preconceptions, misconceptsion Student study and work habits / processes May be: Anonymous Ungraded Diagnostic Designed to enable you to intervene quickly before semester / unit / project even class! -- is over 9. How Are Things Going Questions Simple / quick tools: Misconception / Preconception check minute paper Muddiest point (minute) paper Classroom opinion polls Course-related self-confidence surveys Productive study-time logs Annotated portfolios Documented problem solutions mid-semester evaluation (These can also be evidence towards answering your other questions) See Angelo & Cross, Classroom Assessment Techniques 10. Whats Your Problem, Anyway? One telling measure of how differently teaching is regarded from traditional scholarship or research within the academy is what a difference it makes to have a problem in one versus the other. In scholarship and research, having a problem is at the heart of the investigation process; it is the compound of the generative questions around which all creative and productive activity revolves. But in ones teaching, a problem is something you dont want to have, and if you have one, you probably want to fix it. Asking a colleague about a problem in his or her research is an invitation; asking about a problem in ones teaching would probably seem like an accusation. Changing the status of the problem in teaching from terminal remediation to ongoing investigation is precisely what the movement for a scholarship of teaching is all about. Randy Bass, The Scholarship of Teaching: Whats the Problem? Inventio: Creative Thinking about Learning and Teaching. (1999) 11. Activity: Draft Your Question(s) By Yourself -- 10 minutes 1. Brainstorm a list of questions you wish to answer at end of project 2. Narrow down list to the top 1 or 2 3. Revise the questions to 1. make them more specific 2. Make them measurable With your peer -- 5 min each 1. Explain your top 1 or 2 questions. 2. Peers job: Help peer revise questions. Pairs: Alex & Yuliya Gary & Jason Richard & Mary Kate & Jimmy Chris & Rachael 12. Answering Your Question with Evidence Evidence . . . AKA Data Artifacts Student work Some things to consider: Direct vs. indirect evidence Qualitative, quantitative, or both? Embedded (normal educational practice) vs. add-on Product vs. process 13. Two Examples From the How to Start worksheet in the Vanderbilt SoTL Guide For literature class, in which a difficulty log has been introduced, to be completed as students read a text. Analyze the logs, looking for themes or patterns in responses. From the logs, document specific types of difficulty. From calculus class, in which students have been asked to document their problem solving steps as a way of helping them develop metacognitive skills. Looking at scores on test prior to and after the new activity. 14. Activity: Whats Your Evidence? By Yourself -- 5 min Brainstorm the types of evidence you will need to collect to answer your question? Evidence should fit: Your questions Your discipline Your course Your timeline With your peer -- 5 min each 1. Explain your evidence choices. 2. Peers job: Help peer refine plan. Pairs: Alex & Yuliya Gary & Jason Richard & Mary Kate & Jimmy Chris & Rachael 15. Blog Post 5 Post to your blog your question and the evidence you plan to collect to answer that question. Discuss your choices. Due: Fri, May 29