Upload
clifton-mccoy
View
215
Download
0
Embed Size (px)
Citation preview
How to Develop and/or Update A Simple and Clear
Assessment Plan
Office of Academic Program Assessment
Spring 2013
Overview
Introduction Definition of Basic Terms Differences Between Goals and Outcomes Criteria for Evaluating Goals and Outcomes Using Rubrics in PLOs The Next Step: Developing the Assessment
Plan
Handouts
1. PowerPoint slides
2. WASC Rubric for Assessing the Quality of Academic Program Learning Outcomes
3. Examples of Goals and Outcomes
4. Curriculum map (Sociology)
5. Rubric (BA Nutrition and Food)
6. VALUE Rubric on Critical Thinking
Learning Goal
“a high-level, very general statement of learning expected of graduates, aligned with the institution’s mission, vision, and values (more specific learning outcomes are derived from goals)”
(WASC Draft 2013 Handbook of Accreditation, p. 45)
Learning Outcome“a concise statement of what the student should know or be able to do. Well-articulated learning outcomes describe how a student can demonstrate the desired outcome; verbs such as “understand” or “appreciate” are avoided in favor of observable actions, e.g., “identify,” “analyze.” Learning outcomes can be formulated for different levels of aggregation and analysis. Student learning outcomes are commonly abbreviated as SLOs, course learning outcomes as CLOs, program learning outcomes as PLOs, and institution-level outcomes as ILOs.”
(WASC Draft 2013 Handbook of Accreditation, p. 48)
Assessment PlanAssessment—an ongoing, iterative process consisting of four basic steps:
1.defining learning outcomes
2.choosing a method or approach and then using it to gather evidence of learning
3.analyzing and interpreting the evidence
4.using this information to improve student learning
(WASC Draft 2013 Handbook of Accreditation, p. 40)
Qualities of Goals(adapted from Graves, 2000)
General but not vague Transparent to all stakeholders (no jargon) Realistic Simple—usually not multifaceted Something the program addresses Stated in terms of the learner
Qualities of Outcomes(adapted from Graves, 2000)
More specific than goals Directly relate to goals
hierarchical relationship cause-effect relationship
Focus on what students learn and/or associated processes
More objectives than goals Stated in terms of the learner Transparent to all stakeholders
More Qualities of Outcomes(WASC Rubric for Assessing the Quality of
Academic Program Learning Outcomes)
Complete & well-organized Both discipline-specific and institution-wide
outcomes Clearly distinguish between levels (e.g.,
undergraduate versus graduate) Have explicit criteria statements developed
by faculty (e.g., rubrics)
Example: BA General Communication
GOAL: Students will communicate effectively in a
variety of contexts
SUB-GOALS:
1. Demonstrate proficiency in oral communication
2. Demonstrate proficiency in written communication
Example: BA General Communication
Learning Outcomes for Sub-Goal 1(Demonstrate proficiency in oral communication)
1.determine presentation needs in different situations
2.correctly use visual aids
3.make appropriate language choices
4.use proper structure
5.effectively deliver presentations
Example: BA General Communication
Learning Outcomes for Sub-Goal 2 (Demonstrate proficiency in written communication)
1.implement a variety of style sheets
2.use thesis statements
3.use appropriate organizational strategies
4.apply transitions
5.include appropriate evidentiary support material
6.employ grammar conventions
The Importance of Verbs(Mager, 1975, cited in Brown, 1995)
Multiple Interpretations to know to understand to really understand to appreciate to fully appreciate to grasp the significance
of to enjoy to believe to have faith in
Fewer Interpretations to write to recite to identify to sort to solve to construct to build to compare to contrast
Criteria for Evaluating GoalsHave program goals been specified?1. Is each goal comprehensible?
2. Is each goal justified? – represents institutional goals– represents what faculty and/or discipline feel to
be important
3. Is each goal stated in terms of the learner?
4. Will all goals actually be addressed in the program?
5. Can all goals be achieved by students in the program?
Criteria for Evaluating Outcomes
Have program learning outcomes (PLOs) been developed?1. Does the program specify clear PLOs?
2. Are there distinct differences in the level of specificity between program goals and program learning outcomes?
3. How will achieving program learning outcomes help to reach program goals?
4. Are PLOs achievable within the curriculum?
Criteria for Evaluating Outcomes5. Have each of the following been considered in
developing PLOs?: – Do PLOs focus on the key knowledge, skills and
values students learn in the program? – Are the verbs in PLOs clear and unambiguous?
Do they describe the observable behaviors by which students will demonstrate learning?
– Are the criteria for performance on each PLO explicit and transparent? (Are there benchmarks?)
– How and where can you observe each PLO?– What expectations do you have for student
achievement for each PLO?
Rubrics for PLO Specification Especially for skills-based outcomes Rubrics allow faculty to agree on explicit criteria
statements that describe observable behaviors differentiate levels of performance
• e.g. , Cultural and global awareness/sensitivity including demonstrated understanding, respect and
• support of multiple perspectives from other disciplines, societies, individuals, groups, and cultures
Example Rubric
BA in Nutrition and Food
Awareness of the integration of the different concentrations in FACS and their importance to the relationships between humans and their diverse environments as individuals and groups as a whole.
Rubric Development
Time-consuming Look at existing rubrics
e.g., VALUE rubrics can be used ... without change with revisions as a heuristic
Group Discussion Your program includes “critical thinking” as a
learning goal and has decided to assess it this year. You have not yet specified PLOs or criteria for this goal.
Look at the VALUE rubric
1. How useful would this rubric be to your program for establishing PLOs and explicit criteria?
2. Choose one dimension. How would you need to revise the rubric to represent how that dimension is realized in your program?
Debrief
1. What did you learn about the process of specifying PLOs and criteria from doing this?
2. What did you learn from trying to adapt the rubric?
The Next Step:Sustainable Assessment PlansGood Assessment Plans: Focus on the program (e.g., majors and
concentrations) rather than courses Have a limited number of goals and outcomes Are developed collaboratively Are ongoing Are manageable Use multiple methods of data collection Anticipate how results will be used and lead to
improvement
Comprehensive Assessment Planning
Include both content area knowledge and skills/competencies
Multi-year plan with 1-2 outcomes assessed each year
Direct and indirect methods Capstone student projects (direct) Student exit surveys: self-reporting on all
outcomes
Writing an effective assessment Plan
Outcome Assessed and/or Assessment Question
Method(s) to Collect EvidenceBrief description of what will be collected, how, and by whom. (Enter one method per row; add rows as needed.)
Method to Analyze/ Evaluate* Brief description of how the evidence will be analyzed or evaluated and by whom. (When applicable, include scoring criteria or rubric in an Appendix.)
Timeline & StatusList the semester/dates when the evidence will be collected and evaluated.
Lead Team membersList the name(s) of those who will oversee collecting, analyzing, reporting, and using results.
LO 1 Capstone Assignment (Direct)
Include sample size, scoring criteria, expected outcomes, etc.
LO 1 Exit Survey (indirect)
From UH-Manoa Assessment Template: http://http://manoa.hawaii.edu/assessment/resources/templates.htm
The Next Step:Sustainable Assessment Plans
What should be assessed??
IPP definition of ProgramsWhat appears on the diplomaUndergraduate majors/concentrationsGraduate majors/concentrations
Do we have to assess all of these separately??
The Next Step:Sustainable Assessment Plans
Issues• too many outcomes, not enough time• too many programs, not enough time
An Example: Biological Sciences Undergraduate programs 1,449 students (excluding pre-major) BA Biological Sciences BS Biological Sciences (7 concentrations) Common curriculum for all concentrations: two LD
and two UD courses
How can we assess so many programs??
Too Many Programs—Not Enough Time
Learning Goals/OutcomesStudents will demonstrate the ability to
communicate in the Biological SciencesWritten CommunicationOral CommunicationInformation Literacy
Students will demonstrate the ability to apply the process of scienceCritical ThinkingQuantitative Reasoning
Skills & Competencies
Learning Goal 1: Students will demonstrate the ability to communicate in the Biological Sciences
Learning Goal 2: Students will demonstrate the ability to apply the process of science
LO’s/Courses
Written Communication
Oral Communication
Information Literacy
Critical Thinking
Quantitative Reasoning
Bio 1 I I I I I
Bio 2 I,D I,D I,D I,D
Bio 100 D D D D D
Bio 184 D
Bio 121 D
Bio 139 D D D
Bio 160 D D D D D
Bio 188 M M M MElectives Various Various Various Various Various
BA & BS-General: Curriculum Map
The problem
Shared courses: Bio 1, 2, 100, 184Other courses: problematic!Bio 188: Evolution
Required by 4 of 7 concentrationsWill capture 70% of students
Five Year Assessment PlanL.O. / Year 1 2 3 4 5
Written Communication
Bio 188/capstone
Oral Communication
Capstone
Information Literacy
Bio 188/capstone
Critical Thinking Bio 188/capstone
Quantitative Reasoning
Bio 188/capstone
Content Knowledge
Bio 188/capstone
Indirect self-assessment
Bio 188/capstone
Bio 188/capstone
Bio 188/capstone
Bio 188/capstone
Bio 188/capstone
For more information
• Please contact Office of Academic Program Assessment– Amy Liu ([email protected])– Julian Heather ([email protected])– Shannon Datwyler ([email protected])– Elizabeth Strasser ([email protected])