Embed Size (px)
Citation preview
Assessment Workshop
Charlene Stokes
Pam Sharma
Assessment Workshop Morning Session 8:30 – 12:00 noon Breakfast 8:30 – 9:00 Closing the Loop 9:05 – 9:30 Bloom’s Taxonomy of Learning 9:35 – 10:00 Student Learning Outcomes 10:05 – 10:30 Break 10: 30 – 10:45 Alignment 10:50 – 11:20 Interactive Activity 11:25 – 12:00 Lunch 12:00 – 1:00 PM Afternoon Session 1:00 – 4:00 PM Examples of Alignment 1:05 – 1:45 Assessment Methods 1:50 – 2:20 Break 2:25 – 2:45 Interactive Activity 2:50 – 3:20 Assessment Report 3:25 – 3:55 Assessment Survey 3:55 – 4:00
Assessment Student Learning
Assessment is a comprehensive concept, centered on improving student learning through a variety of ways.
Closing the Loop
Mission
Quality Products Graduates with AA,
AS, AAS degrees and certificates
Quality Services
Education
Support Services Institutional
Effectiveness and Research
Outcome Assessment
The systematic collection, analysis, evaluation, and interpretation of information pertaining to student learning and institutional operations in order to inform decision making about the ways in which to improve learning, teaching, and overall institutional effectiveness
http://iacbe.org/oa-definition.asp
Needs Analysis
• Master Course Guide
• Previously Submitted Assessment Reports
• Pre-Assessment Forms submitted spring 2016
• Survey Result fall 2016
Survey Results
0
20
40
60
80
100
Components of SLO
Percent Fulltime
PercentAdjunct
0
10
20
30
40
50
60
70
80
90
100
Assessmetn reportpreparation
Percent Full time
Percent Adjunct
0102030405060708090
Percent Full time %
Percent Adjunct%
Course Assessment Student Learning
Course Description
Course Outcomes
Student Learning Outcomes
General Education Outcomes
Goals Objectives Outcomes
Teacher Centered
Goals
Objectives
Student Centered
Course Outcomes
Student Learning
Outcomes
I N TE N D E D
O B SE R V E D
Course Objectives and Learning Outcomes
The difference between course objectives and learning outcomes is the former describes an intended state (what you hope your students will learn), whereas the latter expresses a present or observed state (what your students actually learned).
http://resources.depaul.edu/teaching-commons/teaching-guides/course-design/Pages/course-objectives-learning-outcomes.aspx
Components of Student Learning Outcome
Audience
Behavior (action)
Condition
Degree
Bloom’s Taxonomy of learning
Available at: http://1.bp.blogspot.com/-qWrdLWtb04w/TmE8SvMtYUI/AAAAAAAAQ9o/fSl9JcHnPPg/s1600/blooms_taxonomy.jpg
Action Verbs for Bloom’s Learning
Available at : https://sites.stanford.edu/teachingcommons/sites/default/files/blooms.png
Examples of Student Learning Outcomes
Art:
Good: • Students will be able to articulate the role art plays in society using
a written critique of an art work. • Students will be able to identify the formal elements and principles
of art, which apply to the creation, and discussion of an artwork. • Students will identify the connection of historical or current events,
which contextualize the making of an artwork. Poor: • Students will appreciate art. • Students will learn how to discuss a work of art. • Students will be familiar with culture and the relationship of art
making.
Examples of Student Learning Outcomes
– Good:
– Learners will demonstrate the ability to communicate effectively in both oral and written forms. (Univ. of Toledo)
– Poor:
– Students will learn how to effectively communicate in both oral and written forms. (Univ. of Toledo)
Student Learning Outcomes
Available at : http://www.cobbk12.org/sites/alt/training/Blooms/circle.GIF
Reasons for Bloom’s Taxonomy
• Accurately measuring students’ abilities requires an understanding of the different levels of cognition that are critical for learning.
• Developing intended student learning outcomes according to Bloom’s Taxonomy helps students understand what is expected of them.
• Plan and deliver appropriate instruction.
• Helps design and implement appropriate assessment tasks, measures, and instruments.
• Ensure instruction and assessment are appropriately aligned with the intended outcomes.
Assessment Methods
• Direct
• Indirect
• Formative
• Summative
Direct Method
Direct methods of assessment ask students to demonstrate the knowledge or skill. Some examples of direct assessment are evaluating students’ abilities to • summarize a process • apply a theory • solve a problem, • synthesize literature. • Tests, essays, presentations, etc. are generally
direct methods of assessment.
Indirect Method A method where students report on their own learning (reflection). Some examples of indirect assessment method are
Asking students to rate their knowledge about a topic after taking the course as compared to what they believe they knew before taking the course.
Surveys and interviews are indirect assessment methods.
Formative Assessment
A regular process of gathering information and feedback on students' learning as the semester is in progress.
Formative assessment is not graded and is conducted as a means to collect immediate feedback on whether students have learned the material or what needs to be done to improve their learning.
Methods include the mid-semester student surveys, student focus groups, classroom assessment techniques such as a minute paper, muddiest points
Summative Assessment
Summative assessments are tests, quizzes, and other graded course activities used to measure student performance. They are cumulative and often reveal what students have learned at the end of a course. Within a course, summative assessment includes the system for calculating individual student grades.
https://www.cte.cornell.edu/teaching-ideas/assessing-student-learning/measuring-student-learning.html
Assessment and Evaluation Dimension Assessment Evaluation
Content: timing,
primary purpose
Formative:ongoing, to
improve learning
Summative: final, to
gauge quality
Orientation:
focus of
measurement
Process-oriented: how
learning is going
Product-oriented:
what’s been learned
Findings: uses
thereof
Diagnostic: identify
areas for improvement
Judgmental: arrive
at an overall
grade/score
Alignment
Student Learning Outcome
Assessment Method
Learning Activities
Assessment Instrument
Alignment
Student Learning Outcome
Assessment Method
Learning Activities
Assessment Instrument
Alignment Examples Type of
learning
objective
Examples of appropriate assessments
Recall
Recognize
Identify
Objective test items such as fill-in-the-blank, matching, labeling, or multiple-
choice questions that require students to:
recall or recognize terms, facts, and concepts
Interpret
Exemplify
Classify
Summarize
Infer
Compare
Explain
Activities such as papers, exams, problem sets, class discussions, or concept
maps that require students to:
summarize readings, films, or speeches
compare and contrast two or more theories, events, or processes
classify or categorize cases, elements, or events using established criteria
paraphrase documents or speeches
find or identify examples or illustrations of a concept or principle
Apply
Execute
Implement
Activities such as problem sets, performances, labs, prototyping, or simulations
that require students to:
use procedures to solve or complete familiar or unfamiliar tasks
determine which procedure(s) are most appropriate for a given task
https://www.cmu.edu/teaching/assessment/basics/alignment.html
Alignment Example Contd.
Type of
Learning
Objective
Example of appropriate assessment
Analyze
Differentiate
Organize
Attribute
Activities such as case studies, critiques, labs, papers, projects, debates, or
concept maps that require students to:
discriminate or select relevant and irrelevant parts
determine how elements function together
determine bias, values, or underlying intent in presented material
Evaluate
Check
Critique
Assess
Activities such as journals, diaries, critiques, problem sets, product reviews, or
studies that require students to:
test, monitor, judge, or critique readings, performances, or products against
established criteria or standards
Create
Generate
Plan
Produce
Design
Activities such as research projects, musical compositions, performances, essays,
business plans, website designs, or set designs that require students to:
make, build, design or generate something new
Putting it all Together
• Pre-Assessment Form
• Pre-Assessment Peer Review
• Assessment Report
• Assessment Report Peer Review
General Education Outcomes
• Communicate effectively in oral and written formats
• Employ or utilize information access and literacy skills
• Demonstrate problem-solving and critical thinking skills
• Employ mathematical and science literacy skills • Acquire a cultural, artistic and global perspective • Demonstrate professional and human relations
skills
WV State Assessment Gen Ed Communications
One-Year Certificate
• Students will use appropriate techniques to deliver messages using various communication channels within the chosen professional environment.
Associate’s Degree
• Students will apply appropriate techniques to deliver messages using a combination of conversing, speaking, listening, corresponding, and writing using various communication channels.
Alignment
• State outcomes for 1-year and Associate Degree graduates
• Gen ed outcome--Communication
• Course outcome—that utilizes communication skills
• Appropriate/matching SLOs
“Student, not teacher driven.”
Teacher
• I will teach my students what the main themes of Romeo and Juliet are.
Student
• Students will write a paper explaining the main themes of Romeo and Juliet.
Questions to Ask: 1) What should students know and be able to do? This list of knowledge and skills becomes your . . .
STANDARDS
2) What indicates students have met these standards? To determine if students have met these standards, you
will design or select relevant . . . AUTHENTIC TASKS
3) What does good performance on this task look like?
To determine if students have performed well on the task, you will identify and look for characteristics of good
performance called . . . CRITERIA
4) How well did the students perform?
To discriminate among student performance across criteria, you will create a . . .
RUBRIC
5) How well should most students perform?
The minimum level at which you would want most students to
perform is your ...
CUT SCORE or BENCHMARK
6) What do students need to improve upon?
Information from the rubric will give students feedback and allow
you to ...
ADJUST INSTRUCTION
Guidelines for writing a standard
• Identify what really matters
• Identify what you want the students to take away from the ….
• Write a standard that is generic and still shows the development of knowledge and skills of the students.
• Students will be able to identify themes across a variety of literature.
Rubric
A coherent set of criteria for students’ work that includes descriptions of performance quality on the levels.
Different types of Rubrics
Analytic • Definition
– Each criterion (dimension, trait) is evaluated separately
• Advantages – Diagnostic info: teacher
– Formative info: student
– Easier link to instruction
• Disadvantages – More time to score
Holistic • Definition
– All criteria (dimensions, traits) are evaluated simultaneously
• Advantages – Faster scoring
– Good for summative
• Disadvantages – Not good for formative
– Overall score doesn’t indicate areas for improvement
• Type of performance: – Products
• Constructed objects
• Written essays, themes, reports, term papers
– Processes • Physical skills
• Use of equipment
• Oral communication
• Examples – Products
• Handmade apron
• Written analysis of the effects of the Marshall Plan
– Processes • Doing a forward roll
• Preparing a slide for the microscope
• Making a speech to the class
Activity Examples
Activity Examples, cont.
• Bookmark: – Create a bookmark to match the theme of the story/book.
• Infomercial: – Students will record a segment that uses persuasion to sell
a product.
• Radio/Newscast: – Describe a legal case and its outcome.
• Brochure: – Create a brochure that explains the steps required to
prepare a microscope slide, hookup an IV, or solve a word problem.
Activity Examples, cont.--flipped
• iClickers/Polling to collect responses to a quiz on the reading assignment.
• Word Webs/Concept Maps “How are these concepts, ideas, people, etc. connected?”
• Dyadic Essays Students come to class prepared with an essay question they have written concerning the topic/concept, along with the “idea response.” Students answer each others’ questions, and following the “idea responses” are shared.
Rubrics used to assess the outcome.
Essential Trait
and
Performance
Level
Level 3
Excellent
Level 2
Average
Level 1
Poor
Organization
Clearly
organized with
headings, intro,
transitions,
body,
conclusion, and
citations
Main idea is
clear, but the
structure is
awkward and
hard to follow.
Ideas are
unfocused or
undeveloped.
Transitions are
nonexistent.
Undeveloped
introduction and
conclusion.
Style/Mechanic
s
Free of
mechanical
errors
1-3 errors More than 3
errors
Activity and the process
Cognitive Hierarchy (Trait/Behavior)
Sample Learning Outcome (What)
Sample Assessment/Activity (How)
Sample Instructional Strategy (Method of preparation)
Analyze Interpret a company’s financial statements to determine credit risk.
Provide students with sample data from a company and assign an analysis report within small groups.
Case studies, lectures introducing credit risk analysis
Remember Identify the insertion and origin of each muscle of the trunk in the human body.
Label a diagram, asking students to identify the origin and insertion of the muscles in the diagram.
Prepare an unlabeled chart and have students group different muscles together by origin/insertion points during a lab with cadavers.
Understand Classify common treatments for mental illnesses
Working in groups, students create a short presentation on a specific mental illness and describe effective treatments used in modern clinical settings.
Instructor provides case studies and the different treatment methods. In groups, class discusses best matches with the cases and then debrief with instructor.
Formative
• Definition: involves activities that measure and provide feedback during the learning process.
• Formative assessment examples include: – Providing verbal or written summaries – Peer activities that reinforce collaboration and assess
the quality of each other’s work – Self-assessments so learners can evaluate their own
performance – Lists, charts, and graphic organizers to encourage
learners to communicate clearly and effectively through making connections and noting relationships
Summative
• Definition: involves using grades, points, or data to demonstrate learner growth after instruction.
• Summative assessment examples include: – Performance tasks to see if certain skills were
acquired
– Multiple-choice tests to examine memory recall
– Participation tasks to encourage learners to engage in lectures, seminars, and online forums
– Oral presentations so learners can demonstrate their knowledge or present research findings
Emerging
(0-12 points)
Developing
(13-16 points)
Advanced
(17-20 points)
Score
1. Organization
(20 points)
Ideas may not be
focused or developed;
the main purpose is
not clear. The
introduction is
undeveloped. Main
points are difficult to
identify. Transitions
may be needed. There
is no conclusion or
may not be clear the
presentation has
concluded. Conclusion
does not tie back to
the introduction.
Audience cannot
understand
presentation because
there is no sequence
of information.
Main idea is evident,
but the organizational
structure many need to
be strengthened; ideas
may not clearly
developed or always
flow smoothly and the
purpose is not clearly
stated. The
introduction may not
be well developed.
Main points are not
clear. Transitions may
be awkward.
Supporting material
may lack in
development. The
conclusion may need
additional
development.
Audience has difficulty
understanding the
presentation because
the sequence of
information is unclear.
Ideas are clearly
organized, developed,
and supported to
achieve a purpose; the
purpose is clear. The
introduction gets the
attention of the
audience and clearly
states the specific
purpose of the speech.
Main points are clear
and organized
effectively. The
conclusion is satisfying
and relates back to
introduction. (If the
purpose of the
presentation is to
persuade, there is a
clear action step
identified and an overt
call to action.)
Style/Mechanics
(20 points)
Language choices may
be limited, peppered
with slang or jargon,
too complex, or too
dull. Language is
questionable or
inappropriate for a
particular audience,
occasion, or setting.
Some biased or
unclear language may
be used.
Language used is
mostly respectful or
inoffensive. Language is
appropriate, but word
choices are not
particularly vivid or
precise.
Language is familiar to
the audience,
appropriate for the
setting, and free of
bias; the presenter
may “code-switch”
(use a different
language form) when
appropriate. Language
choices are vivid and
precise.
Task Description: Each student will make a 5-minute presentation on the changes in one Portland community over the past 30 years. The student may focus the
presentation in any way s/he wishes, but there needs to be a thesis of some sort, not just a chronological exposition. The presentation should include appropriate
photographs, maps, graphs, and other visual aids for the audience.
Excellent (5) Competent (3) Needs Work (1)
Knowledge The presentation
demonstrates a depth of
historical understanding
by using relevant and
accurate detail to
support the student’s
thesis.
Research is thorough and
goes beyond what was
presented in class or in
the assigned texts.
The presentation uses
knowledge which is
generally accurate with
only minor inaccuracies,
and which is generally
relevant to the student’s
thesis.
Research is adequate but
does not go much
beyond what was
presented in class or in
the assigned text.
The presentation uses
little relevant or accurate
information, not even
that which was
presented in class or in
the assigned texts.
Little or no research is
apparent.
Presentation skills The presenter speaks
clearly and loudly
enough to be heard,
using eye contact, a
lively tone, gestures, and
body language to engage
the audience.
The presenter speaks
clearly and loudly
enough to be heard, but
tends to drone and/or
fails to use eye contact,
gestures, and body
language consistently or
effectively at times.
The presenter cannot be
heard and/or speaks so
unclearly that s/he
cannot be understood.
There is no attempt to
engage the audience
through eye contact,
gestures, or body
language.
General Education Outcome: Students will prepare an outline organizing their
speech using good grammar. Essential
Trait and
Performance
Level
Learning
Activity
Assessment
Instrument
or Method
Results
Organization
Student
prepares an
outline of
the speech. Rubric
12/15 (80%) at
Level 5
2/15 (13%)at
Level 3
1/15 (7%) at
Level 1
Style/Mecha
nics
Student
prepares and
outline of
the speech.
Rubric
13/15 (87%) at
Level 5
2/15 (13%) at
Level 3
References
• The Educational Value of Course-level Learning Objectives/Outcomes http://www.cmu.edu/teaching/resources/Teaching/CourseDesign/Objectives/CourseLearningObjectivesValue.pdf
• Common Assessment Terms-Teaching Excellence & Educational Innovation - Carnegie Mellon University https://www.cmu.edu/teaching/assessment/basics/glossary.html
• Align Assessments, Objectives, Instructional Strategies-Teaching Excellence & Educational Innovation - Carnegie Mellon University https://www.cmu.edu/teaching/assessment/assesslearning/tools/groupselfassessmenttool.pdf
• https://www.cmu.edu/teaching/assessment/basics/alignment.html
• Assessment Tools
https://www.cmu.edu/teaching/assessment/assesslearning/tools/groupselfassessmenttool.pdf
• How to develop Rubrics
https://facultycenter.stonybrook.edu/sites/facultycenter.stonybrook.edu/files/basic-pages/91/files/rubric_information.pdf
• Summative Assessment
https://www.cte.cornell.edu/teaching-ideas/assessing-student-learning/using-rubrics.html
• Rubric Component Guide
http://iacbe.org/pdf/rubric-component-guide.pdf
References
• http://www.elearninglearning.com/activities/assessment/examples/?open-article-id=4410553&article-title=formative-vs--summative-assessment&blog-domain=learndash.com&blog-title=learndash
• http://www.aacu.org/sites/default/files/files/publications/LevelsOfAssessment.pdf
• https://uwaterloo.ca/centre-for-teaching-excellence/teaching-resources/teaching-tips/lecturing-and-presenting/delivery/class-activities-and-assessment-flipped-classroom
• http://www.teachhub.com/40-alternative-assessments-learning
References
• https://uwaterloo.ca/centre-for-teaching-excellence/teaching-resources/teaching-tips/assessing-students/aligning-outcomes-assessments-and-instruction
• http://jfmueller.faculty.noctrl.edu/toolbox/standards.htm • http://jfmueller.faculty.noctrl.edu/toolbox/standardtypes.h
tm • http://jfmueller.faculty.noctrl.edu/toolbox/workshopstanda
rd.htm • http://jfmueller.faculty.noctrl.edu/toolbox/workshoprubric.
htm • http://jfmueller.faculty.noctrl.edu/toolbox/workshoptask.ht
m • http://www.uwstout.edu/soe/profdev/rubrics.cfm
• http://www.learndash.com/formative-vs-summative-assessment/
• http://www.introductiontorubrics.com/samples.html
