Assessing for Critical Thinking Narrowing the Gulf Conference March 2009

Assessing for Critical Thinking

Narrowing the Gulf Conference

March 2009

http://www.spcollege.edu/

March 27, 2009 Narrowing the Gulf Conference 2


St. Petersburg College

Presenters

Dr. Carol Weideman, Mathematics Professor, and Assessment Coordinator for the QEP

Maggie Tymms, Assessment Coordinator for Academic Programs

Dr. James Coraggio, Director of Institutional Research and Effectiveness




SPC, established in 1927, is the oldest

2-year college in Florida First Community College in Florida to offer 4

year degrees (2002) 9 Campuses throughout the county 2007-08 FTE: 16,086 (LD), 1,415 (UD) Opening Fall 2008 credited enrollment: 27,076 Annual 2007-08 headcount: 62,200



SACS

SPC was recently reaffirmed in 2008 as part of the SACS affirmation process. Keys to the process included… Compliance with all SACS requirements and

standards Assessment of all academic programs and

support processes Credentialing of all faculty and key staff Continuous improvement of student learning

through a Quality Enhancement Plan (QEP)



The Quality Enhancement Plan (QEP)

Second major requirement by SACS (after reaffirmation). Requirements include… Focus: Significant issue related to student

learning Faculty-driven Broad-based involvement Institution demonstrates ability to carry out

plan Includes methods of assessment Report results to SACS after five years



Broad-based involvement

Selected ‘improve student learning through the enhancement of CT skills’

First step after deciding on a topic was the development of a standardized definition of CT

Involved the entire college including a web blog review process to gather information




SPC’s definition of Critical Thinking:

“Critical thinking is the active and systematic process of communication, problem-solving, evaluation, analysis, synthesis, and reflection, both individually and in community, to foster understanding, support sound decision-making, and guide action.”



Measureable Learning Outcomes

In order to link specific and measureable student learning outcomes, SPC’s definition of critical thinking was operationalized in order to provide a more concrete and less abstract linkage or bridge between the original SPC definition of critical thinking and the student learning outcomes.



Measureable Learning Outcomes



Learning Outcomes



Learning Outcomes



Linkage between Student Learning Outcomes and Assessments

SPC’s QEP contained a number of direct measures and indirect measures for use in assessing student learning in the area of critical thinking.

These assessments have been linked to the specific Core Operational Elements (KSAs) and student learning outcomes.

Provides the relationship between the Core Operational Elements (KSAs), the student learning outcomes, and the specific assessment measures.







SPC will determine improvement in students’ critical thinking skills using the multiple measures.

These include standardized instruments, authentic assessments, and indirect methods.

Before we can measure Critical Thinking… we need to understand what it looks like.



Class Without CT

Didactic instruction where students are presented with factual information from a text book

Assessment is primarily multiple choice items where students are expected to regurgitate factual information

http://www.youtube.com/watch?v=ITmxpe5wjJo






Class With CT

Interactive learning environment where students not only learn facts but the relationship between the facts and the application of that information

http://www.youtube.com/watch?v=UKnZc-MJbTE

Authentic assessment where students are able to model the applications of the discipline through simulations, projects, etc.

http://www.youtube.com/watch?v=UrK2mg9h7kE











How do you measure Critical Thinking in your classroom?



Authentic Assessments

Authentic assessments serve dual purposes of encouraging students to think critically and providing assessment data for measuring improved

student learning.

These assessment techniques fall into three general categories: criterion-referenced rubrics, student reports (reflection or self-assessments), and student portfolios.



Assessment Rubric for Critical Thinking

SPC developed a multi-measure model to assess critical thinking skills, which includes a critical thinking rubric.

Assessment Rubric for Critical Thinking (ARC)

A global rubric template developed to provide a snapshot view of how student learning is being affected by the critical thinking initiative.




Designed to be flexible enough to address a number of student project modalities including written and oral communications.

Will evaluate the student’s use of critical thinking skills in the development of the paper as opposed to specifically evaluating the quality of student’s writing skills.




ARC was designed by the QEP staff and the Faculty Champions to…

Enhance the QEP Align with the College’s definition of critical

thinking Be flexible for use in multi-disciplines




Performance Element

Exemplary(4)

Proficient(3)

Developing(2)

Emerging(1)

Not Present (0)

I. CommunicationDefine problem in your own words.

Identifies the main idea or problem with numerous supporting details and examples which are organized logically and coherently.

Identifies the main idea or problem with some supporting details and examples in an organized manner.

Identifies the main idea or problem with few details or examples in a somewhat organized manner.

Identifies the main idea or problem poorly with few or no details or states the main idea or problem verbatim from the text.

Does not identify the main idea or problem.

II. AnalysisCompare & contrast the available solutions.

Uses specific inductive or deductive reasoning to make inferences regarding premises; addresses implications and consequences; identifies facts and relevant information correctly.

Uses logical reasoning to make inferences regarding solutions; addresses implications and consequences; Identifies facts and relevant information correctly.

Uses superficial reasoning to make inferences regarding solutions; Shows some confusion regarding facts, opinions, and relevant, evidence, data, or information.

Makes unexplained, unsupported, or unreasonable inferences regarding solutions; makes multiple errors in distinguishing fact from fiction or in selecting relevant evidence.

Does not analyze multiple solutions.

III. Problem SolvingSelect & defend your final solution.

Thoroughly identifies and addresses key aspects of the problem and insightfully uses facts and relevant evidence from analysis to support and defend potentially valid solutions.

Identifies and addresses key aspects of the problem and uses facts and relevant evidence from analysis to develop potentially valid conclusions or solutions.

Identifies and addresses some aspects of the problem; develops possible conclusions or solutions using some inappropriate opinions and irrelevant information from analysis.

Identifies and addresses only one aspect of the problem but develops untestable hypothesis; or develops invalid conclusions or solutions based on opinion or irrelevant information.

Does not select and defend a solution.




Performance Element

Exemplary(4)

Proficient(3)

Developing(2)

Emerging(1)

Not Present (0)

IV. EvaluationIdentify weaknesses in your final solution.

Insightfully interprets data or information;identifies obvious as well as hidden assumptions, establishes credibility of sources on points other than authority alone, avoids fallacies in reasoning; distinguishes appropriate arguments from extraneous elements; provides sufficient logical support.

Accurately interprets data or information;identifies obvious assumptions, establishes credibility of sources on points other than authority alone, avoids fallacies in reasoning; distinguishes appropriate arguments from extraneous elements; provides sufficient logical support.

Makes some errors in data or information interpretation; makes arguments using weak evidence; provides superficial support for conclusions or solutions.

Interprets data or information incorrectly; Supports conclusions or solutions without evidence or logic; uses data, information, or evidence skewed by invalid assumptions; uses poor sources of information; uses fallacious arguments.

Does not evaluate data, information, or evidence related to final solution.

V. SynthesisSuggest ways to improve/strengthen your final solution.

Insightfully relates concepts and ideas from multiple sources; uses new information to enhance final solution; recognizes missing information; correctly identifies potential effects of new information.

Accurately relates concepts and ideas from multiple sources; uses new information to enhance final solution; correctly identifies potential effects of new information.

Inaccurately or incompletely relates concepts and ideas from multiple sources; shallow determination of effect of new information on final solution.

Poorly integrates information from more than one source to support final solution; Incorrectly predicts the effect of new information on final solution.

Does not identify new information for final solution.




Performance Element

Exemplary(4)

Proficient(3)

Developing(2)

Emerging(1)

Not Present (0)

VI. ReflectionReflect on your own thought process.“What did you learn from this process?”“What would you do differently next time to improve?”

Identifies strengths and weaknesses in own thinking: recognizes personal assumptions, values and perspectives, compares to others’, and evaluates them in the context of alternate points of view.

Identifies strengths and weaknesses in own thinking: recognizes personal assumptions, values and perspectives, compares to others’, with some comparisons of alternate points of view.

Identifies some personal assumptions, values, and perspectives;recognizes some assumptions, values and perspectives of others; shallow comparisons of alternate points of view.

Identifies some personal assumptions, values, and perspectives;does not consider alternate points of view.

Does not reflect on own thinking.



Sample Scenario (Deer)

Three teenagers were seriously injured in a car accident when swerving to avoid a deer on a two-lane road near a small, rural town in Florida. The residents of the town have seen more and more deer enter the town’s populated areas over recent years. Local law enforcement has been called numerous times this year to remove the animals from backyards and neighborhood streets, and one deer even caused considerable damage as it entered a restaurant in town. The mayor has been charged by the city leaders to keep the town residents safe.




Local crops have even been damaged by the animals. Some long time residents have requested that the hunting season and catch limits be extended in order to reduce the deer population. One city leader even proposed that the city purchase electronic devices to deter the deer from entering populated areas. Health concerns have recently been elevated as three deer carcasses were found at the edge of town and local law enforcement suspect that the animals had been poisoned.



Group Project

Task 1: Communication

Define the main idea or problem in your own words.



Small Group Project

Task 2: Analysis

Identify the Three Available Solutions Compare and Contrast Available Solutions

to the Problem.




Possible Solutions:1. Some long time residents have requested that the

hunting season and catch limits be extended in order to reduce the deer population.

2. One city leader even proposed that the city purchase electronic devices to deter the deer from entering populated areas.

3. Health concerns have recently been elevated as three deer carcasses were found at the edge of town and local law enforcement suspect that the animals had been poisoned.



Task 3: Problem-SolvingSelect the best solution from the possible solutions analyzed.

0% 0%0%

1. Hunting season and catch limits

2. Electronic devices to deter the deer

3. Poison Animals

0 of 30



Group Project

Task 4: Evaluation

Critically evaluate the weaknesses to the solution you chose.



Small Group Project

Task 5: Synthesis

Explain ways to improve and strengthen the solution you chose, and improve any weaknesses you identified. (You may introduce new information)



Group Project

Task 6: Reflection

What did you learn from this process? What were the strengths and weaknesses in

your thinking process? What personal assumptions influenced your

thinking during the process? What could you do differently to improve your

thinking process?



The Deer Scenario was a Helpful Critical Thinking Exercise.

0% 0%0%0%

1. Strongly Agree

2. Agree

3. Disagree

4. Strongly Disagree

0 of 30



Lessons Learned

Provide rubric to students prior to assessment

Anonymous papers Anchor papers defining levels of

proficiency for reference Use of multiple scorers Interrater reliability statistics during

training and grading



Other Critical Thinking Assessments

The CAT - Critical Thinking Assessment Test is designed to assess and promote the improvement of critical thinking and real-world problem solving skills. The instrument is the product of extensive development, testing, and refinement with a broad range of institutions, faculty, and students across the country.

The iSkills™ assessment (former ICT Literacy Assessment), externally developed by ETS, Inc., is a comprehensive test of Information and Communication Technology proficiency that uses scenario-based critical thinking tasks to measure both cognitive and technical skills.

The Measure of Academic Proficiency and Progress (MAPP) test, externally developed by Educational Testing Services, Inc. (ETS), Inc., is a measure of college-level reading, mathematics, writing, and critical thinking in the context of the humanities, social sciences, and natural sciences.



Critical Thinking Assessment Test

“The CAT instrument is a unique tool designed to assess and promote the improvement of critical thinking and real-world problem solving…

The test was designed to be interesting and engaging for students. Most of the questions require short answer essay responses and a detailed scoring guide helps insure good scoring reliability”

(Tennessee Tech University, Critical Thinking Assessment Test Overview).










In collaboration with Tennessee Technological University and with support from the National Science Foundation, St. Petersburg College received a grant to administer the Critical Thinking Assessment Test (CAT) instrument to a representative sample of approximately 100 students enrolled in the College during 2008.




SPC administrators attended regional training workshops at Tennessee Technological University in 2008 and 2009.

Eighty-seven CAT assessments were administered during 2008 to SPC students enrolled in five courses.

A CAT Scoring Session was held on July 11th, 2008 to score these assessments.




Source:



I Will Be Able To Apply The Information ILearned In This Presentation.

0% 0%0%0%

1. Strongly Agree

2. Agree

3. Disagree


0of30



I Enjoyed This Presentation!

0% 0%0%0%

1. Strongly Agree

2. Agree

3. Disagree


0of30


Narrowing the Gulf Conference

March 2009

http://www.spcollege.edu/

Documents

Assessing for Critical Thinking Narrowing the Gulf Conference March 2009