Assessing Critical Thinking Summer Critical Thinking Institute

Embed Size (px)

DESCRIPTION

Assessing Critical Thinking Summer Critical Thinking Institute. QEP Team, Faculty Champions, and Academic Roundtables. 2008. Critical Thinking. “Beyond the Obvious”. Assessment Basics. Purpose of assessment Creating valid and reliable measures Alignment of goals/measures - PowerPoint PPT Presentation

Text of Assessing Critical Thinking Summer Critical Thinking Institute

  • Assessing Critical ThinkingSummer Critical Thinking InstituteQEP Team, Faculty Champions, and Academic Roundtables

    2008

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    Critical Thinking

    Beyond the Obvious

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    Assessment BasicsPurpose of assessmentCreating valid and reliable measuresAlignment of goals/measures Use of multiple methods

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    Assessment BasicsWhy do we assess?To see how well we are doingTo confirm what we already knowTo share our progress with othersTo see where we can improve and changeIn some cases to demonstrate what does not work

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    Assessment BasicsWhy do we assess?Source: http://www.c-pal.net/course/module2/pdf/Week1_Lesson5.pdf

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    Assessment BasicsDoes one size fit all?Assessments need to be validAssessments need to be reliable

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    ValidityDoes the assessment measure what it is suppose to measure?Validation is the process of accumulating evidence that supports the appropriateness of inferences that are made of student responses (AERA, APA, & NCME, 1999)

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    Types of Validity EvidenceContent Related - the extent to which a students responses to a given assessment reflect that students knowledge of the content areaConstruct Related - the extent to which the responses being evaluated are appropriate indicators of the underlying constructCriterion Related - the extent to which the results of the assessment correlate with a current or future event

    Consequential the consequences or use of the assessment results

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    Questions to Examine ValidityContent Validity EvidenceDoes the evaluation criteria address any extraneous content?Does the evaluation criteria address all of the aspects of the intended content?Is there any content addressed in the task that should be evaluated, but is not?

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    Questions to Examine ValidityConstruct Validity EvidenceAre all the important elements of the material evaluated through the scoring criteria?Are any of the evaluation criteria NOT relevant to the material?

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    Questions to Examine ValidityCriterion Validity EvidenceWhat are the important components of the future performance that may be evaluated through the use of this assessment?How does the scoring criteria measure the important components of the future performance?Are there any elements of the future performance that are not reflected in the scoring criteria?

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    ReliabilityConsistency of the assessment scoresTypes of reliabilityInterrater Reliability scores vary from instructor to instructor.Intrarater Reliability scores vary from a single instructor from paper to paperA test can be reliable and not valid, but never valid and not reliable

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    Reliability ConcernsReliabilityAre the score categories well defined?Are the differences between the score categories clear?Would two independent raters arrive at the same score for a given student response based on the scoring rubric?

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    Improving Scoring ConsistencyProvide grading rubrics or scoring criteria to students prior to assessmentGrade papers anonymouslyUse anchor papers to define levels of proficiency for referenceUse multiple scorers Calculate reliability statistics during training and grading

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    Assessment BasicsAssessment PurposeEverything needs to align (objectives through assessment)SPC QEP example

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    Assessment BasicsDefinitionMeasurable Learning OutcomesOperational Elements (KSAs)Appropriate Assessment Measures

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    SPC DefinitionCritical thinking is the active and systematic process of communication, problem-solving, evaluation, analysis, synthesis, and reflection, both individually and in community, to foster understanding, support sound decision-making, and guide action.

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    Student Learning OutcomesIn order to link specific and measurable student learning outcomes, SPCs definition of critical thinking was operationalized. This provided a more concrete and less abstract linkage or bridge between the student learning outcomes and the definition of critical thinking.

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    Student Learning Outcomes

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    Student Learning Outcomes

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    Outcomes to AssessmentsStudent Learning Outcomes were then linked to appropriate assessment instrumentsSPCs QEP contained multiple measures for use in assessing student learning in the area of critical thinking

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    Outcomes to Assessments

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    Recent Alumni SurveyQuestion 31: Thinking logically and critically to solve problemsGathering and assessing relevant informationInquiring about and interpreting informationOrganizing and evaluating informationAnalyzing and explaining information to othersUsing Information to solve problems

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    Employer SurveyQuestion 3: Use mathematical and computational skillsComfortable with mathematical calculationsUses computational skills appropriatelyAccurately interprets mathematical data

    Question 5: Think logically and critically to solve problemsGathers and assesses relevant informationInquires and interprets informationOrganizes and evaluates informationAnalyzes and explains information to othersUses Information to solve problems

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    CCSSEQuestion 5: During the current school year, how much has your coursework at this college emphasized the following mental activities? b. Analyzing the basic elements of an idea, experience, or theoryd. Making judgments about the value or soundness of information, arguments, or methods

    Question 12: How much has YOUR EXPERIENCE AT THIS COLLEGE contributed to your knowledge, skills, and personal development in the following areas? e. Thinking critically and analytically

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    Assessment BasicsMultiple MeasuresSPC will determine improvement in students critical thinking skills using the multiple measures.These include standardized direct instruments, authentic assessments, and indirect methods.

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    Student Assessment Points

    Critical Thinking Institue

    Assessing Critical Thinking 2008

    Standardized Direct InstrumentsDirect assessments include:CAT - Critical Thinking Assessment Test is designed to assess and pro

Recommended

View more >