Gregory Light, PhD

Brazil Science, Technology, Humanities, Engineering and Mathematics (STHEM) Consortium

1ST WORKSHOP: LORENA, BRAZIL, MAY 26 30, 2014‐

Changing Teaching, Transforming Learning

Session II: Assessing Learning

OverviewSession I: Design Principles & Learning Outcomes

• Global & National Challenges (Interactive Presentation)

• Teaching for Learning: a Framework (Interactive Presentation)

• Constructing Learning Outcomes (Activity)

Session II: Assessing Learning

• Assessing Student Learning (Discussion)• Dimensions of Assessment (Interactive

Presentation/Activity)• Aligning Assessment (Activity)

“The quickest way to

change student

learning is to change

the assessment system.”

-- Elton & Laurillard, 1979, p. 10

ASSESSMENT

Group Question:

Why do we assess students?

At TablesShare

Purposes of AssessmentSummative

• To pass or fail a student• To grade or rank a student• To license to proceed• To select for future courses• To license to practice• To predict success in future courses• To predict success in employment• To select for future employment

Purposes of AssessmentFormative

• To provide feedback to students to improve their learning

• To motivate students• To diagnose a student’s strengths and

weaknesses• To help students develop their skills of

self-assessment• To provide a profile of what a student

has learnt

Purposes of AssessmentEvaluative

• To provide feedback to teachers• To improve teaching• To evaluate a course’s strengths and

weaknesses• To make the course appear

‘respectable’ and credit worthy to other institutions and employers

Dimensions of Assessment

Dimensions of Assessment

FormativeSummative

Reliability The extent to which the results of the assessment method can be trusted

ValidityThe extent to which the

assessment methods reflect student learning and the learning

goals of the course

Essentially designed to sum

up someone’s achievement

Essentially designed for use

in helping the learning process

Practicality

Criterion Referenced

Norm-Referenced

Based on self-reflection of

learning achieved on the course

Self-ReferencedBased on

knowledge and skills learned on

course

Based on comparisons with others in

the group

Peer-ReferencedBased on peer

appraisal of learning achieved

on the course

Dimensions of Assessment

FormativeSummative

Reliability The extent to which the results of the assessment method can be trusted

ValidityThe extent to which the

assessment methods reflect student learning and the learning

goals of the course

Essentially designed to sum

up someone’s achievement

Essentially designed for use

in helping the learning process

Practicality

Criterion Referenced

Norm-Referenced

Based on self-reflection of

learning achieved on the course

Self-ReferencedBased on

knowledge and skills learned on

course

Based on comparisons with others in

the group

Peer-ReferencedBased on peer

appraisal of learning achieved

on the course

Dimensions of Assessment

FormativeSummative

Reliability The extent to which the results of the assessment method can be trusted

ValidityThe extent to which the

assessment methods reflect student learning and the learning

goals of the course

Essentially designed to sum

up someone’s achievement

Essentially designed for use

in helping the learning process

Practicality

Criterion Referenced

Norm-Referenced

Based on self-reflection of

learning achieved on the course

Self-ReferencedBased on

knowledge and skills learned on

course

Based on comparisons with others in

the group

Peer-ReferencedBased on peer

appraisal of learning achieved

on the course

Dimensions of Assessment

FormativeSummative

Reliability The extent to which the results of the assessment method can be trusted

ValidityThe extent to which the

assessment methods reflect student learning and the learning

goals of the course

Essentially designed to sum

up someone’s achievement

Essentially designed for use

in helping the learning process

Practicality

Criterion Referenced

Norm-Referenced

Based on self-reflection of

learning achieved on the course

Self-ReferencedBased on

knowledge and skills learned on

course

Based on comparisons with others in

the group

Peer-ReferencedBased on peer

appraisal of learning achieved

on the course

Dimensions of Assessment

FormativeSummative

Reliability The extent to which the results of the assessment method can be trusted

ValidityThe extent to which the

assessment methods reflect student learning and the learning

goals of the course

Essentially designed to sum

up someone’s achievement

Essentially designed for use

in helping the learning process

Practicality

Criterion Referenced

Norm-Referenced

Based on self-reflection of

learning achieved on the course

Self-ReferencedBased on

knowledge and skills learned on

course

Based on comparisons with others in

the group

Peer-ReferencedBased on peer

appraisal of learning achieved

on the course

Dimensions of Assessment

FormativeSummative

Reliability The extent to which the results of the assessment method can be trusted

ValidityThe extent to which the

assessment methods reflect student learning and the learning

goals of the course

Essentially designed to sum

up someone’s achievement

Essentially designed for use

in helping the learning process

Practicality

Criterion Referenced

Norm-Referenced

Based on self-reflection of

learning achieved on the course

Self-ReferencedBased on

knowledge and skills learned on

course

Based on comparisons with others in

the group

Peer-ReferencedBased on peer

appraisal of learning achieved

on the course

TEACHING-CENTERED

LEARNING-CENTERED

Activity: Assessment Methods

Choose an assessment method(s) you currently use and “map” it on to 1) a learning outcome2) these dimensions.

Do individually, then share at tableThen with whole group

Assessment Methods(Some Examples)

• final exam (written)

• Homework• projects• reports• Presentations• observed

discussion groups

• office hours

• final exam (multiple choice)

• quizzes• cold calling• career

performance• standardized

tests• personal

response systemsHigher validity – less reliable Higher reliability – less validity

Aligning Assessment

Using Rubrics

(Making valid assessments more

reliable)QuickShare

Aligning Assessment with a Learning Outcome (Biology Example)

Course Goal

Learning Outcome

FormativeAssessment

Summative Assessment

What will students learn?

If they have learned it, what will students know and be able to do?

What will students do to learn it?

How will students demonstrate they know it or are able to do it?

Students will understand the transfer of information from DNA to proteins.

Students will be able to predict changes in amino acid sequences caused by mutations.

In groups students are given sequence of DNA corresponding to amino acid sequence. Students identify reading frame and predict amino acid changes due to mutations in that sequence.

On exam students will predict the new amino acid sequence that results from a mutation in a given gene sequence.

Knight, (2011)

A Typical Rubric Format STUDENT EVIDENCE

(1) WeakLittle or

no evidence

of outcome

(2) BasicSome

evidence of outcome

(3) ProficientDetailed

evidence of outcome

(4) StrongHighly

creative; outcome

TEACHERCRITERIA

Criterion 1Argument  

Criterion 2 Integration of literature

Criterion 3Writing quality

Adapted from Beauchamp et al 1996

Assessing Writing: Example Rubric

  (1) WeakLittle or

no evidence

of outcome

(2) BasicSome

evidence of outcome

(3) ProficientDetailed

evidence of outcome

(4) StrongHighly

creative; outcome

Criterion 1Argument  

Rambling; poor use of logic; personal opinion

Basic structure with some use of evidence

Strong structure & logic, evidence used throughout

Strong structure & evidence; sophisticated discussion

Criterion 2 Integration of literature

No or minimal use of sources

Sources used, but no integration of writer’s ideas

Sources well integrated into author’s ideas

Sources well integrated; critique made

Criterion 3Writing quality

Poor organization, grammar, syntax

Some effort to organize ideas; grammar/syntax problems

Good organization; very few grammar/ syntax problems

Well-developed flow; error-free; elegant style

Adapted from Beauchamp et al 1996

Benefits of Rubrics

Used for grading (Summative) or feedback (Formative)

Clear criteria (Criteria Ref.) Ensure grading aligns with learning

outcomes (Validity) Shared with multiple teacher

graders/situations (Reliability) Can be used with students (Self Ref.)

and peers (Peer Ref.)

Activity

In small group, identify

and design a rubric for a

specific Learning outcome

Large Group Discussion

Aligning Assessment

Using Multiple Choice Questions (MSQ)

(Making reliable assessments more valid)

QuickShare

MCQs as formative instruction

Answer and provide your rationale:

1. Which of the following has/have intrinsic pacemaker characteristics?

a) Medulla c) Sinoatrial node

b) Pons d) Atrioventricular node

Use rationales (short open-ended explanation for choice) to let students demonstrate their learning - can be graded or not..

Stanford Learning Lab: Nash & Shaeffer, 1999; Schaeffer et al., 1999

Ideal rationale: SA node is the normal pacemaker for the entire heart. AV node also has pacemaker potential, but is overshadowed by SA node. Medulla has pacemaker potential for breathing rhythm as well. Pons helps refine rhythm, but does not have pacemaker potential.

Less-than-ideal rationales: Offering an incomplete answer: Normally the SA node is responsible for generating heart rate, and it is able to do this because of its intrinsic rhythm. The AV node also has an intrinsic rhythm, but it is “overshadowed” by that of the SA node.

Providing a book definition: The sinoatrial node is the pacemaker of the mammalian heart.

Providing irrelevant information: Stretch receptors are located in the aortic arch and the carotid sinus. They have the ability to respond to changes in pressure.

Restating the question: The SA node, AV node, and medulla all possess intrinsic pacemaker characteristics as they all serve as intrinsic pacemakers.

Blind appeal to authority: This answer is right because Professor Heller said that it was, and Professor Heller is cool.

Activity

In small group, share other

ways in which the validity

of MCQs might be

enhanced.

Share with Large Group

Testing low-level knowledge

Purely economic loss is recoverable in a product liability action.a) Trueb) False

Purely economic recovery will be barred in which of the following causes of action?

c) Negligenced) Fraud e) Defamationf) Product liability

“What’s the rule?”

No context, not allowing for interpretation/analysis

from Case & Donohue, 2008

Revision: Higher-level understanding

A restaurant hired an exterminator to eliminate cockroaches from the basement under the restaurant. Around midnight, the exterminator applied to the basement floor and walls an effective pesticide that he had purchased from the manufacturer. A toxic gas released by the pesticide penetrated into the restaurant kitchen and did not disperse by the next day. As a result, the restaurant was required to close that day.

The restaurant brought a tort action based on product liability against the pesticide manufacturer for lost profits.

Will the restaurant prevail?

a) No, because in this action purely economic loss in not recoverable. *b) No, because the exterminator was the proximate cause of the restaurant's

damages. c) Yes, because the manufacture of pesticides is an abnormally dangerous

activity. d) Yes, because the pesticide was being used as intended.

from Case & Donohue, 2008

3 ideas to take with you

Assessment is teaching

Students need to be involved

Assessment gives you no more than what you ask for

Final

QUESTIONS?

References• Entwistle, N. & Tait, H. (1990) ‘Approaches to learning,

evaluations of teaching and preferences for contrasting academic environments’, Higher Education, 19 (2): 169–94.

• Knight, J. (2011) University of Colorado. NAS/HHMI Summer Institute on Undergraduate Biology Education, Madison, WI.

• Light, G, & Micari, M. (In press) Making Scientists: Six Principles for Effective College Teaching, Harvard University Press.

• Beauchamp, McConaghy, Parsons & Sanford. (1996) Teaching From the Outside In. Duval: 1996, 37.