The “Self Regarding Institution” Using Assessment for Accountability and Improvement Douglas R....

Preview:

Citation preview

The “Self Regarding Institution” Using Assessment for Accountability and Improvement

Douglas R. Davenport

Dean, College of Arts and Sciences

Truman State University

Kirksville, Missouri 63501

douglas@truman.edu

Indiana University South BendAssessment WorkshopMarch 28, 2008

What are your goals?

List three things you hope to gain by attending this workshop.

Rank them in terms of importance.

Questions to Consider…

Why does higher education engage in assessment?

What motivates YOU to do assessment?

Assessment at IUSB…

What are the best examples of assessment you know of on campus?

What might you learn from them? What are the key challenges facing the

implementation of assessment on campus? Who is responsible for assessment at IUSB? (Everybody!)

Where Do We Begin?

“Begin with the end in mind.” (Covey)

What are our Learning Goals?” Write down the student learning goals for…

Your favorite course to teach Your academic program The university

Can they be assessed?

Writing Student Learning Outcomes

Learning outcome statements should: reflect essential knowledge, skills or

attitudes; focus on results of the learning experiences; reflect the desired end of the learning

experience, not the means or the process; Learning outcomes statements may be

considered to be exit behaviors.(Adapted from Lesch, 2008)

Writing Student Learning Outcomes

Key Components: Action word (performance) Learning statement (the learning) Criterion (the conditions of the performance)

(Adapted from Lesch, 2008)

Writing Student Learning Outcomes

Action words to avoid: Appreciate Enjoy Know Realize Be aware of Perceive

Appropriate action words: Use Develop Analyze Express Evaluate Organize Create Write Plan Apply

(Adapted from Lesch, 2008)

Exercise…Write one or more learning outcome statements

for the following goals…

1. Graduates will be able to communicate effectively in writing.

2. Graduates will have an understanding of cultures beyond the United States.

3. Graduates will appreciate the fine and performing arts.

Time for a Temperature Check…“What is the most important thing you

have learned thus far?”

“What important questions remain unanswered?”

How Do We Assess Outcomes?General design principles… Know your purposes (accountability and improvement) Use multiple measures and mixed-methods Make it your own Don’t reinvent the wheel Start slowly Don’t assess every outcome every year Involve faculty Use authentic assessment wherever possible (e.g., embed

assessment in courses, etc)

The Truman Experience Began assessment program in 1972 Started with senior tests Added a writing assessment (SWE) and portfolio

(1980s) Emphasized value-added assessment (1990s) Added self-reports (CSEQ, GSQ, CIRP) (2000s)

Truman Assessment Today Nationally-normed senior tests in every major CLA MAPP and CAPP CSEQ CIRP Portfolio Project Student Interview Project Graduating Student Questionnaire http://assessment.truman.edu

Using Portfolios for Multiple Assessments at Truman Writing Critical Thinking “Modes of Inquiry” Interdisciplinary Thinking http://assessment.truman.edu/components/P

ortfolio/index.asp

Writing Assessment & Portfolios:Rationale “Leverage” current assessment activities A natural location Retains the benefit of embedded assessment Minimizes additional student work

Writing Assessment & Portfolios:Putting it into Practice Tested in 2004, using Critical Thinking

Submissions Incorporated in 2005, using a new “Critical

Thinking and Writing” prompt

Portfolio Findings

The vast majority of students demonstrated competence in all three areas

Organization – 81% Style – 81% Mechanics – 84%

Portfolio Findings

Organization, 2004-2005

2

25

44

28

1

18

47

34

0

10

20

30

40

50

None Weak Competent Strong

Rating

Perc

ent

2004

2005

Portfolio FindingsStyle, 2004-2005

2

21

46

31

1

18

53

28

0

10

20

30

40

50

60

None Weak Competent Strong

Rating

Per

cen

t2004

2005

Portfolio FindingsMechanics, 2004-2005

40

2

16

42

1

15

45

39

0

10

20

30

40

50

None Weak Competent Strong

Rating

Perc

ent

2004

2005

Questions?

e-mail: douglas@truman.edu

Why Assessment? - Accountability External Forces…

Spellings Report, A Test of Leadership (NCLB, Part 2)

Public Funding Priorities Rising Cost of Higher Education Call for Objective, Comparative Measures “It's proper for the government to define ‘legally

drunk,’ but not ‘legally intelligent.’” (Eubanks, 2006)

College Portrait: Voluntary System of Accountability (VSA) Sponsored by NASULGC and AASCU Provides consumer information regarding

institutional measures, student experiences and perceptions, and student learning outcomes.

Learning outcomes– CLA, MAPP, and CAPP Experiences and perceptions – CSEQ, CIRP,

NSSE, CSS

Why Assessment? - Improvement Assessment for Learning The Learning Organization (Senge) Turbulent Times “It is knowledge creation, not knowledge

reproduction, that creates competitive advantage.” (Banta, 2007)

Are These Motivations Mutually Exclusive? Not at all… Accountability is a hallmark of democratic

institutions. Higher education requires transparency. However, assessment to merely pacify is

problematic – it reinforces shallow learning. So…

Characteristics of Assessment by Purpose Accountability Improvement

Audience External Internal

Stakes – Institution High High

Stakes – Programs Low High

Stakes – Students Low Varies

Methods StandardizedNationally-NormedCriterion Referenced

CustomizedLocally-developed

Unit of Analysis Institution Program or Student

Assessment Type Summative Formative

Turbulent Times Did you know?…

Today’s learner will have 10-14 jobs by age 38. The top 10 jobs that will be demand in 2010 did not

exist in 2004. We are preparing students for jobs that do not yet

exist, using technologies that have not been invented in order to solve problems that we have not yet recognized.

Nationally Normed/Criterion Referenced Assessment Instruments – General Collegiate Learning Assessment (CLA - CAE) College Basic Academic Subjects Examination (C

BASE – Univ. of MO) Collegiate Assessment of Academic Proficiency

(CAAP - ACT) Measure of Academic Proficiency and Progress

(MAPP - ETS) iSkills (formerly ICT Literacy Assessment - ETS) Standardized Assessment of Information Literacy

Skills (SAILS – Kent State) WorkKeys (ACT)

Nationally Normed/Criterion Referenced Assessment Instruments – Subject Specific Area Concentration Achievement Test

(ACAT) Major Field Test (MFT – ETS) Praxis II (ETS) A vast array of subject specific exams are

administered through accrediting bodies.

Nationally Normed Assessment Instruments – Self Report National Survey of Student Engagement

(NSSE) College Student Experiences Questionnaire

(CSEQ) Cooperative Institutional Research Program

(CIRP) Freshman Survey (HERI) College Senior Survey (CSS – HERI)

Local Instruments – Some Examples Portfolios Exit Surveys Capstone Projects Juried Theses, Exhibits, Performances

What Do Our Constituents Think?Employers deem both multiple-choice tests of general

content knowledge and institutional assessment that show how colleges compare in advancing critical-

thinking skills of limited value for evaluating applicants’ potential for success in the workplace …

faculty-assessed internships, community-based projects, and senior projects would be the most useful in gauging graduates’ readiness for the

workplace.P. Hart Research Associates, AAC&U (January, 2008)

Recommended