ASSESSMENT AS SCHOLARSHIP:IMPROVING PROGRAMS,

INSTITUTIONS, AND DISCIPLINES

Thomas P. Pusateri, Ph.D.Florida Atlantic University

pusateri@fau.edu

Closing Plenary Session 13th Annual Georgia Conference on

College & University TeachingMarch 25, 2006

RESOURCES FROM JOSSEY-BASS

2004

AssessmentClear and

Simple

(Walvoord)

1990

ScholarshipReconsidered

(Boyer)

1997

ScholarshipAssessed

(Glassick, Huber, & Maeroff

2001

ScholarshipRevisited

(Kreber)

ASSESSMENT IN THE NEWSThe Atlantic Monthly (November 2005)

• How College Affects Students (Pascarella & Terenzini) Going to college makes a difference But no differences among colleges, controlling for student quality

• Hersh: Current assessment measures may not be picking up differences

• Who will control the agenda for assessing the worth of college? U.S. Congress: Reauthorization of the Higher Education Act 40 State legislators: FL Board of Governors–Academic Learning Compacts Bush administration: Commission on the Future of Higher Education

• Hersh: Institutions ought to take charge to assess cumulative learning Formative assessment: Carleton College (MN); Alverno College (WI) Value-added assessment: Wabash College (IN) study of 16 institutions

ASSESSMENT AS SCHOLARSHIP

• This presentation focuses on assessment for program improvement– Accountability: Track individual students– Program improvement: Aggregate the data

• Assessment is “action research” for “informed intervention” – Models from social science and education

Sampling; Pre-post differences; Comparisons– Quantitative and Qualitative data

GOALS OF THIS PRESENTATION

• Provide practical examples of assessment that you might adapt at your institution– Examples from Loras College and FAU

• Suggest strategies for communicating findings to colleagues (“Lessons learned”)– Graphing data; Conducting/Reporting audits

• Identify venues for presenting and publishing research on assessment– SoTL organizations, conferences, journals

EXAMPLE 1MEASURING STUDENT ATTITUDES

• Loras College: Attitudes towards general education– Faculty developed 41 outcomes statements

“Loras graduates should be able to….” (“understand” or “appreciate” disciplines)

– Survey requested ratings on a 5-point scale from “Extremely important” to “No importance”

– Most students completed the survey twiceEarly in the first year and late in the senior year

• Comparison groups– Longitudinal (First year – Senior) comparisons

Value added approach: Did students’ values change?

– Comparison groups: FTICs to Transfer studentsDid completing four years at Loras matter?

Items on the First-Year/ Senior Surveys 1st 4th DIFF

20: Understand foreign cultures 2.87 3.52 0.65

32: Skills in foreign language 2.60 3.24 0.64

07: Appreciate visual arts 2.51 3.09 0.58

11: Understand other cultures 3.82 4.38 0.56

12: Skills in speaking 3.65 4.21 0.56

16: Open to controversies 3.89 4.43 0.54

06: Volunteer service 3.03 3.52 0.49

14: Appreciate literature 2.98 3.47 0.49

36: Appreciate musical styles 2.43 2.91 0.48

34: Understand/interpret history 3.27 3.74 0.47

01: Evaluate others' opinions 3.80 4.26 0.46

24: Appreciate theatrical arts 2.91 3.36 0.45

39: Understand artists's works 2.37 2.81 0.44

40: Familiar with several fields 3.90 4.32 0.42

28: Skills in critical thinking 4.04 4.46 0.42

31: Interpret human behavior 3.47 3.89 0.42

19: Concern for good of society 3.24 3.65 0.41

27: Can reason scientifically 3.14 3.55 0.41

21: Open to different values 3.69 4.09 0.40

41: Understand historical events 3.50 3.89 0.39

Items on the First-Year/ Senior Surveys 1st 4th DIFF

13: Skills in writing 4.20 4.57 0.37

38: Understand social structures 3.34 3.71 0.37

15: Understand human behavior 3.20 3.57 0.37

04: Understand scientific method 2.97 3.33 0.36

30: Appreciate ancient culture 2.31 2.67 0.36

05: Skills in persuasion 3.88 4.23 0.35

03: Physical fitness & health 3.13 3.44 0.31

22: Understand biology's role 3.05 3.34 0.29

18: Relate ethics/faith to morality 3.57 3.85 0.28

29: Skills in reading 4.22 4.50 0.28

33: Understand chemistry’s role 2.72 2.99 0.27

08: Conduct library research 4.36 4.62 0.26

17: Skills in technology 4.39 4.63 0.24

26: Understand political systems 3.51 3.70 0.19

02: Work effectively with others 4.41 4.58 0.17

10: Understand Catholicism 3.18 3.33 0.15

23: Skills in mathematics 3.49 3.63 0.14

09: Appreciate mathematics 3.23 3.32 0.09

35: Understand economics 3.79 3.86 0.07

37: Appreciate computers 4.02 4.05 0.03

25: Prepared for careers 4.82 4.71 -0.11

LESSON 1MEANS CAN BE MEANINGLESS

• Recommendation: Graph frequency distributions– In Microsoft Excel, choose “100% Stacked Bar”

Based on 380 students who completed both versions of the survey.

Based on 441 non-transfers and 43 transfer students.

Differences in FTIC & Transfers

Differences in 1st year & Senior year

EXAMPLE 2ASSESSING WRITING SKILLS

Loras College: Sophomore writing portfolios• Phase 1: Assessment committee (2 summers)

– Collected portfolios (3 papers from each student)

– Read a subset of portfolios; wrote a scoring rubric

– Evaluated remaining portfolios (interrater reliability)

– Provided students feedback• Phase 2: Faculty workshops (2 summers)

– Training on using the scoring rubric– Evaluating portfolios (interrater reliability)– Discussing findings and applying to courses– Provided students feedback

Based on assessment by 21 faculty who read a subset of 49 student portfolios

LESSON 2MISSED OPPORTUNITIES

Carleton College (MN): AAC&U News (December 2004)Sophomore writing portfolio requirement Students submit papers from at least 2 of 4

divisions Faculty ratings: “Exemplary” “Pass” “Needs

work”14% 78% 8% (resubmit)

Evaluation criteria:KEY WRITING SKILLS QUALITY OF WRITINGReport on observation Attend to audienceAnalyze complex information Clarity of proseProvide interpretation Clear organizationUse and document sources Use of evidenceArticulate and support thesis Distinctive voice

Appropriate dictionControl of error

LESSON 2MISSED OPPORTUNITIES

REVISITED

University of South Carolina – Aiken “Communicating Assessment Results Online to

PromoteCurricular Change” at SACS-COC 2005 Annual Meeting

Rising Junior portfolioWRITING OUTCOMES/EVALUATIVE RUBRICClarity of PurposeQuality of ThoughtOrganization of ContentUse of SourcesLanguage and StyleGrammar and Mechanics

LESSON 2MISSED OPPORTUNITIES

RECLAIMED?CARLETON COLLEGE LORAS COLLEGE USC AIKENAttend to audience Creativity/Voice/Audience Clarity of purpose

and purpose

Distinctive voice

Clarity of prose Demonstrates critical Quality of thought thinking

Clear organization Organizes the writing Organization of contentUse of evidence Supports ideas with Use of sources

evidence

Appropriate diction Uses standard English Language and styleControl of error and effective prose Grammar and

mechanics

EXAMPLE 3USING STANDARDIZED TESTS

• Loras College: Assessment of General EducationCAAP Tests in selected sophomore-level courses

– Reading

– Essay Writing

– Science Reasoning

– Mathematics

• Loras College: Senior Majors in Psychology

– ETS Major Field Test in Psychology

LESSON 3STANDARDIZED TESTS CAN BE

USEFUL(IF SELECTED & USED

APPROPRIATELY)• Comparisons to other institutions

Essay writing: Provided additional evidence for faculty concerns

Mathematics: Supported the math department’s recommendation for a curriculum change

Science reasoning & Reading: Added little of value

Psychology: Provided useful information concerning

the quality of content knowledge among majors

EXAMPLE 4FLORIDA’S ACADEMIC LEARNING

COMPACTS• 2004 Florida Board of Governors resolution to

implement Academic Learning Compacts for each

baccalaureate degree program at 11 universities

– Discipline-specific knowledge/skills

– Communication skills

– Critical thinking skills

• Each baccalaureate program must also

– Identify where and how students are assessed

– Provide students access to this information

– Monitor performance for program improvement

Implementing Academic Learning Compacts

• Review of past assessment plans• Develop flexible definitions of outcomes• Assist departments to

– Identify discipline-specific outcomes– Identify location of student assessments

• Courses (Core requirements; Gateway; Capstone)[Shared learning outcomes/syllabi/assignments]

• Standardized examinations for some programs– Develop methods to track student achievement

• Embedded questions on examinations• Scoring rubrics on shared assignments• Performance on standardized examinations

SCHOLARSHIP OF ASSESSMENT

• Presentations at conferences– Using Psychological Expertise in College-Wide

Assessment and Departmental Program Review presented at MACTOP ’97

– Assessing Classes, Courses, and Curriculapresented at MACTOP ’02, MISTOP ’01, ITOP ’00

– Aligning Assessment for Program Improvement with Accountability for Individual Student Learningpresented at SACS ’05; NCSU ’06 Assessment Institute

• Publications– Designing and Implementing Psychology Program

Reviews (2004 Measuring Up book chapter)– A Decade of Changes since the St. Mary’s

Conference (2002 Teaching of Psychology interview)

Accounting: Issues in Accounting EducationAnthropology: Anthropology and EducationArt: Art EducationBiology: American Biology TeacherBusiness: Business EducationChemistry: Journal of Chemical EducationCommunication: Communication Education Computer Science: Computer Science EducationEconomics: The Journal of Economic EducationEducation: Journal of Teacher EducationEnglish: Research in the Teaching of EnglishFinance: Journal of Financial EducationGeography: Journal of Geography in Higher EducationGeology: Journal of Geological EducationHistory: Teaching History - A Journal of MethodsInternational Business: Journal of Teaching in International Business

DISCIPLINE-BASED JOURNALS ON TEACHING PEDAGOGY

DISCIPLINE-BASED JOURNALS ON TEACHING PEDAGOGYJournalism: Journalism EducatorManagement: Journal of Management EducationMarketing: Journal of Marketing EducationMathematics: Journal for Research in Mathematics EducationModern Languages: Modern Language JournalMusic: Journal of Research in Music EducationNursing: Journal of Nursing Education Philosophy: Teaching Philosophy Physics: Physics TeacherPolitical Science: Political Science Teacher; Teaching Political Science Psychology: Teaching of Psychology Social Work: Journal of Teaching Social Work Sociology: Teaching Sociology Statistics: The American StatisticianTheater: Theater TopicsWomen's Education: The Feminist Teacher