Assessing Student Learning Outcomes in Student Development – Part II

Student Development Division RetreatSUNY OneontaMay 27, 2008

Presenter

Patty Francis, Associate Provost for Institutional Assessment and Effectiveness

Topics/Activities for Today

Continued themes from May 9 division meeting

Developing a unit assessment plan Writing good outcome statements Measuring student learning outcomes Hands-on opportunity to develop outcome

statements/measures for department

Continued Themes from May 9 Meeting

Assessment Basics and Rationale

Assessment as an ongoing, iterative process which must culminate with “closing the loop”

Importance of congruence in the assessment process

Assessment as an opportunity for ongoing dialogue, professional development and, most important, improving student services

Developing a Unit Assessment Plan

Overall, Unit Objectives Would Reflect:

Institutional effectiveness performance indicators

Documentation of all services and programs offered

Tracking of use of services (and by whom) Student satisfaction with services/programs Direct impact of services/programs on

students

Writing Good OutcomeStatements

Issues to Address at the Outset

Consistency between college, division, and unit mission statement (and, ultimately, outcome statements)

Do all staff have the opportunity to provide input?

Who are all your constituents? What results do you expect your

programs and services to produce?

Components of a Good Outcome Statement

Who is the target? What domain of student development is

the target? What change is expected?

But Other Things to Keep in Mind

Do you have a program/activity in place to bring about outcome?

Can desired change in students be measured?

How will you know you were successful? Do external standards apply (i.e., in case

of external accreditation/certification)?

Outcome Writing Rule #1: Focus on Student, Not Program

Good example: “Student workers will identify, provide, and implement technical equipment that is appropriate for specific union events.”

Poor example: “College union work study program will teach student workers how to select and set up equipment for union events.”

Outcome Writing Rule #2: State Outcomes Using Concrete Language and Action Verbs

Good example: “Students will negotiate necessary academic accommodations with faculty members.”

Poor example: “Our objective is to enhance students’ independence and self-confidence.”

Outcome Writing Rule #3: Focus on Results, Not Process

Good example: “Students will demonstrate increased job search skills (e.g., letter and resume writing, interviewing, employer research).”

Poor example: “Students will participate in three 2-hour sessions on letter writing, interviewing, and employer research.”

Measuring Student Learning Outcomes

Basic Principles Measurement techniques must be rigorous,

since unreliable data are of minimal value Best to use variety of quantitative and

qualitative measuresQuantitative easier, but not often as richQualitative often more informative, but require

check on scoring (e.g., rubrics) Rely as much as possible on existing data

sources

Types of Information to Include

1. Normative/benchmark information whenever possible (external sources, own performance over time)

2. Locally collected data

Survey data SUNY SOS NSSE Local surveys of

student satisfaction/ perceptions

National benchmarking data (e.g., ACUHO/EBI, ACUI/EBI)

Performance-based data Focus groups

General Types of Measures (Maki, 2004)

Direct measures – students actually demonstrate learning so that evaluators can match results to expectationsStandardized testsAuthentic, performance-based – embedded

into students’ actual educational context Indirect – students’ perceptions of learning

Should never be used as sole indicator

From Measures to Criteria Assessment criteria reflect your

expectations about student performance (i.e., how you know you were successful)

Set criteria at reasonable but challenging levels

Often take these forms: “90% of students will …..”“80% of students will score at least 70% on…”

Authentic, Performance-Based Assessment: The Value of Rubrics

Rubrics provide reliable means of rating student performance in more qualitative way

Steps in developing rubricsUse entire staff to help develop and be as

specific as possible in differentiating between performance levels

Use existing rubrics as guidePilot test to assure scoring is reliable

Rubric Example #1: Recreation & Sport Services (Univ. of W. Florida)

Variable Fails to Meet Approaches Meets ExceedsRespect Makes negative comments

about others and recreation. Engages in disrespectful behavior during work and/or participation.

Keeps self from making negative comments about others and recreation. Sometimes engages in disrespectful behavior during work and/or participation activities.

Sometimes makes positive comments about others and recreation. Generally engages in positive behavior during work and/or participation opportunities. May engage in disrespectful behavior during work and/or participation activities but learns from the experience and corrects behavior.

Frequently makes positive comments about others and recreation. Generally engages in positive behavior during work and/or participation opportunities. Rarely engages in disrespectful behavior during work and/or participation activities and always learns from the experience and corrects behavior.

Rubric Example #2: Career Services (Interviewing)

(Univ. of W. Florida)Variable Fails to Meet Approaches Meets ExceedsAppearance Did not dress in proper

interview attire or was messy and wrinkled - wore shorts, flip flops, hat, t-shirt, or other apparel that was improper

Wore casual khakis or jeans with a semi-casual shirt or blouse. Candidate may have not worn tie or professional heels

Professional appearance, but had one or two items that stood out to be less professional - for example, too short of skirt

Wore exceptional interview attire - conservative suit, proper length skirt, well-pressed garments and overall - well groomed

First Impression

May have been late to the interview, appeared too casual or uninterested initially, provided poor handshake or greeting, etc.

Was on time to the interview, but offered poor greeting, handshake or appearance, etc.

Was on time to the interview; looked interested and had a professional appearance - may need to practice greeting or handshake

Was on time to the interview, offered a professional appearance, and gave a great handshake and greeting

Non-Verbal Behavior

May have used many distracting hand gestures or no eye contact - Candidate appeared completely uninterested or heavily slouched

May have used distracting hand gestures or may have slouched in the interview for approximately half of their interview time

Used some distracting hand gestures or may have slouched some of the time during the interview, but overall looked interested

Used appropriate hand gestures and displayed good posture & appropriate eye contact - generally illustrated that they were interested

Communi-cation

May have organized thoughts in an unclear way, used slang, or may have been too concise or brief with answers given

May need some work with organizing thoughts and using examples to strengthen issues discussed

Organizes thoughts well, uses proper grammar, but may still need to use examples

Gives solid introductions and conclusions to thoughts, expresses self in proper grammar, and uses examples

Congruence in the Assessment Process: A Detailed Example

Handout #1

Your Turn!

Handout #2