Program Assessment

Professional Development Days August 25th and 26th 2009

Workshop Learning Outcomes

• Describe the purpose of program assessment• Outline the steps in program assessment• Write measurable program learning outcomes• Choose program assessment delivery points

Your worksheets will be collected at the end and returned to you next week.

What is the Purpose?

Program “assessment is a framework for focusing faculty attention on student learning and for providing meaningful discussions of program objectives, curricular organization, pedagogy, and student development.”

Mary Allen, Assessing Academic Programs in Higher Education

The Goal is Improvement

Program assessment concentrates on the group of students in a program.

To improve:CoursesProgramsServicesPlanningBudgeting

Last but not least - Accreditation

By 2012, the ACCJC expects us to:• Have SLOs for all courses, programs and the

institution. • Have assessed SLOs at all levels• Have reported and discussed the results and

used them to guide decision-making and improvements.

Assessment Levels

• Institutional outcomes should be reflected in the programs.

• Program outcomes should be reflected in the courses.

What is a program?

It’s up to the college to decide how to group activities for assessment purposes. Some common ones:

• Sequence of courses leading to Degree or Certificate• General Education• Basic Skills• Library, Learning Assistance Center, etc.• Special sequences for groups of students like PACE,

Honors, Extension• Student services (not addressed here)

By our normal way of defining “program”, there are over 100!

How can we get it all done?

One step at a time!1.Identify assessable programs2.Develop program learning outcomes 3.Develop an assessment plan4.Collect assessment data, analyze and report5.Use results to improve the program

This workshop concentrates on 1, 2 and 3

Identify Assessable Programs

For efficiency, create groupings of programs with similar goals and courses:– CSU General Education + Graduation Plan A +

Graduation Plan B + IGETC + Liberal Studies Degree + ?

– Business + Real Estate + Accounting– Industrial Engineering + Electronic

Engineering + Engineering Technology + ?– Is Political Science a program?

STEP

1

Identify Programs

Set up a focus group to work on the next steps. It could include:

• Division chair• Faculty point person• Full time and adjunct faculty• Students

STEP

1

Worksheet and Discussion

Take a few minutes to answer the Step 1 questions on your worksheet.

STEP

1

2. Write Program SLOs

A. Review LAHC institutional SLOsB. Look at course SLOs in your programC. Search the Web for outcomes to

“borrow”D. Describe the ideal graduateE. Check draft SLOs for measurabilityF. Build course to program matrix

STEP

2

Institutional SLOs (ISLO)

5 ISLOs (handouts and campus web site).

Modify them to be specific to your program. They don’t ALL have to fit but most will.

This makes it easier for us to conduct institutional assessment!

STEP

2A

ISLO #1 Communication

Use language and non-verbal modes of expression appropriate to the audience and purpose.

Example SLO for a Child Development program:

Teach and care for young children using effective communication and relationship skills.

STEP

2A

ISLO #2 Cognition

Use critical thinking skills to analyze, synthesize and evaluate ideas and information.

Example SLO for a Math program:Estimate and check answers to

mathematical problems in order to determine reasonableness, identify alternatives, and select optimal results.

STEP

2A

ISLO #3 Information Competency

Utilize research skills necessary to achieve educational, and professional objectives.

Example SLO for a History program:Use primary and secondary sources to

construct sophisticated, persuasive, and logical interpretations of historical problems and events

STEP

2A

ISLO #4 Social Responsibility

Demonstrate sensitivity to and respect for others and participate actively in group decision making.

Example SLO for a Computer Science program:

Analyze the global impact of computing on individuals, organizations, and society.

STEP

2A

ISLO #5 Personal Development

Demonstrate self-management, maturity, and growth through practices that promote physical, mental and emotional well-being.

Example SLO for a Human Services program:Identify their personal values and articulate

how their value systems inform and influence their professional practices.

STEP

2A

Look at the course SLOs

• Find repeating– Words– Activities– Ideas

• Course assessment forms are on the campus web site (Faculty/Staff, Student Learning Outcomes/Assessment, by department)

STEP

2B

Course SLO Word Cloud

History program word cloud created at wordle.net

Search the Web • Other institutions

Use www.google.com or www.bing.com to search. Search: history program learning outcomes

Example: University of Maryland http://www.ugst.umd.edu/core/LearningOutcome.htm

• Professional Organizations– Go to the site and search for learning outcomesACM (Computer Science)ABET (Engineering)MAA (Math)Warning: you will find bad SLOs and good ones.

STEP

2C

The Ideal Graduate

• Look at program review– Mission and goals

• Advisory committee members• Transfer institution contacts– Call or email and ask what the students

transferring in need to have– What do they find lacking?

STEP

2D

The Ideal Graduate Search the Internet or ask Bob Richards for help locating data.For students studying computers:

According to Kelly IT Resources, there are a number of technical skills currently in demand, including Java 2 Platform, Enterprise Edition (J2EE), Visual Java++, WebSphere, ERP, and CRM.

However, the IT worker who wants to get ahead needs more than just technical savvy. In a survey conducted by the Journal of Computer Information Systems, results indicated that attributes including, “interpersonal and personal skills/traits, such as team skills, communication skills, critical/creative thinking skills, and personal motivation” are often valued higher than the traditional technical requirements of a job.

From “IN-DEMAND IT SKILL SETS: WHAT EMPLOYERS WANT, WHAT WORKERS NEED” Published by Kelly IT Resources at http://www.kellyit.com.pr/res/content/us/kit/en/docs/kitr_in-demand_skillsets.pdf

STEP

2D

Check: Measurability

• Use active and specific terms. See Bloom’s taxonomy (in handouts)

• Don’t use words like “understand” – go for higher level skills

• The meaning of a word can be defined more clearly in the scoring rubric. For example, “sophisticated” from the history SLO.

STEP

2E

Check: Simplicity

• Keep the number down to around 5• Have more if your outcomes are

determined by an outside accrediting body like Nursing.

• Concentrate on the most important skills or knowledge the students will come away from the program with and what they will produce to show you that they have mastered those skills.

STEP

2E

Worksheet and Discussion

Take a few minutes to answer the Step 2 questions on your worksheet.

STEP

2E

Check Alignment with CoursesCurriculum Alignment Matrix

Course PSLO#1

PSLO#2

PSLO#3

PSLO#4

PSLO#5

PSLO #6

100 I I

120 I P

200 P P P

204 P

300 P P

329 D P

400 P D

480

490 D D D D

A matrix ensures that student have been exposed to the outcome in the program.

Can point out problems!

From Assessing Academic Programs in Higher Education by Mary J. Allen p. 43

STEP

2F

Program Assessment Form

– Blank form is a document on the campus Assessment Web site

– Download and type in your program SLOs with Word

– Send document to Division Chair who will forward to Dean and Assessment Coordinator for Web publication.

– Program Learning Outcomes are also to be published in the college catalog and all other program materials.

STEP

2

Develop Assessment Plan

A. Select delivery points for assessmentB. Select tool(s)C. Set timelineD. Create assessment and rubric

STEP

3

Delivery Point

Is there a course or set of courses all majors would take before graduating?

If not, is there a course many majors take? Give a survey to identify who is a major.

See Accounting program example on handouts.

STEP

3A

Delivery Point

If there isn’t a terminal course,• Can you reach students as they are

graduating to give out an assessment?– One campus had a free lunch for students

who filed petitions

• Can you have students build a portfolio as they progress through the program?

STEP

3A

Worksheet and Discussion

Take a few minutes to answer the Step 3 questions on your worksheet.

STEP

3A

Choose Assessment Tool(s)

• Direct Assessment• Indirect AssessmentExperts recommend more than one type of

data collection method!

STEP

3B

Choose Assessment Tool(s)Direct methods prompt students to represent of

demonstrate their learning or produce work so that observers can assess how well students’ texts or responses fit expectations

Includes:Standardized testsLocally developed tests and assignmentsProjectsPresentationsPortfolios

See handouts

STEP

3B

Choose Assessment Tool(s)

Indirect methods capture students’ perceptions of their learning and the educational environment that supports that learning.– Student surveys– Factbook information

Definitions of direct and indirect from Assessing for Learning by Peggy Maki p. 88

STEP

3B

Set Timeline

– Which semester/year will each SLO be assessed during?

– The general recommendation is to measure one per year to give time for careful review of results and integration into planning

– Send the timeline to the Division Chair and to the Assessment Coordinator

STEP

3C

Create Assessment and Rubric

– Develop the tool (test questions, assignment, survey, etc.)

– Create the scoring guide or rubric– Get input from all of the faculty who will be

involved.

STEP

3D

For Future Workshops?

• Designing assessments and rubrics• Collecting assessment data, analyzing

and reporting• Using results to improve the program

You tell us!

Thank You For Attending!

• Please turn in:The WorksheetThe Assessment Needs SurveyThe evaluation form

• Presentation will be available on the campus assessment web pages.

• Questions? – Lora Lane x4178 or campus email– Dean Humphreys x4025 or campus email