1

Annual Outcomes Assessment Report Workshop:

Academic ProgramsOffice of Institutional Planning, Research and Effectiveness (OIPRE)

October 6 – 8, 2015Drake LRC

2

Workshop Objectives

1. Provide a rationale for engaging in outcomes assessment2. Describe the concept of institutional effectiveness3. Outline steps in the outcomes assessment process4. Distinguish between student learning outcomes and program

outcomes5. Describe AAMU outcomes assessment process requirements6. Outline annual outcomes assessment report timeline

3

Rationale for Assessment

• Outcomes assessment is a formative process• Outcomes assessment is not an end; rather, it is a means for improving

student learning

• Motivation to engage in outcomes assessment• Intrinsic value• External accountability demands

4

Rationale for Assessment

• External accountability demands• Outcomes assessment is a requirement of our regional accreditor – SACSCOC• Without regional accreditation our students are not eligible for federal

student aid

• Intrinsic value• Outcomes assessment allows us to determine how well we are living up to

the implicit promise we make to our students when they enroll and where appropriate, make the necessary changes to curriculum and/or pedagogy to ensure students reach the expected performance levels

5

Institutional Effectiveness

• Institutional effectiveness is the systematic, explicit, and documented process of measuring institutional performance against mission in all aspects of an institution (p. 16).• A commitment to continuous improvement is at the heart of an on-going

planning and evaluation process. It is a continuous, cyclical process that is participative, strategic, flexible, relevant and responsive (p. 16).• This process [outcomes assessment] serves as the cornerstone of institutional

effectiveness. Institutional Effectiveness focuses on the design and improvement of educational experiences to enhance student learning (pp. 48 – 49).

Source: Southern Association of Colleges and Schools Commission on Colleges, Resource Manual for the Principles of Accreditation

6

University Mission StatementAlabama Agricultural and Mechanical University is a public, comprehensive 1890 Land-Grant institution, committed to access and opportunity, of knowledge

and excellence in teaching, research and service is responsive to the needs of a diverse student population and the social and economic needs of the state and region. The university offers contemporary baccalaureate, master’s educational specialist and doctoral level degrees to prepare student for careers in the arts, sciences, business, engineering, education, agriculture and technology. As a center of excellence, the University is dedicated to providing a student-centered

educational environment for the emergence of scholars, scientist, leaders and critical thinkers, who are equipped to excel through their contribution and leadership in a 21st century national and global society.

Strategic Priority 2: Strengthen Structures,

Operations, and Systems to

Promote and Support

Organizational Excellence and

Stewardship

Strategic Priority 6:

Enhance university Engagement

through Expanded Outreach

Strategic Priority 5:

Enhance the University’s Image and Recognition

Strategic Priority 4:

Secure the University’s

Financial Future

Strategic Priority 3:

Upgrade University

Infrastructure and Facilities

Strategic Priority 1:

Enhance AAMU’s Distinctiveness

and Competitiveness

PROGRAM 1MISSION

PROGRAM 2MISSION

PROGRAM 3MISSION

PROGRAM 4MISSION

PROGRAM 5MISSION

7

Institutional Effectiveness

• Comprehensive Standard 3.3.1 (Institutional Effectiveness – unit level)

The institution 1) identifies expected outcomes*, 2) assesses the extent to which it achieves these outcomes, and 3) provides evidence of improvement based on analysis of results.

*For academic programs this includes student learning outcomes.

8

Assessment Steps

1. Develop program mission statement2. Develop outcomes3. Develop appropriate assessment measures4. Develop desired performance targets5. Analyze results6. Develop and implement improvement plans

Mission Statement

Step 1

OutcomesStep 2

Assessment Measures

Step 3

Performance TargetsStep 4

AnalysisStep 5

Improvement PlansStep 6

Assessment Steps

10

Program Mission Statements

• Mission statements should…

• Be specific, distinctive• Clearly state the purpose of the academic program• Identify the program’s stakeholders• State the values and principles that guide the curriculum/program goals• Speak to what the program is currently doing• Align with AAMU’s strategic goals

11

Mission Statement Template

• The mission of <<enter program name>> is to <<program’s primary

purpose(s)>> by providing <<program’s primary functions or

activities>>.

12

Sample Mission Statements

• “The International Baccalaureate® aims to develop inquiring, knowledgeable and caring young people who help to create a better and more peaceful world through intercultural understanding and respect. To this end the organization works with schools, governments and international organizations to develop challenging programmes of international education and rigorous assessment.”• “The mission of the Department of Physician Assistant Studies is to

educate patient-centered medical providers of the highest quality who are critical thinkers, leaders, and life-long learners.”

13

Types of Outcomes

• Student learning outcomes (SLOs)• The knowledge, skills or abilities students acquire through the successful

completion of a course or academic program (what do you expect your graduates to know, think or be able to do upon graduation?)• Unit of analysis = students enrolled in your program

• Program outcomes• Illustrate what you want your program to do. These outcomes differ from

learning outcomes in that you discuss what it is that you want your program to accomplish (Bresciani, 2001)• Unit of analysis = the program

14

Outcomes and the Annual Assessment Report• For the annual assessment report, academic programs must report

on:

• Three (3) student learning outcomes

• Two (2) program outcomes

15

Writing Student Learning Outcomes

• SLO statements should…• Specify what students are expected to know, think or be able to do upon

program completion• Address only one outcome per statement• Be under the control/responsibility of the program• Be measurable• Yield non “Yes/no” results• Be at the program level

16

Student Learning Outcomes Template

• Graduates of the <<program name>> will be able to/will demonstrate

<<skill, knowledge, value>>.

17

Student Learning Outcomes Examples

• Graduates of the B.S. Nursing Program will have published at least one article or presented a paper at a conference. • The program has no control over

a student’s paper or conference presentation proposal being accepted.

• Graduates of the B.S. Nursing Program will be able to integrate evidence, critical thinking and patient preference in evaluating options for care.

18

Student Learning Outcomes Examples

• Graduates in Food and Beverage Management will appreciate the value of customer service.

• This outcome is not measurable.

• Graduates in Food and Beverage Management will demonstrate their understanding of the value of customer service.

19

Student Learning Outcomes Examples

• Graduates in Computer Systems will be able to develop web pages, write and debug programs in multiple languages, understand basic computer concepts, and be able to describe the impact of technology on society.

• This outcome is bundled – the statement addresses more than one outcome.

• Graduates in Computer Systems will be able to describe the impact of technology on society.• Graduates in Computer Systems

will be able to analyze technical requirements and select the most appropriate solution.

20

Student Learning Outcomes Examples

• Graduates in Social Work will have successfully completed an internship.

• This is a yes/no outcome that will not necessarily yield data for improvement. Also, completion of the internship is not under the program’s control.

• Graduates in Social Work will be able to apply the knowledge and skills of a generalist social worker.

21

Program Outcome Examples

• Program Outcomes measure things other than direct student learning

• Enrollment levels, # of graduates, course completion rates

• Alumni satisfaction, employer satisfaction, satisfaction with advising

• Licensure pass rates

• Job/graduate school placement rates

• Faculty productivity, research grant funding levels

22

Outcomes and the Annual Assessment Report• For the annual assessment report, academic programs must report

on:

• Three (3) student learning outcomes

• Two (2) program outcomes

23

Assessment Measures

• Now that you have developed sound outcomes, you must

• Indicate how the outcome will be assessed

24

Assessment Measures

• Assessment measures should be designed to yield meaningful,

actionable data that can lead to improvements.

25

Assessment Measures

• Should…• Be directly related to the outcome• Address all aspects of the outcome over time• Provide measurable results• Provide adequate data for analysis• Provide actionable data• Focus on student work at or near graduation

26

Assessment Measures

• Should…• Outline a systematic way to assess the outcome• Include the following details:• What will be collected• When the data will be collected• What assessment tool will be used• How will the data be collected

27

Choosing Assessment Measures

• Direct Assessments

• Are based on analysis of student work, performances or behaviors

• Can be• Quantitative – correct/incorrect (e.g., multiple choice, true/false, problem sets, etc.)

• Qualitative – requires judgment on the part of the evaluator (e.g., term paper, short answer/essay exam, project, performance, exhibits, etc.)

28

Choosing Assessment Measures

• Indirect Assessments

• Based on reported perceptions of student learning (e.g., student survey)

• Only appropriate as a secondary assessment method

29

Choosing Assessment Methods

• When assessing a Student Learning Outcome

• Multiple (at least two) assessment methods should be used where possible

• Direct assessments should be the primary assessment method utilized

30

Direct Assessment Measures

• Tests (completion, essay, matching, multiple choice, true-false)• Embedded assignments and course activities• Classroom assessment activities• Capstone projects (e.g., senior theses)• Homework assignments• Class presentations• Poster presentations• Recitals/exhibitions

• Portfolios

31

Grades as Assessment Measures

• While grades are appropriate as class assessment measures, they are not acceptable as program assessment measures.• Why is that?• Grades take the individual student as the unit of analysis, not the program• May include factors not related to student performance (e.g., attendance,

participation)• Don’t necessarily map neatly on to specific student learning outcomes and

therefore don’t lead to actionable data

32

Performance Targets

• Why do we need performance targets?

• Allows the program to identify desired performance levels

• Provide standard for determining success

• Puts data analysis in perspective

33

Performance Targets

• Targets should…

• Use component (not student or respondent) as unit of analysis

• Relate to outcome and assessment method

• Be specific (“x% of responses will be correct for each question” or “y% of

scores for each item on rubric or survey will be at least 4”

• Avoid vague words such as “most” or “majority”

• Avoid “all” or “100%” targets

34

Assessment Measures and Performance Target Example

• We will analyze final exams from the capstone course. 85% of our majors will pass the final with a score of at least 60%.• This measure uses students

rather than learning components as the unit of analysis. As such, the measure will not lead to data that can inform improvement plans.

• We will analyze final exams from the capstone course. 80% of the responses for each of the ten questions will be correct.

• In this revised measure and target, each of the ten exam items represents a learning component.

35

Assessment Measures and Performance Target Example

• We will keep a record of the number of majors who participate in internships. The criterion for success is that all majors will successfully complete an internship.• Example of a 100% target. Also

the measure – internship completion – is not under the program’s control.

• Each intern supervisor will be asked to complete an evaluation form that directly measures the intern’s performance.

At least 85% of the scores for each item on the evaluation form will be “satisfactory” or “highly satisfactory.”

36

Assessment Measures and Performance Target Example

• A random sample of lab reports from the sophomore lab courses will be graded using a rubric. The target is that 75% of lab reports will be judged as “satisfactory” or “excellent.”• The assessment measure is

conducted early in the program rather than near graduation. Also, the performance target focuses on the assignment grade rather than the learning components evaluated in the assignment.

• A random sample of lab reports from the capstone lab course will be graded using a rubric. The target is that at least 75% of scores for each item on the rubric will be “satisfactory” or “excellent.”

37

Assessment Measures and Performance Target Example

• A survey will be conducted of graduating majors during the spring semester. The average score on the question related to “overall satisfaction with the program” will be at 3.5 out of a possible 5 points.• This is an indirect measure (should

only be used in concert with a direct measure). Target uses averages which can be skewed by outlier scores. Instead we should focus on the proportion of students meeting the performance criteria.

• A survey will be conducted of graduating majors during the spring semester. At least 80% of the response for each question on the survey will be “satisfied” or “highly satisfied.”

38

Analysis of Results

• Here we move from planning the assessment to conducting it

• Provides data on which you can base changes

39

Analysis of Results

Analysis should…

• Provide detailed data (avoid “a majority” or “most”)• Use specific numbers (no rounding)• Avoid technical language• Be clearly and succinctly presented• Align with outcome and target• Support actions taken later to foster improvement

40

Improvement Plans

• Should…

• Address outcomes where the performance target was not met

• State what future actions will take place in the future to address the outcome

41

Types of Improvement Plans

• Curriculum revision• Additional assignment• Course modification• Instructional methodology

change• Sequence change

• Technology update• Assessment method change• Target adjustment• Faculty development• Procedure, process change

42

Example for Discussion

Student Learning Outcome

Biology graduates will demonstrate an understanding of cell structure and function.

43

Example for Discussion

Assessment Measure• Students in BIOL 432 will submit a research project based on cell

structure and function. A faculty panel will use a rubric that has a scale of 1 to 5 (where 5 is excellent) to assess the five components of the project. Panel scores will be analyzed by each component across all student work.

44

Example for Discussion

Target• 80% of scores provided by the faculty panel for each of the five

components of the Biology Project Rubric will be 4 (very good) or 5 (excellent).

We will revisit this in a few slides.

45

Example for Discussion

Biology Project Results

Component # Scores 4 or 5 Total # Scores % Scores 4 or 5

Sources 68 72 94.4%

Methods 45 72 62.5%

Analysis 67 72 93.1%

Conclusion 48 72 66.7%

Organization 59 72 81.9%

46

Example for Discussion

Analysis

The 24 biology projects were assessed by a three-member faculty panel for each of the five components of the rubric and for overall performance. The target was not met for the Methods (62.5%) and Conclusion (66.7%) components of the projects.

47

Example for Discussion

Improvement PlansFewer than 80% of scores provided by the faculty panel were 4 (very good) or 5 (excellent) for the Methods and Conclusion components of the Biology project. The department will develop a short video detailing the process of writing a methods section and make this available to BIOL 432 students through the course Blackboard site. A new assignment requiring students to critique the conclusion sections of sample lab reports will also be introduced in BIOL 432.

48

Annual Assessment Report TemplateUniversity Mission: Alabama Agricultural and Mechanical University is a public, comprehensive 1890 Land-Grant institution, committed to access and opportunity, and dedicated to intellectual inquiry. The application of knowledge and excellence in teaching, research and service is responsive to the needs of a diverse student population and the social and economic needs of the state and region. The University offers contemporary baccalaureate, master’s, educational specialist and doctoral level degrees to prepare students for careers in the arts, sciences, business, engineering, education, agriculture and technology. As a center of excellence, the University is dedicated to providing a student-centered educational environment for the emergence of scholars, scientists, leaders and critical thinkers, who are equipped to excel through their contributions and leadership in a 21st century national and global society.

Program Purpose: Step 1 Student Learning Outcomes

Assessment Methods Performance Criteria

Performance Target

Assessment Results Improvement Plans Improvements Implemented

Step 2

Step 3 Step 4 Step 5 Step 6 Step 6

Program Outcomes Assessment Methods

Performance Criteria Performance Target

Assessment Results Improvement Plans Improvements Implemented

Step 2

Step 3 Step 4 Step 5 Step 6 Step 6

49

Assessment Dates

• Assessment Plans Due• Steps 1 – 4• Mission, outcomes, measures and targets• Friday, November 13, 2015

• Assessment Reports Due• Steps 5 – 6• Findings, improvement plans• May ??, 2016 (specific date to be determined)

50

OIPRE Contact Information

Office of Institutional Planning, Research and Assessment306 Patton Hall

James Walke, Directorjames.walke@aamu.edu 256-372-8876

Dianne Bozeman, Assessment Coordinatordianne.bozeman@aamu.edu 256-372-8644