Non-Academic Assessment: Completing an Assessment Report

OFFICE OF ASSESSMENTJULY 12, 2012

Non-Academic Assessment: Completing an Assessment

Report

Goals of Workshop

Current assessment contextWhat’s new this yearIntroduce new Higher Learning Commission

CriteriaReview items on the Assessment Report

templateWalk through new online reporting form

2

Assessment Context3

Assessment Context

Assessment Context Last year participating in the Higher Learning

Commission’s (HLC) Academy for the Assessment of Student Learning

New HLC Criteria effective January 1, 2013 Campus focus on 2020 Strategic Plan alignment

Assessment Reporting Themes Focus on use of data Reflect on changes and impact on learning Streamline reporting process

Future Goals Longer reporting cycle Integration with annual reporting

4

What’s New This Year

Online reporting Fewer questions Questions “tagged” with New HLC Criteria Optional question on New HLC Criteria Report quality threshold to 70%

5

HLC New Criteria

Guiding Values, includes:1. Focus on student learning4. Culture of continuous improvement5. Evidence-based institutional learning and self-

presentation9. Mission-centered evaluation

Anatomy of Criteria Criterion—broad statement

Core Component—areas of particular focus Sub-Component—further elaboration on Core

Component, but not exhaustive (and not always included)

Some criteria now specifically mention certain units (e.g., advising)

6

HLC Criteria7

HLC Criterion One. Mission

The institution’s mission is clear and articulated publicly; it guides the institution’s operations. Core Component 1.A.

The institution’s mission is broadly understood within the institution and guides its operations. Subcomponent 2.

The institution’s academic programs, student support services, and enrollment profile are consistent with its stated mission.

Criterion Title and

broad statement Core

Component statement—particular aspect of

criterion on which to

focusSub-component—specific area from

which evidence is expected

HLC New Criteria

The Five Criteria1. Mission2. Integrity: Ethical and Responsible Conduct3. Teaching and Learning: Quality, Resources, and Support4. Teaching and Learning: Evaluation and Improvement5. Resources, Planning, and Institutional Effectiveness

Assumed Practices Won’t check, but we are responsible to ensure/remediate Notable is B.2.c.1): Faculty participate substantially in:

analysis of data and appropriate action on assessment of student learning and program completion.

8

Upcoming Assessment Workshops

Developing an Assessment PlanThursday, July 19th, 10-11 am

Completing the Assessment Report Tuesday, July 17th, 1-2 pm

Note: workshops/forms overlap

9

Completing an Assessment Report

10

Unit Information

1. Explain how your unit supports 2020 Strategic Plan of YSU.

Removed mission; strategic plan operationalizes mission

11

HLC Criterion One. Mission The institution’s mission is clear and articulated publicly; it guides the institution’s operations.

Core Component 1.A. The institution’s mission is broadly understood within the institution and guides its operations.

Sub-component 2. The institution’s academic programs, student support services, and enrollment profile are consistent with its stated mission.

HLC Criterion Three. Teaching and Learning: Quality, Resources, and Support The institution provides high quality education, wherever and however its offerings are delivered.

Core Component 3.E.The institution fulfills the claims it makes for an enriched educational environment.

Sub-component 1. Co-curricular programs are suited to the institution’s mission and contribute to the educational experience of its students.

YSU’s 2020 Strategic PlanYSU’s 2020 Strategic Plan

Office of Assessment (OOA)Office of Assessment (OOA)

The new University-wideStrategic Plan is organizedaround four cornerstones Student success The urban research

university mission/transition

Regional impact Institutional accountability

and sustainability

Two goals of the OOA (to increase campus engagement in assessment and to successfully demonstrate progress in the HLC Academy for the Assessment of Student Learning) are directly tied to 2 cornerstones of the 2020 Plan

Accountability and Sustainability which emphasizes shared responsibility.

Student Success since it documents student achievement in curricular and co-curricular areas.

The new Engagement in Assessment Portfolio will provide documentation.

Example: Strategic Plan Link12

YSU’s 2020 Strategic PlanYSU’s 2020 Strategic Plan

Office of Community InvolvementOffice of Community Involvement

Theme 2.2. Satisfaction with non-

academic experiences and campus life-including the importance of participation in service learning and community activities

The main goal of this unit is to increase participation in community activities as well as meaningful reflection on the impact of those activities on students’ development

Example: Strategic Plan Link13

Unit Information

2. Describe the goals for your unit from the previous academic/fiscal year

Example (from Office of Campus Planning and Community Partnerships):

to annually update the Campus Master Plan and undertake such studies as may be required to implement specific recommendations in accordance with the University’s annual capital improvement program;

to complete the Campus Roadway Study initiated by the City of Youngstown and focused upon the major arterial roadways serving the campus;

to implement improved bus service to the campus in collaboration with the WRTA and the University’s Alternative Transportation Community; and

to collaborate with Student Government to develop a matrix of student volunteer requirements and opportunities.

14

Learning Outcome Focus

3. State at least one learning outcome that is relevant

for your unit Keep to 1 (maybe 2 LO’s) Keep to 1 focused project (not a laundry list) What are you interested in?

15

HLC Criterion Four: Teaching and Learning: Evaluation and Improvement The institution demonstrates responsibility for the quality of its educational programs, learning environments, and support services, and it evaluates their effectiveness for student learning through processes designed to promote continuous improvement.

Core Component 4.B. The institution demonstrates a commitment to educational achievement and improvement through ongoing assessment of student learning.

Sub-component 1. The institution has clearly stated goals for student learning and effective processes for assessment of student learning and achievement of learning goals.

Developing Learning Outcomes

Sources for LO: A particular project that your unit will be

focusing on in 2011-12 ORLO that staff from your unit develop ORYSU’s General Education learning outcomes ORThe Council for the Advancement of Standards

in Higher Education (CAS) General Standards OR

Learning Reconsidered

16


Developing a learning outcome:Framing Question: What do you want

students/staff/faculty to know and be able to do as a result of interacting with your office or program you offer?

Should be SMART: Specific, Measurable, Attainable, Realistic, Timely

Two common methods to writing a learning outcome: ABCD Model SVP Model

17

Learning outcome material excerpted from GLACUHO Webinar on Writing Learning Outcomes by Jeff Long, May 2010


ABCD Model of Learning OutcomesAudience-target of the outcomeBehavior-verb in future tenseCondition-activity causing learningDegree of Achievement (optional)

________ ________ ________ ________ Condition Audience Behavior

Degree

18


ABCD Model Examples________ ________ ________ ________

Condition Audience Behavior (Degree-optional)

As a result of student staff training, student staff will be able to list (4 of the 5) department’s values.

Through publicity and training, staff will correctly complete online timesheets (90% of the time).

As a result of this workshop, attendees will explain the components of a well-written assessment report.

19


SVO Model of Learning OutcomesSubject = learnerVerb = cognitive processObject = content/knowledgeNote: removes condition (e.g. specific intervention);

somewhat more like academic learning outcomes.

Examples:Students will identify academic resources on

campus.Faculty will use DARS system in advising.

20


Using Verbs in Learning OutcomesAvoid “know,” “understand,” “think,” etc. –

how will you measure?Consider the level of learning you expect

from your students/staff/faculty—do you want them to comprehend? Apply knowledge?

Verb list that considers the level of learning (aka Bloom’s Taxonomy)--very helpful!: http://www.clemson.edu/assessment/assessmentpractices/refe

rencematerials/documents/Blooms%20Taxonomy%20Action%20Verbs.pdf

21

Example: Learning Outcomes

Academic Advisors Learning Outcomes Example:

1.Students can independently seek out resources to aid in evaluating their progress toward degree completion

2.Students will be knowledgeable of the curriculum requirements for their chosen major

22


4. How does this learning outcome fit your unit? Why did you pick this LO?

Example (for fictional Office of Community Involvement):

This is directly related to our unit’s purpose to

promote the benefit of community involvement

to our students

23


5. How does this learning outcome fit in with the YSU

2020 Strategic Plan? Consider how the learning relates to the larger goals of the

university

24

HLC Criterion One. Mission The institution’s mission is clear and articulated publicly; it guides the institution’s operations.

Core Component1.A. The institution’s mission is broadly understood within the institution and guides its operations.

Sub-component 2. The institution’s academic programs, student support services, and enrollment profile are consistent with its stated mission.

HLC Criterion Three. Teaching and Learning: Quality, Resources, and Support The institution provides high quality education, wherever and however its offerings are delivered.

Core Component 3.E.The institution fulfills the claims it makes for an enriched educational environment.

Sub-component 1. Co-curricular programs are suited to the institution’s mission and contribute to the educational experience of its students.


Learning Outcome Relation to Unit Relation to YSU’s Strategic Plan

As a result of a unit publicity campaign, students will be more aware of the importance of community involvement

This is directly related to our unit’s purpose, to promote the benefit of community involvement to our students

Promoting the importance of community involvement helps to encourage participation, and the Regional Engagement Cornerstone outlines the importance of student involvement in the community to support quality learning experiences and positive community change (in particular Theme 1, Priority 1).

Staff demonstrate awareness of professional responsibilities of students serving in community organizations

By ensuring our staff are responsive to the needs of the community organization with which we partner…we are well positioned to expand our community service opportunities

By maintaining positive and responsive relationships with our community partners, we enable students to interact positively with community partners, and foster the collaborative relationships with community organizations outlined in the Regional Engagement Cornerstone (in particular Theme 1, Priority 1).

Example, Questions 3-5—how do they align?

25

Assessment and Evaluation of LO Data

6. Provide a summary of LO-related data collected for the previous academic/fiscal year (i.e., explain how you measured the LO-related activities). Include methods used and number of individuals

assessed Minimum of one tool, but we recommend no more

than two Have one direct measure to receive an “exemplary”

score

26



Sub-component 2. The institution assesses achievement of the learning outcomes that it claims for its curricular and co-curricular programs.

Direct MeasuresDirect Measures Indirect MeasuresIndirect Measures

Direct evidence of learning outcome performance:

Factual knowledge questions embedded in a survey

Presentations Performances Simulations Internship supervisors’

evaluations

Self-reported achievement or proxy evidence:

Surveys Group discussions Focus groups Exit interviews Employment

placement rates

Examples: Direct vs. Indirect Measures

27

Methods: Both Direct and Indirect

Method Indirect Direct

Minute paper after a workshop on diversity

Perceptions of moral dilemmas regarding diversity

Factual question on the definition of diversity

Telephone calls to a Unit

Caller’s satisfaction with technology

A qualitative analyses of callers knowledge of technology based on their questions

Survey after a training Teacher and learner satisfaction

Factual question on knowledge of workshop content

28

Tools may be direct or indirect, depending upon how they are used and their purposes:

Example: Data Summary

Rate of Participation (from Housing and Residence Life):

Housing & Residence Life participated in the national EBI Resident Study. 618 of our 745 residents (83%) of our residents participated. This was double the rate of participation from the previous year, before “Love Where You Live” was established.

29

Example: Data Summary

Results (from Housing and Residence Life):From EBI Resident Study:excellent scores and showed marked

improvement in both factor means and national rankings.

Of 19 factors assessed in the EBI Resident Study, YSU was ranked in the top 100 of 300 participating schools in seven factors, including overall program effectiveness.

More important than rankings, our improved scores and participation reflected an increased investment in our community by our residents.

30


For questions 7-9, please provide an evaluation of the 2011-2012 data you collected related to your unit’s learning outcome.

31



Sub-component 3. The institution uses the information gained from assessment to improve student learning.


7. Based on the data, what is one key strength of (student, staff, or faculty) learning?

8. Based on the data, what is one important challenge to (student, staff, or faculty) learning?

Strengths/challenges of learning, not of your unit You can bring in other sources of information, but be

sure that the conclusions you form are based on the data you’ve collected

32

Example: Data Evaluation

Key Strength Example (from Housing):

Our positive assessment results tell us that our campaign is making a difference—students are more aware of our staff’s efforts to create a more conducive living space, students are more intentional about maintaining a strong community and are more respectful of their fellow residents.

33

Example: Data Evaluation

Challenge (from Housing):engaging our residents to remain actively

involved in the residence hall and University community.

generating new ideas that keep our programs evolving.

increasing faculty awareness about our programs & incorporating more faculty into our programs.

34


9. What action step has been identified based on this evaluation of the data? What is the status of this action step?

Considering the evaluation you conducted in questions 7 & 8, what could be changed in order to improve the learning outcome you are examining?

Data may not indicate the need for change; this is fine. Include evidence for your unit’s conclusion based on the data.

35

Example: Action Step

Action Step Example (from Housing):

The Director of Housing & Residence Life conducted Speak Out Be Heard meetings in each residence hall to assess what University Housing could do to increase the desirability of student housing. Many of the comments made in those meetings became improvement projects and several have already been completed.

36

Relation to New HLC Criteria

10. (OPTIONAL FOR 2012) Please provide an example of evidence from your unit for the new HLC Criterion Five (Resources, Planning, and Institutional Effectiveness), Core Component 5A (The institution engages in systematic and integrated planning.), Sub-component 2 (The institution links its processes for assessment of student learning, evaluation of operations, planning, and budgeting.): Can help to begin familiarizing yourself with new HLC Criteria How will assessment activities you are planning impact your

planning and budget decisions? How does your work help contribute to YSU’s body of

evidence for this criterion?

37

Example: HLC Criteria

Aligning Assessment with Budget Example (from fictional Office of Community Engagement):

While we didn’t think of it this way before reading this question, the 2010 survey results made it clear that more staff and financial resources needed to be devoted to a campus campaign. When planning for the 2011-12 fiscal year, funding was allocated specifically to the student awareness campaign, and staff responsibilities were arranged to allow for sufficient attention to outreach activities during the year.

38

Use of Data

11. Feedback Loop: How and when are both internal and external stakeholder involved in the assessment process (e.g., data sharing and recommendation gathering)? share with groups outside your unit use active ways rather than passive ways to share findings seek recommendations from groups inside and outside the

unit

39



Sub-component 4. The institution’s processes and methodologies to assess student learning reflect good practice, including the substantial participation of faculty and other instructional staff members.

Example: Feedback Loop

Feedback Loop Example (from fictional Office of Community Engagement):

We compiled the results from our survey and shared the information by: meeting monthly with internal staff to present data and solicit recommendations; soliciting a story about the results in the Jambar for students in March; and inviting other community service-oriented campus units (e.g., University Honors, WYSU, Alumni Services) to come in person for a presentation and discussion of the data on April 17th.

40

Use of Data

12. Considering action steps from previous years (2008-2011), what has been an impact on student learning as a result of (one of) those action steps? You can refer to previous reports for action steps Only one specific example is needed In particular, we are interested in changes you made

previously that have since impacted student/faculty/staff learning

41



Sub-component 3. The institution uses the information gained from assessment to improve student learning.

Example: Previous Action Step Impact

Impact of previous action steps from fictional Office of Community Engagement:

The results of the 2010 student survey made it apparent to us that there was a need to increase student awareness about the opportunities for and importance of community service activities as a student. We decided that for our action step we would undertake a campus campaign to increase awareness; the campaign was completed in February 2012. The results of the survey repeated in spring 2012 shows that students had an increase in the perceived value of participation—we see that as a positive impact of the campaign and a change in the students’ learning.

42

13. How did your unit utilize feedback from the OOA regarding last year’s assessment report?

o If this is your first time completing a plan or report, answer “first reporting year”

o This is an important part of completing a full assessment cycle/spiral/loop

Use of Data43

Assessment process image credit: http://pdx.edu/studentaffairs/assessment-planning-and-practice.

Example: OOA Feedback

OOA Feedback Example (from Academic Advisors):

Based on feedback from OOA to develop a LO that could lend itself to direct method of measure, added a LO that included: Students will be advised in a timely manner Student will correctly identify their advisor (is being

considered for the future)

44

Conclusion

14. Executive Summary: Summarize the key ideas from your report

one paragraph is fine a maximum length of one half page allows us to compile and develop themes from 40+

units

45

Conclusion

15. (Optional) Is there anything else you would like to share regarding your assessment report and/or is there any particular area on which you would like assistance or feedback?

This is a chance to share something that didn’t “fit” with our questions but is important to your assessment “story”

Or, it is something about which you could use our help or advice

46

Example: Additional Information

We would love some examples of rubrics that have already been developed to measure community engagement. Do you have some instruments to share with us

We would like to use the institutional account for survey monkey. How do we go about learning more about using this survey?

47

Example: Additional Information

We need help in having a forum to share our data from our prior year’s report. Can you help us set up a workshop or forum?

We are not certain how to incorporate a direct measure with this learning outcome-do you have any ideas

48

New Online Reporting Format

Online Assessment Report Submission Form: www.jotform.us/ysuassessment/2012nareport

49

Thank you for your participation!

50

Documents

Non-Academic Assessment: Completing an Assessment Report