1

October 2016

Office of Institutional Research and Assessment

106 Holy Family Hall

Chad May, MS Director

Lisa Belfield, Ed.D. Associate Director

Kymber Taylor, MS Research Analyst

Cher Cheryl Glover, Research Assistant Research Assistant

You’ve got questions? We’ve got data! By Chad L. May, Director In this issue of Assessment Now, you will read about the new course evaluation system (EvaluationKIT), some preliminary results of our administration of the Cooperative Institutional Research Program first-year student survey (CIRP), reflections of the recent Drexel Assessment conference, some upcoming assessment conferences, and our fall Lunch & Learn series. As you can see, the office remains engaged in helping the campus continue along with path towards cultivating a culture of evidence. The other side of the office deals with compliance; both compliance with our regional accreditor Middle States Commission on Higher Education as well as the US Department of Education. The number of external data collection surveys over the last five years that we have responded to have increased by 144%, more than doubling in number and many of those are mandated; however, we have seen an increase in the number of voluntary ones as well. Note these numbers don’t include the extensive report we prepared on compliance with federal regulations required as a supplement to our Periodic Review Report. The other large stream of work within the office has to do with constituent surveys from the first-year student survey, accepted student surveys, evaluation of orientation, to the standard exit, one-year out, and current student satisfaction survey. We have a growing number of departments and programs with whom we support and assist in surveying a broad array of constituents, you see those activities have grown by 114% over the last five years, again more than doubling. Note, we suspect those to increase beyond what is planned as needs arise through this academic year. We want the community to know that if you have a need or interest in surveying students, faculty, staff, please contact our office and we can help you either setup your own survey or engage us to conduct the survey on your behalf. This is a service we want to grow and encourage the campus community to join with us as we turn data into action. We, of course, in the Office of Institutional Research and Assessment wish the entire campus community the best as we enter now the fourth/fifth week of the fall semester as we all settle in for the new academic year.

Assessment Now Newsletter of the Office of Institutional Research and Assessment

Inside This Issue

Introducing EvalKIT Page 2

“Hello Class of 2020”

Page 3

Drexel Assessment Conference Reflection

Page 4

HFU Students out-vote other universities

Page 5

Lunch & Learns October 5 & October 26

See Page 8 for details

2

Introducing EvaluationKIT By Lisa Belfield, Ed. D, Associate Director Holy Family University introduced a new course evaluation system over the summer that will provide us with more efficient, effective, and robust course evaluation data. We contracted with EvaluationKIT (EvalKIT) to adapt their software to the needs of our students, faculty, and administrators. Over the summer we piloted EvaluationKIT and have fully implemented it for the fall semester. The new course evaluation software includes new tools and features that can support faculty, administrators, and students in the course evaluation process. EvalKIT can be accessed directly through CANVAS, each term faculty will receive an e-mail with a unique link when their evaluation results are ready. Faculty can view results for the courses in which they taught, as well as relevant comparison reports for the department/school, and for the university overall. Summary reports provide comparative information so that course instructors can draw meaningful conclusions about their students’ experience compared to some institutional norms. As with the previous course evaluation software, summary reports will give you means and frequencies across all of the questions on the form; however, you have the following additional capabilities in EvaluationKIT:

You will be able to combine evaluation results across classes (e.g., look at all gateway courses as a group, look at specific question groups across classes)

You can select or deselect any question or

groups of questions. The standard three

question groups pertain to course content, student contributions, and instructor contributions (instructor 1 and, if applicable, instructor 2)

You can combine evaluation results across

terms in order to analyze results for multiple terms, look for trends, and patterns in the data. You also have access to the raw data which can be downloaded in Excel.

Custom questions which are specific to a

course or a program can be added to an evaluation by the instructor and/or the dean

Training for deans was held in August, and an FAQ is being developed for faculty who have questions about the new system. Course instructors and academic departments have diverse reporting needs and preferences; therefore, depending on individual school needs, we may hold mini-training sessions at various points in the fall and spring semesters. The Office of Institutional Research and Assessment will oversee the system. We want the evaluation system to support your assessment work, and we look forward to collaborating with you on this venture. It may take a bit of time to get used to a new system, especially since you have more options and capabilities in EvalKIT. Please do not hesitate to contact Lisa Belfield in the Office of Institutional Research & Assessment at x3621 or lbelfield@holyfamily.edu if you need a hand.

3

Hello Class of 2020: Using CIRP Surveys to learn more about our First-Year Students By Kymber L. Taylor, Research Analyst During the very first week of class, Holy Family University’s First Year students participated in the national CIRP Freshman Survey, joining the ranks of over 15 million students that have completed the survey since it was first introduced in 1966. The Cooperative Institutional Research Program (CIRP) is now administered by the Higher Education Research Institute at UCLA (since 1973) and is the largest and oldest empirical study of higher education in the United States, consisting of a suite of five surveys: (1) The CIRP Freshman Survey, (2) Your First College Year, (3) Diverse Learning Environments, (4) College Senior Survey, and (5) HERI Faculty Survey. And while the core of the freshman survey remains unchanged, numerous questions have come and gone. In 1967, for example, First Year students were asked if they could perform the following items: “break 100 in golf, sail a boat, make a dry martini, or bake a cake from scratch.” While these items are no longer included (thank goodness, I do not do any of those well!), newer items have been added to the mix. In 2014, for example, students were first asked to estimate the likelihood that they will vote in a local, state, or national election. Of note, nationally, close to

50% of students in 2014 said there was a “very good chance” they would vote. In 2015, close to 60% said there was a “very good chance” they would vote. However, this year, our preliminary results indicated that only 37% of our first-time first-year students said there is a “very good chance” they will vote. Once the final and comparative data are available, it will be interesting to see if this reflects a national decline in interest in voting. But note that these surveys are not limited to identifying national trends; they also can be used at the campus level to better predict retention and completion, as well as to inform and assess institutional strategic goals. This past July I attended the CIRP Summer Institute at UCLA to learn more ways Holy Family can use the suite of CIRP/HERI surveys. The Institute began with lectures and hands on analytical exercises in regression modeling, and concluded with each participant presenting their own mini research study. Results from the Freshman Survey will be available in January 2017, but in the meantime feel free to stop by our office to learn more about national or historic trends.

2016-17 Academic Year Survey Schedule

Survey Who takes it? When? The Freshman Survey (administered by HERI/CIRP) First year students August 2016

Faculty Survey (administered by HERI/CIRP) All full time and part time faculty October 2016

Current Student Satisfaction Survey All students November 2016

Exit Survey (December) All students graduating December 2016 December 2016

College Senior Survey (administered by HERI/CIRP) Graduating Undergraduate Students January 2017

Your First College Year (administered by HERI/CIRP)

First year students April 2017

Exit Survey (May and Summer) All Graduating Students May 2017

One Year Out Survey All graduates of 2016-16AY June 2017

Five Year Out Survey All graduates of 2011-12AY June 2017

4

Drexel University Assessment Conference

Conference Reflection By Lisa D. Belfield, Ed.D. Associate Director

From September 7th-9th, over 400 attendees from 28 states and 5 foreign countries [Israel, Pakistan, Canada, Andorra and Morocco] gathered in Philadelphia, PA for Drexel University’s third annual assessment conference. Guided by the conference theme, Academic Quality: Driving Assessment and Accreditation, participants embraced the opportunity to restate their commitment to academic quality, and acknowledge that the quality of the student academic experience must be the driver behind all assessment and accreditation activities. By my account, and I am sure others would agree, the conference was successful in providing three-days of thought-provoking speakers, workshops, and invaluable networking. Many of the workshops I attended were facilitated by college and university representatives. Most were faculty members who self-describe as “somehow became involved” in assessment activities at their institution. All were faculty and administrators who wanted to see improvement in student learning. One of the best sessions I attended demonstrated questioning strategies and activities to engage students in deeper conversations. In this interactive session, after the facilitator from D’Youville College explained the rationale and some cueing and pacing strategies, workshop attendees assumed the role of students and participated in class discussions. Participants came away from the workshop with knowledge of how to use observational and tangible records from questioning as evidence of student learning and to inform instruction. A ‘snapshot’ session I attended focused on student-driven assessment efforts on campus. We often, very often, survey students to try to make sense of student-related issues on campus. The presenter from Flagler College remarked on how, in many cases, students are best equipped to help us gather true insights from fellow students. Students can play an active role in leading assessment efforts.

Attendees at one of the conference sessions

Engaging students to lead survey design, focus group conversations, and presentation of findings can make the conversations even more meaningful and useful. Student-led surveying and focus groups can be a powerful tool in increasing assessment buy-in and quality. This was a very interesting idea and one that perhaps can be explored further here at Holy Family University. One of the many benefits of involving students in the assessment process is that students develop an understanding of the desired outcomes and success criteria. Finally, throughout the conference we were reminded that discussion about assessment should happen often and can help to clear up rumors and frustrations. This discussion should be happening in your department meetings, among faculty, among administrative units, and at as many venues as possible. Assessment should be a culture that unites us, regardless of discipline, department, or unit. The conference solidified in my mind that we are on the right track, but that we still have our work cut out for us to truly create a culture of evidence. Holy Family University was well-represented at the conference as five faculty and staff members presented three different sessions on assessment. You can sit in on a snapshot of those presentations at our upcoming Lunch & Learn on October 5th, read more about it on the back cover of the newsletter and reserve your spot!

5

HFU students out-vote students from other universities By Kymber L. Taylor, Research Analyst How engaged are Holy Family students in the local political process? Are they registered to vote? Do they actually show up on Election Day? And how do our students compare to students at other institutions? To find these answers, Holy Family University joined Tufts University’s National Study of Learning, Voting and Engagement (NSLVE), which compiles student voter and registration data. The study examines and matches enrollment records submitted to the National Student Clearinghouse, public voting records compiled by Catalist, and data reported to the Integrated Postsecondary Education Data System (IPEDS). Since 2013, when the study first launched, over 900 campuses from all 50 states have participated, and nearly 8 million records been compiled. Participating institutions then receive custom campus reports that give comparison data of student voting rates as well as breaking down the voting rates of students by age, education level, gender, race/ethnicity, field of study and voting method. See below for a brief summary of HFU student rates. The full report is available on our Canvas site, OIRA Internal Resources Page, titled NSLVE Campus Report.

6

Upcoming MSCHE Events This Fall in Philadelphia

Creating an Efficient Assessment Structure That Drives Impactful Change October 11, 2016 - Courtyard by Marriott Philadelphia Downtown - Philadelphia, PA Facilitators: Dr. Marc Gillespie, Professor, and Dr. Anthony Marziliano, Associate Director, Assessment, Saint John's University Early Registration, $325.00 until September 20. For more information visit http://www.msche.org/?Nav1=EVENTS&Nav2=2016.05.08

Assessing and Improving the Effectiveness of General Education Programs October 14, 2016 - Sheraton Philadelphia University City Hotel - Philadelphia, PA Facilitator: LTC Christopher Mayer, Associate Dean for Strategy, Policy, and Assessment, United States Military Academy. Early Registration, $325.00 until September 20. For more information visit http://www.msche.org/?Nav1=EVENTS&Nav2=2016.05.07

Annual ConferenceDecember 7-9, 2016 Philadelphia, PA

As the Commission moves toward the full implementation of its new accreditation standards, works with member institutions on the details of new accreditation processes and a new accreditation cycle, and faces a changing higher education regulatory landscape in Washington, this year's annual Conference provides member institutions with an invaluable opportunity to learn about the challenges they will face.

The pre-conference workshops and the conference's concurrent sessions will examine such issues as

assessment, strategic planning, budgeting, governance, the self-study process, substantive change, compliance with federal regulations, and much more. Among the concurrent sessions will be presentations by institutions that have been involved in the Commission's Collaborative Implementation Project, piloting the new accreditation standards as they develop their self-studies. The Annual Conference will include four plenary sessions, including one that will provide a detailed update on the Commission's planned process and cycle changes.

Early Registration, before November 4, $635.00 On-site Registration after November 4, $715.00

7

8