A WINDOW ON ASSESSMENT - California State University ......program learning outcomes (PLOs) that will best benefit students. Campus Labs is a data-driven portfolio platform that allows

Spring 2018 | Issue 21

Office of Academic Program Assessment

(OAPA)

Dr. Amy Liu, Director Location: South-End Library 67

Phone: 916-278-2497

Email: [email protected]

Office Open: Monday - Friday

Phone: 916-278-2497 | [email protected]

Contact us anytime for updates or support!

Visit us at: csus.edu/programassessment

Inside this Issue:

1. Canvas/Capstone Project:

A Revolutionary Experiment to Integrate

Course, Program, and University Learning

Outcomes and Assessment

Dr. Amy Liu

Shelby Zahn

2. Zoom In & Zoom Out: Deepening Our

Communication Internally and Externally

Dr. Chia-Jung Chung

3. Using Canvas to Do Assessment:

Experimenting with a Live Sociology

Course

Dr. Jacqueline Brooks

4. The Cusp

Paul Schoenmannn

5. Workshops, College Visits, and Events

A WINDOW ON

ASSESSMENTC A L I F O R N I A S T A T E U N I V E R S I T Y , S A C R A M E N T O O F F I C E O F A C A D E M I C A F F A I R S

A Window on Assessment: Spring 2018 | 2

Canvas/Capstone Project: A Revolutionary Experiment to Integrate Course, Program, and University Learning Outcomes and Assessment Amy Liu, Ph. D., Director, OAPA and Professor of Sociology

Shelby Zahn, Student Assistant, OAPA and Graduate Student in Sociology

The Office of Academic Program Assessment (OAPA) has advised programs for many years to use assessment data from capstone projects to improve student learning and success. However, course assessment, program assessment, and university assessment are often separate processes. With the adoption of Canvas (a Learning Management System), OAPA sees an opportunity to not only improve, but also transform, the program assessment process, as well as student learning and success in courses, and at the university as a whole. Inspired by UCLA’s Capstone Initiative, Kansas State University’s successful integration of Canvas, and Campus Labs, we will attempt to do so through the utilization of these new technologies and initiatives. At Sacramento State University, we want our faculty and students to be active and engaged participants in the education process, which we believe capstone projects, Canvas, and Campus Labs can facilitate. Canvas is a tool that enables instructors to upload their course content into one location and grade using course, program, and university-level rubrics and outcomes. It encourages and requires the collaboration of faculty to modify and develop program learning outcomes (PLOs) that will best benefit students. Campus Labs is a data-driven portfolio platform

that allows faculty and institutions to visualize and assess student success, providing centralized data and predictive analytics. It provides more in-depth information about outcomes, allowing faculty and administrators to view outcome achievements by years and levels at the program and course levels, just as Canvas uses leverage filters and predictors to assess trends in student success. Faculty can easily build rubrics from scratch, or import rubrics from Canvas. Capstones require the same level of collaboration to develop PLOs and have shown to improve student success. Thus, combining these tools should yield useful quality data concerning areas in which students need improvement and how faculty may try to facilitate that positive development. For assessment, Campus Labs can be integrated for program review, accreditation, insight, rubrics, grading, and academic planning. Its dashboard allows faculty to use curriculum maps to see how previous courses have helped students achieve student learning outcomes (SLOs) necessary to succeed in future courses. Faculty now have the ability to assess data, which can be imported from Canvas capstone courses, and create assessment plans all using one software for maximum convenience. The software provides customizable templates where faculty can set user permissions and add content. It also has built-in course evaluations, which faculty can customize with course or

DR. AMY LIU SHELBY ZAHN


program-specific questions and use to assess how well they are teaching their students, and which students can use to share their experiences and feedback for improvement. After all, students are active rather than passive learners, and education is a reciprocal process wherein students influence professors as much as professors influence students.

In keeping with our goal of using technology to make our program assessment process simple, clear, and of high quality, we have been testing Canvas since fall 2017. Thus far, we have had three faculty consultants creating dummy classes, as well as live classes, as testing sites. They have uploaded their syllabi, quizzes, assignments, files, and other teaching materials. The student assistants have been given administrative access and created a group (or folder) of university-level AAC&U VALUE rubrics. Within that group, we have created separate subgroups for each outcome, including the five WSCUC core competencies (Critical Thinking, Information Literacy, Oral Communication, Quantitative Literacy, and Written Communication). In each of those subgroups, we have taken each criterion on the rubric and created them as separate outcomes. For instance, Critical Thinking’s first criterion “Explanation of Issues” is one outcome, and “Evidence” is a second outcome. For each outcome, we have set the points in a range of 4-1 (4: Exceed standards, 3: Meet standards, 2: Meet minimum standards, and 1: Do not meet standards), and any faculty or program can copy and adopt these outcomes as their course and/or program outcomes. Then at the course level, we have had faculty consultants implement these rubrics into their own dummy and live courses by adding them to assignments and quizzes using Canvas’s “Find” and “Import” features. The student assistants and others who have access to the dummy accounts submitted assignments and took quizzes in order to generate sample data for twenty dummy students, and real data was obtained from the live classes. For the first time in our university history, we have student learning outcome data aside from just the course grades.

Some programs are already interested in using Canvas to simplify their program assessment process. We are currently working with the Nursing and Criminal Justice departments to integrate some of their courses into Canvas for the course and program assessment at the same time. We hope to help them set up course outcomes and rubrics and systematically integrate them in the program level that can be modified to reflect the course, program, and university learning goals simultaneously. Within each course in the department, we would like to help them further tailor their criteria of outcomes to be specific to their subject matter. This will help familiarize faculty with university and program level outcomes and how they can be aligned with their courses in the assessment process to show them how they may improve student learning and success. To illustrate how these technologies can help simplify the assessment process, we presented a demo of Canvas and Campus Labs on May 3rd from 12:30-2:30pm in Library 11.

To close, capstones bring programmatic outcomes for the major into focus, and the assessment of students’ capstone performances serves as an effective diagnostic tool to facilitate the curricular review and reform process. The decision to align the Capstone Project can facilitate and revolutionize the process of setting criteria and standards for our campus. With the help of Information, Resources and Technology (IRT), three faculty from our office have used Canvas to articulate the program and course learning outcomes and develop a corresponding assessment framework for the course, the program, and the university altogether. The capstone projects have highlighted program outcomes specific to each major, and the assessment of students’ capstone performances can serve as an effective feedback tool for the course instructor and for the program, thus facilitating in the processes of annual assessment, 6-year program curricular review and reform, and providing feedback to the students in the class. Our experiment would not be successful without the help of Shawn Sumner and Matt Kay of the Academic Technology Center (ATC), April Qian and Kyle Cage of the Center for Teaching and Learning (CTL), and Information, Resources and Technology (IRT). We really appreciate their help!


Zoom In & Zoom Out: Deeping Our Communication Internally and Externally Chia-Jung Chung, Ph. D., Consultant, OAPA Professor of iMET

Assessment involves lots of collaboration and communication to ensure the quality of student learning and success. In the Office of Academic Program Assessment (OAPA), we have been striving for more frequent and deeper communication with all academic units on campus, including faculty, programs, departments, and colleges. This year we moved one step further to meet with the Assessment Offices in other universities that have inspirational and successful stories.

Zoom In: College Visits In spring 2018, OAPA visited almost all colleges on campus with the intention of listening to their needs in order to provide better support. We have sincerely appreciated the colleges’ effort to schedule meeting times with us, as it is beneficial to everyone involved. In these face-to-face meetings, the Assessment Coordinators, Chairs, Associate Deans and Deans have the opportunity to ask questions or express their concerns, and OAPA’s Director and consultants are able to provide immediate feedback or suggestions.

Zoom Out 1: Kansas State University: Getting the Most out of Canvas: Automating Assessment There are two main reasons we looked for an opportunity to meet with the Director and Assistant Director at Kansas State University. First, Kansas State has implemented Canvas as a data collection tool and has been using it to conduct meaningful assessment for years. Second, Kansas State University’s Office of Assessment website is featured on the National Institution for Learning Outcome Assessment page. According to the National Institution for Learning Outcome Assessment, Kansas State University’s Office of Assessment website features interactive infographics that allow site visitors to sort and view detailed information and data on the survey results presented in the infographics. Kansas State’s website also includes information regarding assessment and student learning outcomes at Kansas State, as well as degree and program-level assessment plans. What really attracts our assessment office is the Quick Start Guide to Assessment in Canvas, developed by Kansas State University’s Office of Assessment. This guide clearly outlines the steps of how to add outcomes, align outcomes with assignments and track and improve student learning on Canvas.

DR. CHIA-JUNG CHUNG


On our campus, we are migrating our learning management system (LMS) from Blackboard to Canvas, and our office truly foresees the potential and value of using Canvas as an assessment tool, as well as a learning management system. This way assessment will not be something to do at the end of each academic year, but an ongoing process while the courses are being offered at the beginning of the academic year. The data from student learning outcomes will be collected throughout the academic year on Canvas at course, program, department, college and university levels. In our Zoom meeting with Kansas State University, our discussion was centered on three focuses: 1) Overall Implementation and Faculty Facilitation of Canvas, 2) Kansas State’s successful story of using Canvas as an assessment tool, and 3) How the data is generated and used. Both the Director and Assistant Director in the Office of Assessment at Kansas State shared these three focuses related to Canvas and Assessment with rich ideas and concrete examples in their presentation, “Getting the Most out of Canvas: Automating Assessment”. What also impressed us is their willingness to share and help us learn from their years of implementation. Seeing colleagues work to improve the assessment process with such passion has really opened my eyes to how much easier and more meaningful assessment can be. It sounds simple but indeed requires a lot of work to make both happen.

Image Source: Screenshots from Zoom Meeting. The content of the image is created by Dr. Frederick Burrack and Mr. Chris Urban


Zoom Out 2: California Polytechnic State University: Assessment Workshop Model OAPA offers multiple assessment workshops each semester as part of our standard operation and responsibility. Workshops are events where we deliver information and knowledge, as well as stress their value as opportunities for deeper communication and collaboration. Even though our 2-hour workshops are a great success, we are curious about how other institutions, such as Cal Poly, offer the assessment workshops. In our Zoom meeting with the Director of Academic Assessment at California Polytechnic State University, we learned about a great 2-full day workshop model that allows the faculty to plan the curriculum map and work on the assessment together. In the most recent 2-day assessment workshop offered by Cal Poly, the faculty and assessment coordinators learn and work on the following critical assessment work in the workshop:

• Create an accurate curriculum map following best practices.

• Analyze maps for patterns and insights regarding a program’s learning outcomes and courses in the curriculum.

• Develop a plan to guide your program faculty’s work creating a preliminary program curriculum map.

• Use the program’s curriculum map to outline the program’s assessment plan.

We definitely learned a lot from Cal Poly’s story of a successful assessment workshop and gained a better understanding of the resources and structure needed for this workshop model. In the near future, we hope to offer a similar workshop for our faculty.

Conclusion By Zooming in and out of meetings with other institutions and our colleagues, we learned we need more frequent, deeper communication. There is no end point because there are always successful stories of assessment for us to learn and improve our own assessment.


Using Canvas to Do Assessment at OAPA: Experimenting with a Live Sociology Course Jacqueline Brooks, Ph. D., Consultant, OAPA Professor of Sociology

This academic year, the CSUS campus rolled out a new LMS system - Canvas. As a replacement for Blackboard, Canvas promises to be more user-friendly as a learning platform. Currently, educators across the university are “testing out” Canvas, with plans of campus-wide integration scheduled for fall 2018. Three OAPA faculty, including myself, are using Canvas to integrate course, program, and university assessment. I am using Canvas in my undergraduate courses, hoping to work out

all of the pesky issues that come along with learning the ins-and-outs of new technology before the fall semester begins. In addition to performing my regular instructor functions in Canvas (e.g. managing the grade book, posting assignments, grading assignments, and communicating with students), I am using my live courses to experiment with assessment.

Throughout the year, the OAPA team had brainstormed how to use Canvas to streamline the process of data collection, assessment, and data presentation. Although the current process is functional, we hope that Canvas will serve as an instrumental tool to improve the data collection process, strengthen the ability of faculty to assess student work in meaningful ways, offer more ways to present assessment data, and align the assessment plan and process across programs, colleges, and the university. Each part should work interdependently to reduce overall faculty workload. Thus, this semester, I am trying my hand at using Canvas for assessment. My goal is to assess the progress of students in my undergraduate theory courses (N=58). I plan to assess their academic progress utilizing three quizzes, three exams, and two papers. I will assess basic Bloom’s Taxonomy skills (i.e. understanding and remembering) on each quiz, their ability to employ critical thinking on each exam, and their ability to use critical thinking and written communication on each paper.

DR. JACQUELINE BROOKS

Bloom’s Taxonomy:

· Understanding· Remembering

AAC&U Value Rubric:Critical Thinking

AAC&U Value Rubrics:

· Critical Thinking· Written Communication

3 QUIZZES

3 EXAMS

2 PAPERS


I would like to share a few insights that I have developed along the way.

First, before utilizing my undergraduate students as “research subjects” in my little experiment, it was necessary to explain the meaning of assessment to them. As you know, students base their “success”

or “failure” on a grade. However, how well students have performed in the course goes well beyond their ability to take exams, classroom participation, and write papers. In the Sociology department, our program learning goals center on Competency in the discipline and knowledge of human cultures, Intellectual and practical skills, Values: Personal and social awareness, and Integrative learning. More specifically, we employ the AAC&U VALUE Rubrics to assist in assessing how well students develop and exhibit skills, such as Inquiry and Analysis, Critical Thinking, Writing, and Quantitative Literacy. In order to “ease” students into the “assessment talk,” I discussed how as an educator, I am seeking new ways to improve student learning. I explained that the assessment tools we will employ this semester will offer me more information to share with students regarding their progress in the course. Overall, despite some confused looks, the discussion made students feel at ease. Basically, they felt secure knowing that I was trying to help them as much as possible.

Second, as referenced in the above illustration, I am interested in assessing how well students exhibit critical thinking, and written communication. Still, I experienced a slight problem when I tried

to assess student performance on the objective quizzes, which comprise multiple-choice, true/false, and matching questions. None of the AAC&U rubrics aligned with the objectives quizzes. Thus, I created my own rubric based on Bloom’s Taxonomy. I chose to assess content knowledge using two levels of Bloom’s Taxonomy: remembering and understanding. I was able to add the rubric to the quizzes. Yet, as I will discuss later, since this rubric was created at the course level, and not the university-level, this poses an interesting assessment dilemma.

Third, thanks to the OAPA Team, the AAC&U value rubrics were uploaded to the Canvas platform making them readily accessible to me. Nonetheless, since we’re in the early days of using

Canvas for assessment, several technical glitches complicated my ability to access the value rubrics. This was just a reminder that technology is not flawless!

Fourth, it is easiest to use Canvas to assess assignments when they are submitted online. However, students complete their essay exams in-class, which causes a problem. Fortunately, you

can add a rubric to many assignments that appear in your gradebook. Thus, this is an easy fix. All the same, grading hard copy assignments, then using an online rubric to assess performance, can be quite cumbersome.

Fifth, we are in the process of identifying an effective software package to analyze our assessment data. Once individual educators assess assignments in their courses, that data (as

long as rubrics created at the university-level were utilized), should aggregate to the program, college, and university level. Thus, educators should be able to pool data across their programs. Currently, I am analyzing my data using Excel, but I hope we will have the ability to analyze data on a larger scale in the near future.

Sixth, it is difficult to use assessment data in a meaningful way in isolation. Until Canvas becomes fully integrated across campus (scheduled for fall 2018), and multiple courses

within a program use Canvas for assessment, we will not understand the true effectiveness of technology-based assessment.

The OAPA Team plans to continue experimenting with Canvas as a means of assessment throughout the 2018-2019 academic year. So, look forward to us approaching your individual programs with valuable insights on how Canvas can serve as a tool for assessment. We look forward to your feedback and cooperation!

13

4

5

62


On the Cusp Paul Schoenmannn, Student Assistant, OAPA, and Graduate Student in Sociology

In 2014, I graduated with my Bachelor’s Degree in Sociology as a member of Alpha Kappa Delta International Honor Society of Sociology and made a single phone call to the then Graduate Director, Dr. Amy Liu.

After starting the graduate program here at Sacramento State, I applied to the student assistant position.

Now in 2018 under Amy’s visionary leadership, I have grown tenfold, both personally and professionally. The biggest learning curve for me, frankly, was remembering my role. I often leapt before I looked, thinking I could actualize what I had learned, which was much easier said than done. During this time, I have developed a thorough understanding of OAPA’s daily operation and the overall mission and vision in terms of, not just course or program, but also university assessment. Amy has a vision for the future of assessment that is so concrete one can almost reach out and touch it. It is because of this that my confidence sometimes swells as I share this vision and what I have learned with faculty, department chairs, and Deans and with the OAPA consultants in our weekly meetings. Unlike the OAPA faculty consultants, however, because I am just a student assistant and cannot specifically represent the office, I sometimes feel like I am right on the cusp of having real impact.

For the time being, I eagerly await the moment when I can help share, shape, and align the motives and intentions of individuals and organizations to better achieve their goals and desired outcomes. OAPA has already profoundly assisted me in this direction through its dynamic professional development. Additionally, those within the office have sincerely aided me in my own personal development as a first-generation college student turned graduate student. This is what has made working at OAPA a truly heartwarming and humbling experience, not to mention one that requests the very skillset in which I was previously trained. I am truly grateful to this office and humbly appreciate Dr. Amy Liu for selecting me for this role. I intend to continue my growth and development and wholeheartedly aim to further improve myself to achieve a self-actualizing model and stalwart ethic worthy of representation.

As I transition to other boding opportunities, I want to take a moment to thank every one of OAPA’s faculty, those in Academic Affairs, and a great many others throughout our university for becoming a home away from home for me: Dr. Amy Liu, Dr. Chia-Jung Chung, Dr. Don Taylor, Ms. Kathy Mine, Dr. Elizabeth Strasser, Dr. Judi Kusnick, Dr. Jackie Brooks, Dr. Jing Pang, and Dr. Milica Markovic.

I thank each one of you for all that I have learned. I couldn’t have done it without you.

PAUL SCHOENMANNN

Director Amy Liu explains the 2016-2017 assessment cycle feedback reports to faculty at the Annual Assessment

and SharePoint workshop.

Dr. Jeffrey Brodd explains the necessity of program review to faculty at our collaborative Program Review Workshop.

Director Amy Liu directs faculty to useful resources and materials for effective assessment at the Program Review workshop.

Drs. Chia-Jung Chung and Jackie Brooks share their experiences using Canvas in their courses and how it can be used to improve

student learning and success at the CTL Teaching Summit.

Workshops, College Visits, and Events




Drs. Amy Liu, Chia-Jung Chung, and Jackie Brooks meet with the College of Arts & Letters’ Deans, Associate Deans, and chairs to

discuss their college’s annual feedback and reports.

Drs. Amy Liu and Milica Markovic meet with the College of Business Administration to discuss their annual reports and feedback.

The Professional Learning Community meets in spring 2018, led by Director Amy Liu and colleagues from CTL and OIR.

Dr. Chia-Jung Chung provides instructions and an overview of SharePoint, where OAPA collects annual reports.



End of Semester 2018 Potluck; Clockwise from right: Dr. Amy Liu, OAPA; Dr. Chia-Jung Chung, OAPA;

Dr. Don Taylor, Academic Affairs; Dr. Milica Markovic, OAPA; Paul Schoenmannn, OAPA; Christian Schoenmann, OIR; Shelby

Zahn, OAPA; Dr. Jackie Brooks, OAPA; April Qian, CTL.

End of Semester 2018 Potluck; Clockwise from right: Christian Schoenmann, OIR; Shelby Zahn, OAPA; Dr. Jackie Brooks,

OAPA; April Qian, CTL; Dr. Lynn Tashiro, CTL; Dr. Amy Liu, OAPA; Dr. Chia-Jung Chung, OAPA; Dr. Don Taylor, Academic Affairs;

Dr. Milica Markovic, OAPA.

Documents

A WINDOW ON ASSESSMENT - California State University ......program learning outcomes (PLOs) that will best benefit students. Campus Labs is a data-driven portfolio platform that allows