Communication Studies Department 2011 Annual Assessment Update

I. Mission Statement, Student Learning Outcomes, Curriculum Map, and Multi-Year Assessment Plan

A. Mission statement: Sapientia et Eloquentia. In this centuries-old phrase promoted by Cicero, St. Augustine and others, the Department of Communication Studies finds its mission, to educate students toward the union of wisdom and eloquence, recognizing the limited impact of wisdom without eloquence and the social harm that comes from eloquence without wisdom.

We are committed to helping students become wise analysts regarding the techniques and structures of human influence in a globally-oriented, media-saturated culture. We intend for our students to be informed by Scripture and able to draw upon resources in the historic discipline of Communication Studies.

We are committed to helping students become eloquent in mind and heart and speech, articulate and compassionate implementers of their wisdom. We intend for our students to communicate well and to imitate Christ by practicing reconciliation. http://www.westmont.edu/_academics/departments/communication_studies/mission.html

1. The mission statement is unchanged.2. The website featuring the mission is unchanged.

B. Revised Program Learning Outcomes (PLOs): The Communication Studies mission statement emphasizes two broad goals, that students are able to:

Analyze human symbolic activity wisely Demonstrate eloquent communication

Specific program learning outcomes that derive from the mission and the two broad goals are:1. Students articulate disciplinary knowledge appropriately2. Students can apply biblical and ethical principles appropriately to

complex communication situations.3. Students communicate effectively in oral form.4. Students communicate effectively in written form.

Discussion: Heeding the advice of the PRC regarding the need for streamlined PLOs and the advisability of matching, where possible, PLOs with scheduled ILO assessment activities, we reduced our PLOs from eight to four. This task required multiple department meeting discussions in Fall 2010, and required a change in our Multi-Year Assessment Plan. We had originally scheduled “Other-centeredness” for assessment during 2010-11, but we dropped this PLO (or, rather, blended this outcome into the other PLOs). This important change meant that we started assessing our new PLO

1

“Disciplinary Knowledge” in spring 2011. The revised Program Learning Outcomes may be found online at: http://www.westmont.edu/_academics/departments/communication_studies/ProgramReview.html

C. Curriculum Map and Outcomes Matrix by course: Curriculum Map

Program Goals Human Symbolic Activity

Human Symbolic Activity

Eloquent Communication

Eloquent Communication

Learning Outcomes

Disciplinary knowledge

Biblical and ethical principles

Oral communication skills

Written communication skills

Where are Learning Outcomes Met?

I 006, 098, 129, 133, OCP

D 101,102, 103, 110, 125, 130, 135, 140,145

I 006, 015, 098, 101,102

D 110, 125, 129, 133, 135, 140,145, OCP

M 140

I 006

D 015, 110, 130

I 006, 098

D 101,102, 133, 140

M 103, 125

How are they assessed? Rubrics, Portfolios, Capstone, External Jury, Exit Interviews, Surveys, etc.

Exit Exam, Alumni Survey, Rubric

Exit Exam, Alumni Survey, Exit Interviews

Rubric, External Jury, Alumni Survey

Rubric, External Jury, Alumni Survey

Benchmark Average or better on rubric and exam

Average or better on exit exam

Average or better on rubric

Average or better on rubric

Link to ILOs Critical Thinking, Intellectual Engagement, Diversity & Global Awareness

Christian Understanding / Practices / Affections

Competence in Written/Oral Communication

Competence in Written/Oral Communication

2

Outcomes Matrix by Communication Studies Course

CourseDisciplinary knowledge

Biblical & ethical

principles

Oral communication

Written communication

006 I I I I015 I D098 I I I101 D I D102 D I D103 D M110 D D D125 D D M129 I D130 D D133 I D D135 D D138 D D140 D M D145 D D DOCP May

I D D

Key: I=Introduced D=Developed M=Mastered

D. Multi-Year Assessment Plan: Scheduled Assessment Activities

(Revised according to Institutional ILO Alignments)

Outcomes 2011-12 2012-13 2013-14 2014-15 Facilitator

Written Communication

X Ochieng

Disciplinary Knowledge

X Dunn

Oral Communication X Stern

Biblical & Ethical Principles

X Spencer

II. Follow up on Action Items Identified in Previous Reports

3

A. PRC: “Revise your Curriculum Map. Align your SLOs with the curriculum map and identify the courses where the student learning outcomes are mastered and should be evaluated.”1. Oversight: Deborah Dunn 2. Update: See I. C.3. We’ve discussed whether or not we might offer more

sequence courses. Obstacles to this possibility include both the broad nature of the discipline (the number of courses that we feel compelled to offer) and the limited number of faculty in the department.

B. PRC: “Choose one outcome per year and one or two follow-up items selected for implementation.”1. Outcome oversight: Greg Spencer. Update: Last year (after

revising our PLOs) we chose “Disciplinary Knowledge” as our SLO to assess. Please see III. for our discussion.

2. Follow-up item 1: As a way of closing the loop on our 2010 data collection in COM 130 and COM 140, we will take action to improve our students’ persuasiveness. This step is also in keeping with the suggestion that we parallel our departmental assessment with campus-wide assessment, which in 2011 is “Oral Communication.”(a) Given our conclusion in spring of 2010 that our

students needed to improve their abilities to construct sound arguments, we have decided to have all assignments involving argument (speeches and essays) require students to attach an argumentation form to the written section. Oversight: Omedi Ochieng

(b) We have posted an introduction to argumentation and the argument form online on our website for students to download. We will use these outlines as data for our assessment of this SLO. URL: http://www.westmont.edu/_academics/departments/communication_studies/argument.html

3. Follow-up item 2: As a way of closing the loop on our discussion of written communication in 2009 (through COM 125 research papers), we are limiting citation options to MLA and APA. This action toward less confusion and better citation compliance will be instituted in all courses this fall. Special emphasis will occur in COM 98, Research Methods.

C. PRC: “Upon consultation with the Dean of Curriculum and Educational Effectiveness, develop your Multi-Year assessment Plan.” Oversight: Deborah Dunn (See I.D.)

D. PRC: “Charge your Administrative Assistant with posting your SLOs, Alignment Chart (Curriculum Map) and Multi-Year Assessment Plan on the departmental website.” Done by Deborah Dunn.

E. Other PRC letter follow-up issues: 1. Advice of PRC to submit Serving Society option to GE: Still

at discussion level; no action taken.

4

2. Expressed department Interest in applying for external grants that would help fund student research: Interest still exists.

3. Address internship supervisor ratings: Ratings by supervisors of student interns were almost universally positive; the one area we identified as having lower scores involved students taking initiative. We do not identify this as a weakness in our students or our program, but rather a feature of young people new to a career, a workplace, and a job. No action taken.

4. Explore Film Studies minor (Oversight: Greg Spencer): As in several previous years, we discussed this option. In the last year we viewed the proposal by Scott Anderson in Art, and as a department we collected examples of various film studies/media studies minors at other colleges. We concluded that Scott’s proposal needed revision. This year we will:(a) Investigate a new minor curriculum that is not tied

“to film” per se, but might be called comparative media studies

(b) Propose a meeting with representatives of Communication Studies, Art, English and Theater.

III. 2011 Focus: Disciplinary Knowledge PLOA. Assessment methods:

1. Eleven students completed an “exit exam” (drawn from a potential pool of 38 graduating seniors).

2. The department developed a rubric to assess the exams. The benchmarks were an average score or above on our rubric.

3. See Appendices B and C.4. URL of Assessment Archive:

http://www.westmont.edu/_offices/institutional_portfolio/program_review/AssessmentandProgramReviewArchive.html

B. Interpretation of the Results: 1. The department met to collectively interpret the data on

May 11, 2011. The minutes for this session are in the assessment archive. The rubric for the data is in the assessment archive.

2. No external voices were consulted. 3. What we learned:

(a) The main thing we learned is that we need to create a new assessment tool. During spring semester, we developed an exit exam that consisted of three questions derived from each course we offered. After assembling this exam, we decided that it did not represent the discipline well. Then we developed a second exam, a more open-ended analysis of two advertisements. See Appendix B.

5

(b) The department discussed ways to design an exam that could do justice to the complexities of what students have learned while not placing too great a burden on our seniors to run through a gauntlet of additional exams or essays. Post-exam debriefing with students indicated that many felt that the exam’s timing was not ideal given that it was taken at a particularly busy time. At this point, our conclusion is that we need to find circumstances more conducive to high-commitment test taking on the part of our students. We will offer an assessment tool earlier in the semester or as a requirement of a course.

(c) Another conclusion we reached was that the tool itself might need revision. If we use an exit exam again we will certainly have to revisit the questions and format, but we will also investigate other means of assessing disciplinary knowledge.

(d) While we did not set out to assess student writing, we gleaned additional information here. We discovered that students generally offered original ideas in their interpretation of texts but were weaker in the areas of formulating clear thesis statements and supporting these theses with strong, rigorous arguments.

C. Closing the Assessment Loop:1. The department has taken a number of steps in light of

what we have learned in our previous cycle of assessment. We will refine our assessment tool for disciplinary knowledge so that we capture the complexity of student learning while being mindful that our students have a variety of ways of processing and integrating what they have learned.

2. Based on data collected this year as well as previous assessments of student writing, as of Fall 2011, the department now requires that all students submit an argument worksheet with each critical essay. This requirement will make students think more carefully about their thesis statements and the structure of their arguments.

3. Based on data collected this year as well as previous assessments of student writing, the department now requires that students master only two kinds of citation styles: the Modern Language Association (MLA) citation style and the American Psychological Association (APA) citation style (reduced from three styles formerly allowed or required). With fewer styles to master students should become more proficient in citing their research sources.

IV. Next StepsA. Action Items:

1. Continue to collect papers in archive for Written Communication SLO assessment.

6

2. Collect new argument forms for later assessment (Fall 2011:Omedi Ochieng).

3. Institute new policy regarding MLA and APA style manuals (Fall 2011: Deborah Dunn).

4. Revisit Disciplinary Knowledge SLO exit exam. Discuss future usefulness and compare with other communication departments’ assessment tools. Create new exam or consider alternatives, including Senior Capstone course (Lesa Stern).

5. Propose Film Studies/Comparative Media Studies minor (Greg Spencer).

6. Initiate Alumni survey (Fall 2011: Deborah Dunn).B. Next 6 year report is due 2015.

V. AppendicesA. Last year’s response from the PRC B. Prompts or instruments used to collect data: Our Disciplinary

Knowledge exit exam. C. Rubric used to evaluate exit exams

7

Appendix A: Last year’s response from the PRC

Program Review Committee Date: 11.23.2010 To: Greg Spencer, Department of Communication Studies From: Program Review Committee /Tatiana Nazarenko Re: 2010 Communication Studies Annual Assessment Update Report

Thank you for submitting your 2010 Annual Assessment Update Report and downloading it into the Program Review Archives. The Program Review Committee appreciated your time and efforts put into preparing this report. The Program Review Committee commends your department on the following attainments: revising and reducing in number your Program Learning Outcomes following the November 10, 2010 departmental meeting after having submitted your Annual Assessment Update Report. Your four new outcomes are clear, concise, measurable and manageable. Great job! developing and testing an Oral Communication rubric for COM 130 inviting external reviewers to rate student research papers for COM 125. inviting alumni to assess group presentations for COM 145 by using the rubric.

It is also laudable that the department faculty are engaged in ongoing discussion of assessment results and use the collected evidence to enhance student leaning and success. We are very pleased to learn that the Communication Studies faculty proposed “that every class have at least one assignment that requires physically checking out a book and referencing at least one article.” Keep up doing a good job! The response to your annual report is intended to affirm that you have accomplished in the past year and discuss the sustainability of your current and future assessment efforts. The following body of the response is tailored directly to specific sections of your report. It addresses the major items identified by the reviewer. Other assessment-related issues will be discussed during your departmental meeting with the Dean of Curriculum and Educational Effectiveness. Mission statement Regarding the content of the Communication Studies mission statement is concerned, it is fine. However, your current mission statement is five sentences long. Have you considered abridging it? I am also interested in the semantics of the photograph that accompanies your mission at http://www.westmont.edu/_academics/departments/communication_studies/mission.html. What message is the photograph placed under the heading “Understanding Symbols” expected to convey to the internal and external constituencies? Program/Student Learning Outcomes I would recommend that you post your new crisp SLOs on the departmental website along with the revised Alignment Chart (Curriculum Map) and Multi-Year Assessment Plan. Please, charge you Administrative Assistant with updating your website. The definition and application of the I (Introduced), D (Developed), and M (Mastered) indicators in the Alignment Chart (Curriculum Map) were discussed at the October 2010 WASC retreat, which I attended. According to WASC assessment expert Dr. Mary Allen, the “mastered” indicator means the highest expected level of proficiency (mastery) which can be attained by any undergraduate student completing the course/program. Understandably, it is not possible to have every single outcome or even the majority of them mastered in every single course offered by the department. Typically, students attain this level of mastery in Capstone courses and Senior Seminars, where the SLOs should be measured. 2

8

The alternative way of mapping the curriculum is to use checkmarks instead of I, D, and M indicators in the Alignment Chart (Curriculum Map). However, the benefit of using I, D, and M indicators is that the department can clearly see which particular courses should be selected for assessing Program/Student Learning Outcomes. Let me reiterate: the good news is that we are not expected to assess student learning in the courses where the outcome was just introduced or developed unless we are interested in a value-added assessment judgment or have already identified the problem that requires fixing at the earlier stages of student learning in relation to the particular outcome. For program-level assessment we use courses – usually senior seminars and capstones – where specific outcomes were mastered. You may find additional information on this topic in the Curriculum Alignment Matrix file at the Educational Effectiveness site at the provost website, the URL is http://www.westmont.edu/_offices/institutional_portfolio/program_review/ProgramReview-FacultyResources.html At this point, it is important for your department to align your P/SLOs with your curriculum using the Curriculum Alignment Matrix as well as identify the courses where you are going to assess your outcomes. Follow-up issues (A2) I encourage you to submit the proposal for the GE credit in “Serving Society” and/or “Cross-Cultural Communication” area(s). (A5) I would also encourage the department to explore the possibility of applying for an external funding grant – in addition to internal funding – to support student research with Kenon Neal, Director of Foundation and Corporate Relations. Have you researched professional organizations and agencies sponsoring undergraduate research in Communication Studies? (B4) I wonder how the department is going to address the low internship supervisor ratings and what assistance is required to improve the situation. (C3) I am interested to know what assistance you need for establishing a Film Studies minor. Have you discussed this issue with the Department of English which is considering offering more film courses?

2009-2010 Focus I commend you again on developing the Oral Communication rubric. I wonder whether I can add this rubric along with the Communication Studies Writing Rubric posted at http://www.westmont.edu/_academics/departments/communication_studies/Style.html to the “Westmont Best Educational Effectiveness Practices” website which is currently under construction. I would like to learn more about the implementation of the assessment data collected in COM 125, COM 130, and COM 145. COM 125. What actions the department is going to take to “close the loop”? What changes in pedagogy, curriculum, student support, faculty support, or resource allocation suggested by your assessment results will be introduced? COM 130. It is my understanding that you want to collect more data on student oral presentations in this course. Have you considered choosing an action item based on the data you have? Then, if needed, you may take steps to gather more or better data. COM 145. Have you already discussed how you are going to improve students’ persuasiveness? What are the greatest problems with learning in this area? What changes in pedagogy can result in enhancement of student learning? What will be the timeline for implementing the change? 3

9

I also wonder whether you can share your COM 145 findings and collaborate with Dana Alexander, Director of the Office of Life Planning, in helping Westmont students to be better prepared for job interviews. This is one of the areas of Student Life which needs improvements. It is my understanding that your help would be appreciated. Have you ever considered doing a collaborative project the Office of Life Planning, whose staff are working on improving Westmont student interview skills? Given that not every single student on campus takes COM 145 course, the materials and tools created by your department for COM 145 may be helpful for the Office of Life Planning and used for reaching out to a greater number of Westmont students. Report organization Your current report lacks pagination. We expect that you will address this issue in your subsequent reports. Report Appendices Your Multi-year Assessment Plan has not been submitted. I wonder whether you need any help in putting it together. Organization of the PR Archives Your 2010 Annual Assessment Update Report follows file naming conventions and is easy to access, which is laudable. Conclusion Thank you again for your good work. Based on my review of your report and assessment activities conducted by your Department, I would expect you to focus on the following items in this academic year and report on their progress in your next your annual report: Revise your Curriculum Map. Align your SLOs with the curriculum map and identify the courses where the student learning outcomes are mastered and should be evaluated. Choose one outcome per year and one or two-follow-up items selected for implementation. Upon consultation with the Dean of Curriculum and Educational Effectiveness, develop your Multi-Year Assessment Plan. Charge your Administrative Assistant with posting your SLOs, Alignment Chart (Curriculum Map) and Multi-Year Assessment Plan on the departmental website. Implement changes suggested by the collected data. The end of assessment is action.

I commend you again on doing good work. Please let me know how I can further assist you in your assessment endeavors.

10

Appendix BDisciplinary Knowledge Exit Exam

Dear **:

Thank you for taking this "mini exam" before graduating. This assignment is not graded, will have no impact on your grades in other classes, and will not be used against you at commencement! This is a way for us to discover what our graduating seniors have learned or not learned as communication studies majors. Rather than ask you multiple questions that get at very specific topics across several different courses, we would like you to analyze the image/advertisement below and then write a scholarly response. While this is not graded, your four full-time professors (Dunn, Ochieng, Spencer, Stern) will discuss the responses in a department meeting. Your name will not be attached to your analysis by the time we come together to discuss the answers. While it is not graded, we do hope you put your best foot forward not just because we think it might be a valuable exercise for you, but also because we are genuinely interested in learning how our graduating seniors think.

Thank you again for your participation.

*************************************************************************************************************************************************Instructions: Please view the image/advertisement below. Then do your own critical analysis and answer the following three questions:

1) How would you critique this advertisement in light of what you learned in classes like interpersonal communication or group communication, or conflict and reconciliation?

2) How would you critique this advertisement in light of what you learned in classes like mass communication, public discourse, organizational communication, and/or international rhetoric?

3) Are there specific principles or guidelines you would bring to bear on your analysis given what you learned in communication criticism, or argumentation, or communication ethics?

11

Copy of advertisement #1

12

Copy of Advertisement #2

13

Appendix CRubric Used for Disciplinary Knowledge Exit Exam

14