View
5
Download
0
Category
Preview:
Citation preview
Assessing the Associate in Arts in Liberal Arts degree at
Leeward Community College Written Communication, Critical Thinking and
Cultural Diversity
November 2, 2016
Assessing the Associate in Arts in Liberal Arts degree at Leeward
Community College
Written Communication, Critical Thinking and Cultural Diversity
Fall 2016
Prepared by:
Eunice L. Brekke, AA Degree Program Review Coordinator
In collaboration with:
Luukia Archer (Arts and Humanities)
P. Jayne Bopp (Social Sciences)
James Fujita (Arts and Humanities)
Leah Gazan (Library Services)
Kathryn Fujioka-Imai (Language Arts)
Michael Lane (Math and Science)
Bruce Lindquist (Social Sciences)
Blanca Polo (Math and Science)
Susan Waldman (Language Arts)
Susan Wood (Language Arts)
Jeff Judd, Faculty Senate, Program Review and Assessment Committee, Chair
Guy Nishimoto, Institutional Effectiveness Officer and Adam Helemano, Assessment Coordinator,
Office of Planning, Policy, and Assessment (OPPA)
James Goodman, Dean of Arts and Sciences
1
Assessing the Associate in Arts in Liberal Arts degree at Leeward Community College Written Communication, Critical Thinking and Cultural Diversity
Submitted by:
Eunice Leung Brekke, AA Degree Program Review Coordinator
Background
In 2011, the college issued a comprehensive review and evaluation of the Associate in Arts Degree
“Assessment of the Associate in Arts Degree at Leeward Community College (2005 - 2010) 1.” The
report’s conclusions included
While the data suggest that students are meeting some general education outcomes,
and not meeting others, the more solid conclusion is that, overall, the college needs to
improve its assessment process. Specifically, the college should recognize that other
program level assessment strategies and methods are available.
In response, Faculty Senate and the Administration created an AA Degree Program Review Coordinator
position to facilitate regular and on-going evaluation of Leeward CC’s Associate of Arts degree and to
recommend and facilitate appropriate actions to respond to the findings of the prior assessment of the
degree.
In Spring 2014, the AA Degree Program Review Coordinator proposed an assessment strategy to the
Dean of Arts & Sciences, the four division chairs of the AA Degree (Arts and Humanities, Language Arts,
Math and Science, Social Sciences) and the Office of Planning, Policy, and Assessment (OPPA). The
assessment of student learning would continue to utilize course level outcome data that is collected
through Tk20 (the college’s assessment database) and reported annually through the Annual Report on
Program Data.
However, a new assessment process involving the development of common rubrics to measure three
general education learning outcomes was proposed as a pilot study to determine if the college could
develop more direct measures of student learning and an ongoing process of assessing the effectiveness
of the degree program. To accomplish this, the AA Degree Program Review Taskforce was created.
Members of the AA Degree Program Review Taskforce included:
Jeff Judd, Faculty Senate, Program Review and Assessment Committee, Chair
Susan Wood, Susan Waldman, Kathryn Fujioka-Imai, Leah Gazan (subcommittee on Written
Communication)
Michael Lane, Blanca Polo, P. Jayne Bopp (subcommittee on Critical Thinking)
James Fujita, Bruce Lindquist, and Luukia Archer (subcommittee on Cultural Diversity)
1 A copy of the report can be found at
http://intranet.leeward.hawaii.edu/system/files/assessment_of_the_aa_degree_fall_2011_final.pdf
2
Guy Nishimoto, Institutional Effectiveness Officer and Adam Helemano, Assessment
Coordinator, OPPA
James Goodman, Dean of Arts and Sciences
Eunice L. Brekke, AA Degree Program Review Coordinator and Chair
Assessment Methodology
The pilot study focused its work on measuring three general education learning outcomes: written
communication, critical thinking and cultural diversity. Task force members were trained on basic
assessment principles and provided with resources and technical assistance to help guide the
development of common rubrics. Each of the sub-committees (organized by learning outcome)
reviewed the degree’s current general education learning outcome and associated academic skill
standards and VALUE rubrics developed by ACC&U and developed a revised learning outcome and its
associated rubric that would best assess student learning of these outcomes.
The three revised general education learning outcomes and their associated criteria (academic skill
standards) are:
1. Written Communication - Develop, support, and communicate ideas to a particular audience
through writing.
Develop a position or thesis to communicate main ideas.
Use language, style, and organization appropriate to particular purposes and audiences to communicate position or thesis.
Develop appropriate content to support position or thesis.
Gather and document credible sources to support position or thesis.
Use appropriate grammar and mechanics to communicate position or thesis.
2. Critical Thinking - Use critical thinking to address issues and solve problems.
Describe an issue or problem.
Analyze the issue or problem.
Develop a position (perspective, thesis or hypothesis) on the issue or problem.
Evaluate conclusions and implications on the issue or problem.
3. Cultural Diversity - Describe and analyze the nature of culture and its variations, past and
present.
Describe culture and the variation of cultural practices, expressions, and/or experiences.
Analyze the variation of cultural practices, expressions, and/or experiences associated with culture.
Compare and contrast cultural practices, expressions, and/or experiences among different societies; past and/or present.
3
To create a rubric, each of these criteria were further defined and described along a scale of exceeds
proficiency, meets proficiency, developing proficiency and below proficiency. Copies of the rubrics can
be found in Appendices A, B, and C.
From Spring 2014 to Fall 2015, the rubrics for each of the learning outcomes were tested or normed by
the task force. Norming sessions typically involved an overview of the learning outcome and rubric
followed by an all group reading of various student papers, using the rubric to score. Each member’s
scores were recorded and reviewed to locate how close or far the group was in administrating or
interpreting the rubric. When discrepancies were found, discussion was held to understand why. In
some cases, revisions to the rubric were suggested and accepted. In some cases, we learned how
various disciplinary perspectives influenced the interpretation of the rubric. This process continued until
each rubric across a variety of student work representing all divisions and a variety of disciplines was
normed with one standard deviation.
Sampling Strategy
Throughout the norming process, it became evident that the better student papers or artifacts to
measure the learning outcomes were papers that captured what students could do at the end of a
course, for example a final research paper or essay. In collaboration with the Writing Intensive Focus
Board, a list of all Writing Intensive (WI) courses for Fall 2015 was obtained representing 30 courses.
Each of the instructors were contacted to request their participation. Instructors were asked if they
were willing to participate by submitting end of the semester final papers and to also indicate which of
the three learning outcomes they thought their assignment would measure. A total of 175 student
papers were collected representing 13 different courses and the four academic divisions of the degree
program.
The AA Degree Program Coordinator collected the papers and respective assignments. Each student
paper and the instructor’s name were hidden to protect student and instructor confidentiality. The
papers, assignments, and rubrics were uploaded into Tk20 and task force members received training on
how to conduct the assessment using Tk20. A final group norming session was held using all three
rubrics.
The task force scored 175 student artifacts for written communication and critical thinking. The number
of student papers scored for cultural diversity was 41 (three courses were used as indicated by
participating instructors). The data was reviewed by the AA Degree Program Review Coordinator to
ensure consistency (i.e., standard deviations were less than 1.0).
In Fall 2016, the task force held its final meeting to review the results. The task force’s discussion of the
results and implications for improvement are presented below.
4
Results
Figure 1 (below) presents the percentage of students meeting or exceeding proficiency in each of the
three general education learning outcomes. A little over half of papers met this level of proficiency in
written communication and critical thinking (58 and 56 percent) and a third (30 percent) in cultural
diversity.
Figure 1. Students Meeting or Exceeding Proficiency in Written Communication, Critical Thinking and
Cultural Diversity
58% 56%
30%
0%
25%
50%
75%
100%
Written Communication(n=175)
Critical Thinking (n=175) Cultural Diversity (n=41)
Meets or Exceeds Proficiency
5
Written Communication
For written communication, 13 percent exceeded proficiency and 45 percent met proficiency levels (58
percent or 102 papers). Another 35 percent were assessed as developing proficiency and 8 percent
below proficiency (43 percent or 73 papers).
In analyzing specific areas of how students perform, student strengths are in developing a position or
main idea (63%) and using appropriate grammar and language (62 and 60 percent). Areas of challenge
are gathering and documenting credible sources (51%) and developing appropriate content to support
the position (53%).
8%
35%
45%
13%
0% 25% 50% 75% 100%
Below Proficiency - (1)
Developing Proficiency - (2)
Meets Proficiency - (3)
Exceeds Proficiency - (4)
Written Communication (n=175)
63%
60%
53%
51%
62%
0% 25% 50% 75% 100%
Develop a position or thesis tocommunicate main ideas
Use language, style, and organizationappropriate to particular purposes…
Develop appropriate content tosupport position or thesis
Gather and document credible sourcesto support position or thesis
Use appropriate grammar andmechanics to communicate position…
Written Communication: Meets or Exceeds Proficiency (n=102)
6
Critical Thinking
For critical thinking, 13 percent exceeded proficiency and 43 percent met proficiency levels (56 percent
or 98 papers). Another 38 percent were assessed as developing proficiency and 7 percent below
proficiency (45 percent or 77 papers).
In evaluating critical thinking, students appear to doing best in describing an issue or problem (65%).
Areas of challenge are analysis, developing a position, and evaluating conclusions and implications (51 to
54 percent).
7%
38%
43%
13%
0% 25% 50% 75% 100%
Below Proficiency - (1)
Developing Proficiency - (2)
Meets Proficiency - (3)
Exceeds Proficiency - (4)
Critical Thinking (n=175)
65%
54%
53%
51%
0% 25% 50% 75% 100%
Describe an issue or problem
Analyze the issue or problem
Develop a position (perspective, thesis,or hypothesis) on the issue or problem
Evaluate conclusions and implicationson the issue or problem
Critical Thinking: Meets or Exceeds Proficiency (n=98)
7
Cultural Diversity
For cultural diversity, 30 percent (12 papers) met proficiency. Another 38 percent were assessed as
developing proficiency and 32 percent below proficiency (70 percent or 29 papers).
Although the number of students meeting or exceeding proficiency in cultural diversity is much smaller
compared to written communication and critical thinking, these students do best in describing culture
and its variations (44%). Analysis and the ability to compare and contrast were lower (22-24 percent).
32%
38%
30%
0%
0% 25% 50% 75% 100%
Below Proficiency - (1)
Developing Proficiency - (2)
Meets Proficiency - (3)
Exceeds Proficiency - (4)
Cultural Diversity (n=41)
44%
22%
24%
0% 25% 50% 75% 100%
Describe culture and the variation ofcultural practices, expressions, and/or
experiences.
Analyze the variation of cultural practices,expressions, and/or experiences associated
with culture.
Compare and contrast cultural practices,expressions, and/or experiences amongdifferent societies; past and/or present.
Cultural Diversity: Meets or Exceeds Expectations (n=12)
8
Conclusions
While these results indicate that students do have areas of strength (description of issues or practices,
developing a thesis or position to communicate main ideas and using appropriate grammar and
language), overall, there is room for improvement.
Specific areas that need improvement are developing a perspective on an issue or problem, developing
and gathering appropriate content and credible sources, and analysis and evaluation of an issue or
problem. These areas reflect higher levels of knowledges according to Bloom’s Taxonomy (see Appendix
D).
Recommendations
The task force respectfully makes the following recommendations to the Faculty Senate:
Take action on these results to provide ongoing assessment of the Associate of Arts in Liberal Arts
degree.
One approach is to convene the WI instructors who participated in this study to review the results,
discuss different strategies to make improvements, select a strategy and after implementation, reassess
student learning in written communication and critical thinking to see what has changed. We encourage
the administration to support this effort.
Identify courses in the degree program that address aspects of culture and cultural diversity (describe
and analyze the nature of culture and its variations, past and present) and also where they practice and
develop this outcome so that they can meet proficiency in this area. The sample size to measure cultural
diversity was much lower than hoped for. There is a need to locate better assignments or student work
to measure cultural diversity.
Finally, we ask that the college accept the work of the task force - the revised general education learning
outcomes, associated criteria and rubrics. These outcomes and criteria used to assess the outcomes
should be reflected in the college’s official description of its general education learning outcomes.
9
Appendix A. Written Communication GELO and Rubric
Written Communication
General Education Learning Outcome:
Current – Use writing to discover, develop, and communicate ideas appropriately.
Revised – Develop, support, and communicate ideas to a particular audience through writing.
Definition:
Written communication is the development and expression of ideas in writing. Written communication
involves learning to work in many genres and styles in order to communicate effectively with a
particular audience for a specific purpose. It can involve working with different writing technologies, and
mixing texts, data, and images. Written communication abilities develop through iterative experiences
across the curriculum.
10
Written Communication Rubric
Exceeds proficiency (4) Meets proficiency (3) Developing proficiency (2) Below proficiency (1)
Develop a position or thesis to communicate main ideas.
A position or thesis is clearly stated. Position or thesis is appropriate, organized logically, and supported by evidence or examples.
A position or thesis is clearly stated. Position or thesis is mostly appropriate, organized logically, and supported by evidence or examples.
A position or thesis is not clearly stated. Position or thesis is somewhat appropriate, organized logically, and supported by evidence or examples.
A position or thesis is not clear or not stated. Position or thesis is not appropriate to topic. It is not organized logically, and not supported by evidence or examples.
Use language, style, and organization appropriate to particular purposes and audiences to communicate position or thesis.
The intended audience is evident. The purpose is appropriate. Content reflects awareness of audience and purpose.
The intended audience is mostly evident. The purpose is mostly appropriate. Content mostly reflects awareness of audience and purpose.
The intended audience is somewhat evident. The purpose is somewhat appropriate. Content somewhat reflects awareness of audience and purpose.
The intended audience is not evident. The purpose is not appropriate. Content does not reflect awareness of audience and purpose.
Develop appropriate content to support position or thesis.
Position or thesis is supported by appropriate, relevant and compelling content.
Position or thesis is mostly supported by appropriate, relevant and compelling content.
Position or thesis is somewhat supported by appropriate, relevant and compelling content.
Position or thesis is not supported by appropriate, relevant and compelling content.
Gather and document credible sources to support position or thesis.
Position or thesis is supported by credible, relevant sources.
Sources are documented appropriately.
Position or thesis is mostly supported by credible, relevant sources. Sources are mostly documented appropriately.
Position or thesis is somewhat supported by credible, relevant sources. Sources are somewhat documented appropriately.
Position or thesis is not supported by credible, relevant sources. Sources are not documented appropriately.
11
Use appropriate grammar and mechanics to communicate position or thesis.
Grammar is used appropriately. Mechanics are used appropriately.
Grammar is mostly used appropriately.
Mechanics are mostly used appropriately.
Grammar is somewhat used appropriately.
Mechanics are somewhat used appropriately.
Grammar is not used appropriately.
Mechanics are not used appropriately.
Adapted from the Association of American Colleges and Universities (AAC&U), Valid Assessment of Learning in Undergraduate Education
(VALUE), 2014.
Notes:
Academic skill standards 1 - 9 have been incorporated into the rubric. These are now criteria to assess the learning outcome.
Last updated: May 2, 2014
12
Appendix B. Critical Thinking GELO and Rubric
Critical Thinking
General Education Learning Outcome:
Current – Make critical judgments and apply critical reasoning to address challenges and solve problems.
Revised – Use critical thinking to address issues and solve problems.
Definition:
Critical thinking is characterized by the comprehensive exploration of issues, ideas, artifacts, and events
before accepting or formulating an opinion or conclusion.
13
Critical Thinking Rubric
4-Exceeds proficiency
3-Meets Proficiency
2-Developing Proficiency
1-Below Proficiency
Describe an
issue or
problem.
The issue or problem is
clearly stated.
The description of the issue
of problem is appropriate to
topic.
The description is
comprehensive.
The issue or problem is clearly
stated.
The description of the issue of
problem is mostly appropriate to
topic.
The description is mostly
comprehensive.
The issue or problem is not
clearly stated.
The description of the issue of
problem is somewhat
appropriate to topic.
The description is somewhat
comprehensive.
The issue or problem is not
clear or is not stated.
The description of the issue of
problem is not appropriate to
topic.
The description is not
comprehensive.
Analyze the
issue or
problem.
Analysis includes an
examination of the context
and assumptions of the issue
or problem.
Supporting evidence is
appropriate, factual,
organized logically and
comprehensively.
Analysis mostly includes an
examination of the context and
assumptions of the issue or
problem.
Supporting evidence is mostly
appropriate, factual, organized
logically and comprehensively.
Analysis somewhat includes an
examination of the context and
assumptions of the issue or
problem.
Supporting evidence is
somewhat appropriate, factual,
organized logically and
comprehensively.
Analysis does not include an
examination of the context and
assumptions of the issue or
problem.
Supporting evidence is not
appropriate, factual, organized
logically or comprehensively.
Develop a
position
(perspectiv
e, thesis or
hypothesis)
on the issue
A position (perspective,
thesis or hypothesis) is
clearly stated.
Thesis is appropriate,
organized logically,
A position (perspective, thesis or
hypothesis) is clearly stated.
The position (perspective, thesis or
hypothesis)is mostly appropriate,
organized logically,
A position (perspective, thesis
or hypothesis) is somewhat
clearly stated.
The position (perspective, thesis
or hypothesis)is somewhat
A position (perspective, thesis
or hypothesis) is not clearly
stated.
The position (perspective,
thesis or hypothesis)is not
14
or problem. comprehensively and
supported by factual
evidence or examples.
comprehensively and supported by
factual evidence or examples.
appropriate, organized logically,
comprehensively and supported
by factual evidence or
examples.
appropriate, organized
logically, or comprehensively
and supported by factual
evidence or examples.
Evaluate
conclusions
and
implications
on the issue
or problem.
Conclusions and
implication(s) are
appropriate, factual,
organized logically and
comprehensively.
Conclusion and implication(s) are
mostly appropriate, factual,
organized logically and
comprehensively.
Conclusions and implication(s)
are somewhat appropriate,
factual, organized logically and
comprehensively.
Conclusions and implication(s)
are not appropriate, factual, or
organized logically or
comprehensively.
Notes:
Academic skill standards 1, 2, 3, 4, and 6 have been incorporated into the rubric. These are now criteria to assess the learning outcome. The task
force has determined that academic skill standard 5 is better assessed under the Arts, Humanities, and Sciences learning outcome.
The task force also determined that a thesis can also be a position, main point or thought, or controlling idea. Further explanation or
adjustments to the rubric’s language will be determined during the pilot study (i.e., rubric calibration).
Last updated: March 14, 2016
15
Appendix C. Cultural Diversity GELO and Rubric
General Education Learning Outcome:
Current – Appreciate the values and beliefs of diverse cultures and recognize responsibility for local,
national, and global issues.
Revised – Describe and analyze the nature of culture and its variations, past and present.
16
Cultural Diversity Rubric
Exceeds proficiency (4) Meets proficiency (3) Developing proficiency (2) Below proficiency (1)
Describe culture and the variation of cultural practices, expressions, and/or experiences.
Description of culture and the variation of culture are appropriate and in-depth and organized in a clear manner.
Description of culture and the variation of culture are appropriate and are mostly in-depth and mostly organized in a clear manner.
Description of culture and the variation of culture are somewhat appropriate, has some depth, and is somewhat organized in a clear manner.
Description of culture and the variation of culture are not appropriate and/or superficial and are not well organized, clear.
Analyze the variation of cultural practices, expressions, and/or experiences associated with culture.
Analysis is appropriate to topic, in-depth, variation is well supported by sufficient and appropriate content and examples, and organized in a clear manner.
Analysis is appropriate to topic, mostly in-depth; variation is supported by mostly sufficient and appropriate content and examples, and mostly organized in a clear manner.
Analysis is mostly
appropriate to topic, somewhat in-depth, variations are supported by somewhat sufficient and appropriate content and examples, and somewhat organized in a clear manner.
Analysis is not appropriate to topic, is superficial, variation is not supported by sufficient and appropriate content and examples, and is not organized.
Compare and contrast cultural practices, expressions, and/or experiences among different societies; past and/or present.
Comparison and contrast are appropriate to topic, in-depth, with practices, expressions, and/or experiences among different societies well supported by sufficient and appropriate content and examples, and organized in a clear manner.
Comparison and contrast are appropriate to topic, mostly in-depth, practices, expressions, and/or experiences among different societies are supported by mostly sufficient and appropriate content and examples, and mostly organized in a clear manner.
Comparison and contrast are mostly appropriate to topic, somewhat in-depth, practices, expressions, and/or experiences are supported by somewhat sufficient and appropriate content and examples, and somewhat organized in a clear manner.
Comparison and contrast are not appropriate to topic, is superficial, practices are not supported by sufficient and appropriate content and examples, and is not clearly organized.
Adapted from the Association of American Colleges and Universities (AAC&U), Valid Assessment of Learning in Undergraduate Education
(VALUE), 2014.
17
Notes:
Academic skill standards 1 – 3 have been incorporated into the rubric. These are now criteria to assess the learning outcome.
The general education learning outcome of civics and its related academic skill standards will be addressed in a separate learning outcome.
Last updated: November 17, 2015
18
Appendix D. Bloom’s Taxonomy Levels of Knowledge
Recommended