Upload
natalie-tate
View
220
Download
0
Embed Size (px)
DESCRIPTION
Importance of Assessment Assessment is required by accrediting boards (e.g., ABET). Those with vested interests (financial and otherwise) in education are demanding accountability. Without assessment it is difficult to know what changes to make to improve learning. 3
Citation preview
Outcomes Assessment in Computer Science
Robert LingardAssessment Coordinator
August 27, 2009
1
Outline The Importance of Assessment Problems with Assessment Goals of an Assessment Process An Approach to Assessment Applying the Software Engineering Paradigm The Annual Assessment Process at CSUN Conclusions
2
Importance of Assessment Assessment is required by accrediting boards
(e.g., ABET). Those with vested interests (financial and
otherwise) in education are demanding accountability.
Without assessment it is difficult to know what changes to make to improve learning.
3
Problems with Assessment Pressure for assessment can cause a “rush to
assessment,” and produce meaningless results. Pressure to “close the loop” can cause premature
decisions regarding program changes. There is a tendency to only assess high performance
areas in order to “showcase” a program. Results are often not validated. The effects of changes are often not assessed.
4
Goals of an Assessment Process To facilitate the direct assessment of student learning To measure student retention and the ability to apply
what they have learned To provide mechanisms to ensure the continuity of
the process To be an efficient and natural extension of normal
operations To satisfy the requirements of ABET and the
University for assessment5
Approach to Assessment “The first and only goal [in education is to]
teach for long-term retention and transfer.” – Diane Halpern
The best assessments of student learning are based on direct measures of achievement.
How well students have learned is best assessed by looking at their ability to apply what they learned earlier to new situations.
6
Requirements for a Complete Assessment Process It must be comprehensive – must cover the full
range of learning outcomes It must include multiple judgments – multiple
sources of evidence must be used It must include multiple dimensions – different facets
of student performance must be included It must collect direct evidence – direct measures of
student attainment must be used
7
The Software Engineering Paradigm Understand the problem Plan a solution Carry out the plan Make sure the solution is correct
8
Understanding the Problem Before beginning assessment, the first step is
to decide what to assess – i.e., pick the most important things to assess.
Collecting information from faculty, students, alumni, and employers can give hints as to where learning problems exist.
Surveys, faculty meetings, and student interviews are ways of collecting this information.
9
Planning a Solution Once a learning outcome is selected, a plan for
conducting the assessment must be developed. It must be determined how the assessment will be
done – i.e., embedded questions in exams, a standard instrument, etc.
How the results will be evaluated must also be determined – e.g., rubrics must be developed.
It must also be determined who will do the various tasks required and when they will be done.
10
Carrying Out the Plan The steps of the plan must be carried out, and
the plan must be monitored to ensure successful completion of the assessment.
Someone must be designated as the lead, and this person has the responsibility for monitoring the plan.
11
Making Sure the Solution is Correct Assessment results obtained must be validated
– this step is often omitted. One way to validate results is to make
comparisons among several independent assessments using different approaches.
Failure to validate the results can result in program changes that may not be appropriate.
After program changes are made the effects of the changes must be assessed.
12
13
Reliability and Validity
Reliable butnot valid
Valid butnot reliable
Reliable& valid
Questions to Ask about an Assessment Was the instrument we used reliable? Did we measure what we intended to
measure? If we used a sample, was the sample size
large enough to be confident in the results? If we are comparing the results of two
assessments, are the differences significant?
14
Iteration Like software engineering this process is
iterative. Once program changes have been made, they
must be assessed to determine whether the desired result has been accomplished.
Analysis of this reassessment might indicate the need for further changes.
15
16
The Annual Assessment Process at CSUN1. Department forms Assessment Committee
The program is divided into seven areas and a coordinator is chosen for each area. These coordinators constitute the Assessment Committee.
2. Assessment Committee recommends outcomes/objectives to be assessed The Assessment considers results from previous assessments,
informal assessments, and the length of time since particular outcomes have been assessed to determine the current set to be assessed.
3. Department approves assessment goals and assessment plans are prepared
After department approval the Assessment Committee decides on an assessment approach and develops a schedule of activities.
17
The Assessment Committee Consists of the Program Area Coordinators
PROGRAM AREA COURSES PROGRAM OUTCOMES
Fundamental ConceptsRick Covington
Intro. to Algorithms & Prog. Data Structures & Prog. DesignAdvanced Data StructuresPlus Electives
Demonstrate an understanding of algorithms and data structures. Demonstrate proficiency in using a high-level computer language. Demonstrate a problem solving ability.
SystemsRobert McIlhenny
Computer Architecture Computer OrganizationOperating Systems & Sys. Arch. Plus Electives
Demonstrate an understanding of computer organization and architecture.
Language/TheoryGloria Melara
Concepts of Prog. LanguagesAutomata, Languages & Comp.Discrete MathematicsCombinatorial AlgorithmsSymbolic LogicPlus Electives
Demonstrate an understanding of programming language concepts and knowledge of a variety of programming language paradigms. Demonstrate an ability to apply mathematical skills appropriate to the computer science discipline.
Software EngineeringGeorge Wang
Intro. to Software EngineeringPlus Electives
Demonstrate proficiency in collecting, analyzing, and interpreting data. Demonstrate an understanding of emerging technologies and a working knowledge of currently available software tools. Demonstrate an understanding of the principles and practices for software design and development. Be able to apply the principles and practices for software design and development to real problems.
Societal IssuesPeter Gabrovsky
Societal Issues in ComputingPlus Electives
Demonstrate an awareness of the evolution and dynamic nature of the foundational core of computer science. Demonstrate knowledge of the social impact of computing. Demonstrate an understanding of the professional and ethical considerations of computing.
CommunicationsDiane Schwartz
Intro. to Software EngineeringSocietal Issues in ComputingPlus Electives
Be able to effectively communicate orally. Be able to effectively communicate in written form. Be able to work effectively on a team.
Lifelong LearningJack Alanen
Societal Issues in ComputingPlus Electives
Demonstrate the knowledge and capabilities necessary for pursuing a professional career or graduate studies Demonstrate the recognition of the need for, and ability for, continuing professional development.
18
The Annual Assessment Process at CSUN1. Department forms Assessment Committee
The program is divided into seven areas and a coordinator is chosen for each area. These coordinators constitute the Assessment Committee.
2. Assessment Committee recommends outcomes/objectives to be assessed The Assessment considers results from previous assessments,
informal assessments, and the length of time since particular outcomes have been assessed to determine the current set to be assessed.
3. Department approves assessment goals and assessment plans are prepared
After department approval the Assessment Committee decides on an assessment approach and develops a schedule of activities.
19
Student Learning Outcomes
1. Demonstrate an understanding of algorithms and data structures. 2. Demonstrate an understanding of computer organization and architecture.3. Demonstrate an understanding of programming language concepts and knowledge of a variety of
programming language paradigms. 4. Demonstrate proficiency in using a high-level computer language. 5. Demonstrate an ability to apply mathematical skills appropriate to the computer science discipline.6. Demonstrate an awareness of the evolution and dynamic nature of the foundational core of computer
science.7. Demonstrate proficiency in collecting, analyzing, and interpreting data and information. 8. Demonstrate a problem solving ability. 9. Demonstrate an understanding of emerging technologies and a working knowledge of currently available
software tools. 10. Demonstrate an understanding of the principles and practices for software design and development. 11. Be able to apply the principles and practices for software design and development to real world
problems.12. Be able to effectively communicate orally. 13. Be able to effectively communicate in written form. 14. Be able to work effectively on a team. 15. Demonstrate knowledge of the social impact of computing. 16. Demonstrate an understanding of the professional and ethical considerations of computing. 17. Demonstrate the knowledge and capabilities necessary for pursuing a professional career or graduate
studies.18. Recognize the need for, and show an ability for, continuing professional development.
20
Program Educational Objectives
A few years after graduation, graduates of the computer science program will:
1. Be able to apply the principles of computer science, mathematics, and scientific investigation to solve real world problems appropriate to the discipline
2. Be able to apply current industry accepted computing practices and new and emerging technologies to analyze, design, implement, and verify high quality computer-based solutions to real world problems.
3. Exhibit teamwork and effective communication skills. 4. Be able to positively and appropriately apply knowledge of societal
impacts of computing technologies in the course of career related activities.
5. Be successfully employed or accepted into a graduate program, and demonstrate a pursuit of lifelong learning.
21
The Annual Assessment Process at CSUN1. Department forms Assessment Committee
The program is divided into seven areas and a coordinator is chosen for each area. These coordinators constitute the Assessment Committee.
2. Assessment Committee recommends outcomes/objectives to be assessed The Assessment considers results from previous assessments,
informal assessments, and the length of time since particular outcomes have been assessed to determine the current set to be assessed.
3. Department approves assessment goals and assessment plans are prepared
After department approval the Assessment Committee decides on an assessment approach and develops a schedule of activities.
22
Sample Assessment Plan Learning Outcome: Be able to effectively communicate in written form
Tool(s) to be Used in the Assessment:1. The Writing Proficiency Exam (WPE)2. Term Papers written in Comp 450Assessment Activities to be Performed, When and by Whom:1. Establish minimum and average WPE score values considered to be acceptable for each version of the WPE.
(E.g., 3% better than the average scores over the past 5 years.) – May 2005, by the Software Engineering Course Group.
2. Collect the WPE scores for all Computer Science students over the last 5 years (noting which version was taken). – September 2005, obtained from Institutional Research by the Software Engineering Course Group.
3. Statistically analyze the WPE scores relative to past scores of similar students. – October 2005, by the Software Engineering Course Group.
4. Create a set of criteria (rubric) for assessing the quality of student written work, including the setting of standards for acceptability. – November 2005, by the Software Engineering Course Group.
5. Gather the term papers written by all students currently taking Comp 450. – December 2005, obtained from the Comp 450 instructors by the Software Engineering Course Group.
6. Select a sample of the term papers of graduating students and assess them according to the established criteria. – January 2006, by a committee selected by the department for the assessment of written communication.
7. Prepare an assessment summary report for the department including recommendations for program improvement. – February 2006, by the Assessment Committee.
23
The Annual Assessment Process at CSUN4. Department approves assessment plans and assessments
are conductedProgram Area coordinators ensure that the planned assessment activities are completed and that a final report is prepared
5. Assessment are analyzed and program changes are recommended
The Assessment Committee analyzes the results of the completed assessments and makes recommendations for program changes
6. Department reviews recommendations and makes program changes that are determined appropriate
At a department meeting the recommendations of the Assessment Committee are discussed and proposed program changes are made only if there is approval from the department as a whole.
24
Sample RubricLearning Outcome: Be able to effectively communicate in written form
Assessment Rubric for Written Communication
Strong2
Acceptable1
Weak0
EFFECTIVENESS OF THE THESIS:Term papers written in an academic context are expected to contain a thoughtful and insightful thesis, main idea, position, or claim that is sustained throughout the paper.
The thesis is clear, insightful and thought-provoking. It is sustained consistently throughout the paper.
The thesis is clear and plausible. It is sustained consistently throughout the paper.
The thesis is weak or absent. It is not sustained throughout the paper.
RESPONSE TO ASSIGNMENT:Papers written in an academic context are expected to address the topic and issues set forth in the assignment and address all aspects of the writing task. Usually requires some discussion and refutation of an opposing view point.
The paper responds to the assignment and addresses the topic and issues. Discussion of a counter-argument is included when appropriate.
The paper responds to the assignment and addresses the topic and issues. Some discussion of a counter-argument is included when appropriate.
The paper does not respond to the assignment or treats the assignment in a superficial, simplistic, or disjointed manner. Little or no discussion of a counter-argument in included.
SUPPORT:Papers written in an academic context are expected to provide support for main points with reasons, explanations, and examples that are appropriate for intended audience.
The thesis is fully and convincingly developed, supported with good reasons, explanations and examples.
The thesis is adequately developed, supported with reasons, explanations, and examples.
The thesis is inadequately developed, unsupported with reasons, explanations, and examples.
ORGANIZATION:Papers written in an academic context are expected to be well-organized, in both overall structure & paragraphs.
The paper is well-structured; its form contributes to its purpose. Paragraphs are well-organized and carefully linked to the thesis.
The paper is generally well-structured, with only a few flaws in overall organization. Paragraphs are adequately organized and generally linked to the thesis.
The paper is poorly structured; organizational flaws undermine its effectiveness. Paragraphs are not well organized; nor are they linked to the thesis.
STYLE;Papers written in an academic context are expected to be stylistically effective – that is, to contain well-structured sentences, well-chosen words, and an appropriate tone, as a means of achieving its purpose.
The sentence structure, word choice, fluency, and tone of the paper enhance its effectiveness and reinforce its purpose.
The sentence structure, word choice, fluency, and tone of the paper contribute to its effectiveness and adequately support its purpose.
The sentence structure, word choice, fluency, and tone of the paper detract from its effectiveness or are inappropriate to its purpose.
GRAMMAR AND MECHANICS:Papers written in an academic context are expected to maintain sentence level correctness in terms of syntax, grammar, spelling, punctuation, and format.
The paper is correct in terms of its syntax, grammar, spelling, punctuation, and format.
Sentence level errors do not seriously detract from the paper’s effectiveness.
Sentence level errors are so frequent and disruptive that they detract from the paper’s effectiveness. 25
The Annual Assessment Process at CSUN4. Department approves assessment plans and assessments are
conductedProgram Area coordinators ensure that the planned assessment activities are completed and that a final report is prepared
5. Assessment are analyzed and program changes are recommended
The Assessment Committee analyzes the results of the completed assessments and makes recommendations for program changes
6. Department reviews recommendations and makes program changes that are determined appropriate
At a department meeting the recommendations of the Assessment Committee are discussed and proposed program changes are made only if there is approval from the department as a whole.
26
Sample Assessment ResultsLearning Outcome: Demonstrate an understanding of the principles and practices for software design and development
0
2
4
6
8
10
12
14
450
580
595VAV
595SEM
450 2 10 9 0
580 3 7 9 2
595VAV 3 13 8 1
595SEM 1 4 2 0
Unacceptable Marginal Adequate Excellent
27
The Annual Assessment Process at CSUN4. Department approves assessment plans and assessments are
conductedProgram Area coordinators ensure that the planned assessment activities are completed and that a final report is prepared
5. Assessment are analyzed and program changes are recommended
The Assessment Committee analyzes the results of the completed assessments and makes recommendations for program changes
6. Department reviews recommendations and makes program changes that are determined appropriate
At a department meeting the recommendations of the Assessment Committee are discussed and proposed program changes are made only if there is approval from the department as a whole.
28
The following recommendations for program improvement were made based on the results of the assessment:
1. Modify the course objectives for introductory computer science courses to include an introduction to software engineering concepts.
STATUS: This was approved by the department and the members of the Software Engineering Course Group will provide information to the Fundamental Concepts Course Group to assist them in appropriately revising the course objectives for the introductory courses.
2. Modify the course objectives for elective courses with software engineering projects to include the reinforcement of software engineering concepts.
STATUS: This was approved by the department and the members of the Software Engineering Course Group will provide information to those teaching elective courses to assist them in appropriately revising the course objectives to reinforce software engineering concepts.
3. Add a senior software engineering design project as a require for graduation.STATUS: After extensive discussions by the department regarding ways of implementing such a requirement, a committee was formed to develop a recommended approach and report back to the department for approval.
Sample Program ImprovementsLearning Outcome: Demonstrate an understanding of the principles and practices for software design and development
29
Conclusions The process has been used successfully for program
assessment. It has been embedded into the normal operations of the
department and become accepted by faculty. Since all faculty are involved, no one has become
overburdened. It has provided an effective means for directly assessing
student learning with a focus on retention and transfer. More attention is needed in the areas of validating results and
ensuring the reliability of the assessment instruments used.
30