PROGRAM OUTCOME ASSESSMENT: CASE STUDY FROM THREE ENGINEERING PROGRAMS Presented by Toni L. Doolen,...
If you can't read please download the document
PROGRAM OUTCOME ASSESSMENT: CASE STUDY FROM THREE ENGINEERING PROGRAMS Presented by Toni L. Doolen, Professor School of Mechanical, Industrial & Manufacturing
PROGRAM OUTCOME ASSESSMENT: CASE STUDY FROM THREE ENGINEERING
PROGRAMS Presented by Toni L. Doolen, Professor School of
Mechanical, Industrial & Manufacturing Engineering Oregon State
University 09/21/2011
Slide 2
Abstract This presentation will introduce a framework for a
continuous improvement process for programs driven by program
outcome assessment. A case study illustrating the application of
this framework to programs in the College of Engineering will be
presented. Program outcome data collected from different sources
(courses, students, and alumni) is being used to provide feedback
to evaluate whether or not changes are needed in programs. A set of
processes has been implemented to create an annual review cycle
with a focus on continuous improvement of programs. Specific
examples and templates for program outcome assessment and reporting
will be provided.
Slide 3
Agenda Overview of the School of Mechanical, Industrial, and
Manufacturing Engineering (MIME) Assessment and Evaluation in MIME
Program Educational Objectives Student Outcomes Course Learning
Outcomes Continuous Improvement process
Slide 4
MIME Demographics ~35 full-time faculty members and instructors
~185 graduate students in three programs (industrial engineering,
material science mechanical engineering) ~1,375 undergraduates in
five programs (energy engineering management, general, industrial,
manufacturing, and mechanical) ~ $7.3 million in research funding
in 2010
Slide 5
Need for Assessment and Evaluation Undergraduate programs
accredited under ABET Many employers require accreditation Students
must graduate from accredited engineering program to sit for PE
(Professional Engineering exam)
Slide 6
Three Levels of Assessment and Evaluation Program Educational
Objectives (PEOs) Student Outcomes (SOs) Course Learning Outcomes
(CLOs)
Slide 7
Assessment Data collected from Multiple Stakeholders Students
Alumni Industrial Advisory Board Members Employers (local,
regional, national, international)
Slide 8
Levels of Assessment in MIME MIME Program Educational
Objectives (PEOs) Student Outcomes Course Learning Outcomes
Slide 9
Program Educational Objectives Program educational objectives
are broad statements that describe what graduates are expected to
attain within a few years of graduation. Program educational
objectives are based on the needs of the programs constituencies.
Program constituencies: students, faculty, employers/graduate
schools, alumni, state of Oregon,
Slide 10
PEOs A & E Process Defined for programs Our graduates will
be systems thinkers. Our graduates will be global collaborators.
Our graduates will be innovative designers and problem solvers.
Assessed and evaluated using alumni (3 5 years after graduation)
survey data, employer survey data, and IAB input. Responsibility
for A&E and CI process is MIME Head/AH, ABET coordinator, and
undergraduate program committees
Slide 11
MIME Program Educational Objectives MIME graduates will be able
to analyze, evaluate, improve, and design engineered systems and
processes using modern engineering tools (hardware and software)
and approaches. They will demonstrate in-depth knowledge of
mechanical, industrial, and/or manufacturing systems. Our graduates
will be systems thinkers. MIME graduates will be able to
communicate effectively across disciplines and cultures. They will
provide management and leadership skills within their organizations
and work effectively in diverse environments. Our graduates will be
global collaborators. MIME graduates will use both structured and
unstructured methodologies to innovate systems and processes. They
will apply technical know-how and creativity to real-world
problems. They will demonstrate a broad understanding of
engineering design and problem-solving processes. Our graduates
will be innovative designers and problem solvers.
Slide 12
PEO Assessment Data from Alumni Assessment of MIME PEOs is
completed using an annual Alumni Survey that is administered to
alumni 2 5 years after graduation. Items include specific questions
about PEOs. Rated on a 7-point scale (1=not at all; 7=extremely).
Respondents also rate importance of each PEO. Survey provides an
assessment of PEO achievement directly and also uses aggregated
Student Outcome achievement ratings provided by alumni for a second
measure. Other demographic details are collected related to
employment and general satisfaction with educational
experiences.
Slide 13
Slide 14
PEO Evaluation PEOs and assessment results reviewed with IAB
annually. PEO data analysis and recommendations prepared by ABET
coordinator. Analysis and recommendations reviewed with UPCs on an
annual basis and actions for improvement identified.
Slide 15
Linkage of PEOs to Student Outcomes Our graduates will be
systems thinkers. Ability to apply mathematics, science, and
engineering Ability to design and conduct experiments, as well as
to analyze and interpret data Ability to identify, formulate, and
solve engineering problems Ability to use the techniques, skills,
and modern engineering tools necessary for engineering practice Our
graduates will be global collaborators. Ability to function on
multi-disciplinary teams Understanding of professional and ethical
responsibility Ability to communicate effectively Knowledge of
contemporary issues Our graduates will be innovative designers and
problem solvers. Ability to design a system, component, or process
to meet desired needs within realistic constraints Broad education
necessary to understand the impact of engineering solutions in a
global and societal context Recognition of the need for, and an
ability to engage in life-long learning
Slide 16
MIME Program Educational Objectives (PEOs) Student Outcomes
Course Learning Outcomes
Slide 17
Student Outcomes Student outcomes describe what students are
expected to know and be able to do by the time of graduation. These
relate to the skills, knowledge, and behaviors that students
acquire as they progress through the program
Slide 18
Student Outcomes A&E Process General and Program-specific
student outcomes defined e.g. Assessed and evaluated using exit
surveys completed by students in the quarter they graduate and
through Course Learning Outcome mapping Responsibility for A&E
and CI process is ABET coordinator and UPC
Slide 19
Student Outcomes Examples General outcomes for all programs
Ability to design and conduct experiments, as well as to analyze
and interpret data Ability to design a system, component, or
process to meet desired needs Ability to function on
multi-disciplinary teams Program-specific outcomes Ability to
design, develop, implement and improve integrated systems that
include people, materials, information, equipment, and people
Ability to work professionally in mechanical systems area including
the design & realization of such systems
Slide 20
Student Outcome Assessment via Exit Survey Assessment of
student outcomes is completed using an annual Exit Survey that is
administered to graduates in the term that they graduate, i.e.
administered all 4 quarters. Items cover Student Outcomes that are
rated on a 7-point scale (1=not at all; 7=extremely). Survey
provides a direct assessment of program specific students outcomes
in addition to a-k. Other demographic details are collected related
to coop experiences, number of interviews, employment status, and
general satisfaction with their educational experiences (labs,
faculty, advising, facilities, classes, etc.)
Slide 21
Slide 22
Slide 23
Slide 24
Student Outcome Evaluation Process Student Outcome data
analysis and recommendations are prepared by ABET coordinator.
Analysis and recommendations reviewed with UPCs on an annual basis
and actions for improvement identified.
Slide 25
MIME Program Educational Objectives (PEOs) Student Outcomes
Course Learning Outcomes
Slide 26
Course-level Assessment and Evaluation CLOs
Slide 27
Course Learning Outcomes Course learning outcomes are
statements that describe what students are expected to know and/or
be able to do at the completion of a specific course.
Slide 28
CLOs 3 7 LOs typical for a course Examples of LOs Be able to
identify and describe (in writing) the most significant challenges
faced by engineering managers in organizations in today's global
environment. Be able to identify and facilitate solutions to
ethical dilemmas faced by engineering managers
Slide 29
Student Outcome Assessment via Course-Level Assessment and
Evaluation Faculty must also identify two other direct measures to
evaluate the achievement of Course Learning Outcomes and ultimately
of the linked Program Outcome. Specific assignments, exams, project
reports, and grading rubrics are used by faculty for direct
measures as appropriate to a given course. Faculty members are
responsible for collecting the necessary data for assessment and
for completing appropriate analyses of the data to determine
whether or not Program Outcomes have been achieved. Faculty must
also provide a narrative summary of their evaluation. This process
is completed annually for all required courses (or every other year
for courses taught on an every other year basis). Faculty members
are responsible for reporting any significant issues in meeting
Program Outcomes to the UPC. Evidence of this process is provided
through the course notebooks.
Slide 30
CLO Assessment Plan Example
Slide 31
Evaluation of Learning Outcomes
Slide 32
CLO Evaluation Example Expectation is for at least three
distinct measures for each learning outcome, one of these measures
can be student evaluation of CLO achievement for SET
Slide 33
Continuous Improvement Process
Slide 34
Continuous Improvement Cycle for all A&E Processes
PlanDoStudyAct Review & Update Desired Outcomes A&E
Performance against Outcomes Measure Impact of Changes Implement
Changes to Address Gaps
Slide 35
PEO CI Process Examples DoAct
Slide 36
Study
Slide 37
PEO CI Process Examples Plan
Slide 38
SO CI Process Study Plan Do Study Act
Slide 39
CI Process: Plan In response to the ABET reviewers concerns on
thermal design and the persistent low achievement rating of Outcome
c, the ME ABET coordinator as well as the UPC began discussions on
possibility of enhancing the design experience for students in Fall
2009. A recommendation from the coordinator and the UPC on this
matter to the ME faculty is tentatively scheduled for Fall 2010.
Plan
Slide 40
CI Process: Do Considering the low achievement rating on
Outcome c and d, ME 382 was revised in its Fall 2009 offering to
include a greater emphasis on designing real world products and
forming diverse teams (using personality and skills as metrics).
This change in project focus, while retaining the same design
methodology topics, is expected to increase student awareness of
the relevance of design to the mechanical engineering profession
through is concrete experience with real mechanical systems
(covering all domains of mechanical systems). Any potential impacts
from these revisions will be first captured in the 2011 exit
surveys (since ME382 is a junior- level course). Since Fall 2009,
ME382 has been made a required course for MfgE and an elective for
IE students. This ensures that the teams are more
multi-disciplinary. Any potential impacts from these revisions will
be first captured in the 2011 exit surveys. Do