Program Assessment Process for Continuous Quality Improvement (CQI)

Program Assessment Process for Continuous Quality Improvement

(CQI)

Prepared by:Abul Fazal M. Arif

Member, ABET Steering CommitteeNovember 16, 2008.

ABET Steering Committee Program Assessment Process 2

Contents

ABET General Criteria, 2008-2009 Cycle Assessment Process – key components Tasks to be Done Closing the Loop Action Plan


ABET GENERAL CRITERIA

“Renumbered” Criteria:(2008-2009 Accreditation Cycle)

Criterion 1. Students Criterion 2. Program Educational ObjectivesCriterion 3. Program OutcomesCriterion 4. Continuous ImprovementCriterion 5. CurriculumCriterion 6. FacultyCriterion 7. FacilitiesCriterion 8. SupportCriterion 9. Program Criteria


Assessment Process for Quality Assurance

Criterion 1. Students Criterion 1. Students

Criterion 2. Program Educational ObjectivesCriterion 3. Program OutcomesCriterion 4. Continuous ImprovementCriterion 5. CurriculumCriterion 5. CurriculumCriterion 6. FacultyCriterion 6. FacultyCriterion 7. FacilitiesCriterion 7. FacilitiesCriterion 8. SupportCriterion 8. SupportCriterion 9. Program CriteriaCriterion 9. Program Criteria

Assessment is an ongoing process of setting high expectations for student learning, measuring progress toward established learning outcomes, and providing a basis for reflection, discussion and feedback to improve University academic programs. It is a systematic and cyclic process that makes expectations and standards explicit and public.


Key Components of Assessment Process

Performance Criteria

Assessment Methods Evaluation

Curriculum Mapping

Closing the Loop Action Taken / To be Taken

Program Learning Outcomes

Program Educational Objectives




Curriculum Mapping




Educational objectives are statements that describe the expected accomplishments of graduates during the first few years after graduation—usually 3-5 years.




Curriculum Mapping




Assessment Process for Quality Assurance

These are statements related to skills, knowledge and behaviors that describe what students are expected to know and/or be able to do by the time of graduation. If students have achieved these outcomes, it is anticipated that they will be able to achieve the educational objectives after graduation.


Assessment Process (cont.)

At the heart of assessment are the Program Objectives and Outcomes. Purpose: To assess the extent a program is

achieving its objectives and outcomes. To be conducted continuously at different stages

continuous improvement.

The assessment process itself should be evaluated and refined in light of emerging insights.

All accrediting bodies, including ABET, require evidence of continuous quality improvement of both, Process and Program.


Tasks to be done

Task 1: Review Program Educational Objectives (PEO)

Task 2: Review Program Learning OutcomesTask 3: Align Learning outcomes with PEOTask 4: Present to ConstituentsTask 5: Get Approval & PublishTask 6: Mapping Course Outcomes to Program

OutcomesTask 7: Develop Performance Indicators Task 8: Identify Assessment Tools/MethodsTask 9: Develop Assessment PlanTask 10: Data Collection and EvaluationTask 11: Closing the Loop


TASK 1: Review Program Educational Objectives (PEO)

Review process will be initiated by Department ABET Committee or PAC

Guidelines for review

1. PEO should be consistent with proper definition.

2. PEO should be aligned with mission of university/College/Department.

3. Numbers of PEO should be manageable.4. PEO should be measurable.


STEP 1: Revise Program Educational Objectives

Examples of PEO Prepare graduates who will be successful in

mechanical engineering-related careers and other diverse career paths

Prepare graduates to be leaders Prepare graduates to be good citizens of

the society Prepare graduates to pursue advanced

degree, if so desired Prepare graduates to be life long learners Prepare graduates to be entrepreneurs


TASK 1: Review Program Educational Objectives

Examples of PEO - Combined Prepare graduates who will have successful

careers, and become leaders, in industry and the public sector

Prepare graduates who will appropriately apply acquired knowledge, work well in team, effectively communicate ideas and technical information, and continue to gain knowledge

Prepare graduates who will continue professional development and will pursue continuing education opportunities relevant to their careers

Note: For more examples of PEO, refer to Aggarwal’s workshop notes.


TASK 2: Review Program Learning Outcomes

Task 2.1: Engineering programs must demonstrate

that their students attain the following (a) to (k)outcomes:

(a) an ability to apply knowledge of mathematics, science, and engineering

(b) an ability to design and conduct experiments, as well as to analyze and interpret data

(c) an ability to design a system, component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability

(d) an ability to function on multidisciplinary teams


Task 2: Review Program Learning Outcomes (cont.)

(e) an ability to identify, formulate, and solve engineering problems

(f) an understanding of professional and ethical responsibility

(g) an ability to communicate effectively(h) the broad education necessary to understand the

impact of engineering solutions in a global, economic, environmental, and societal context

(i) a recognition of the need for, and an ability to engage in life-long learning

(j) a knowledge of contemporary issues(k) an ability to use the techniques, skills, and modern

engineering tools necessary for engineering practice


Task 2: Review Program Learning Outcomes (cont.)

Task 2.2: Check if any additional outcomes that may be articulated by the program criteria.

Check Criterion 9 on Program Criteria from ABET 2008-2009, (For Mechanical Engineering)Mechanical Engineering program criteria include the requirement that the program must demonstrate that graduates have the ability to: apply principles of engineering, basic science, and mathematics (including multivariate calculus and differential equations) to model, analyze, design, and realize physical systems, components or processes; and work professionally in both thermal and mechanical systems areas.


TASK 3: Align Learning outcomes with Program Objectives. Program outcomes must foster attainment

of program educational objectives. Link outcomes with PEO All Learning Outcomes may or may not

link to all/any PEOProgram Learning Outcomes

PEO

(a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k)

1 X X X

2 X X X

3 X X X

4 X X X

5 X X X


TASK 4: Present to Constituents

Main Stakeholders/Constituents Faculty through Department Council Employers through Industrial Advisory

Committee Students through various Professional Groups

or Student Societies etc.Others: Administration Alumni Other Graduate Schools Recruiters Employers

Note: Mostly faculty define Program Learning Outcomes


TASK 5: Get Approval & Publish

Get Approval Final form of the Program Educational

Objectives and Learning Outcomes should be approved by the Department Council

Keep record in “ABET File”

Publish at: Department Website General Catalog Bulletin Boards etc

Keep minutes of meeting of constituent’s involvement and Council approval as evidence in record.




Curriculum Mapping




Curriculum Mapping is the alignment of the program learning outcomes with the curriculum and, therefore, it needs development of performance indicators based on the curriculum.




Curriculum Mapping




TASK 6: Mapping of Course Learning Outcomes to Program Learning Outcomes Include only Core (required) courses Select courses for Outcome Assessment


Mapping: Course Learning Outcomes and Program Outcomes


Mapping: Core Courses & Program Outcomes




Curriculum Mapping




TASK 7: Develop Performance Indicators for each Program Outcome




Curriculum Mapping




Performance Criteria (indicators, PIs) are defined as the specific, measurable statements identifying the specific knowledge, skills, attitudes and/or behavior students must demonstrate as indicators of achieving the outcomes.


Examples of Performance Indicators Outcome (a): Ability to apply knowledge of

mathematics………

1. Applies mathematics principles to obtain solutions.

2. Uses principles of sciences and engineering in solving engineering problems.

3. Combines scientific and engineering principles to formulate models of processes and systems.

4. Combines mathematics principles to formulate models of processes and systems.


Examples of Performance Indicators Outcome (a): Ability to apply knowledge of

mathematics………1. Applies mathematics principles to obtain Applies mathematics principles to obtain

solutions.solutions.2. Uses principles of sciences and engineering Uses principles of sciences and engineering

in solving engineering problems.in solving engineering problems.3. Combines scientific and engineering Combines scientific and engineering

principles to formulate models of processes principles to formulate models of processes and systems.and systems.

4. Combines mathematics principles to Combines mathematics principles to formulate models of processes and formulate models of processes and systems.systems.

Note: For more examples of PI, refer to Sub-Committee Report.

Generally, it is recommended to limit performance criteria to three-four for each outcome.




Curriculum Mapping



Program Educational ObjectivesAssessment is one or more

processes that identify, collect, and prepare data to evaluate the achievement of program outcomes and program educational objectives.


TASK 8: Identify Assessment

Tools/Methods. Assessment tools are of two types:

Direct measures are based on facts (it also includes opinion based on facts).

Indirect measures are based on opinion only.

Multi-source or multi-method approach is generally recommended for data collection and assessment. Indirect measures are necessary but not sufficient. They

should be complemented with at least one direct measure.

Sometimes it may not be possible to use direct methods to measure some of the outcomes (for example team work). In those cases, we can rely on indirect methods.


Task 8: Identify Assessment Tools (cont.)

Examples of Direct measures

1. Student course portfolios and senior capstone course projects using overall score of learning outcome rubric

2. External review or performance appraisal through direct observation on internship

3. Locally developed exams.4. Oral exams 5. Standardized exams such as FE, GRE, GMAT6. Special student achievements (awards, student

publications and presentations at various meetings)

7. External examiners8. Graduate schools


Learning Outcome RubricPerformanc

e Criteria

Unsatisfactory1

Developing2

Satisfactory3

Exemplary4

Score

Contribute

Research & gather information.

Does not collect any

information that relates to the topic.

Collects very little

information – some

relates to the topic.

Collects some basic

information – most

relates to the topic

Collects a great deal of

information – all relates to the topic

Take Responsibility

Fulfill team role’s duties.

Does not perform any

duties of assigned

team role.

Perform very little duties.

Perform nearly all duties.

Perform all duties of assigned

team role.

Shear equally

Always relies on others to do

the work.

Rarely does the assigned

work – often needs

reminding.

Usually does the assigned work –

rarely needs reminding.

Always does the assigned work – without

having to be reminded.

Value Other’s Viewpoints

Listen to other

teammates.

Is always talking – never allows

anyone else to speak.

Usually doing most of the

talking – rarely allows

others to speak.

Listens, but sometimes

talk too much.

Listen and speaks a

fair amount

Average

Rubrics are developed from performance indicators and consist of sets of categories that define the process towards meeting the goal.


Example of Rubric on TeamworkPerformanc

e Criteria

Unsatisfactory1

Developing2

Satisfactory3

Exemplary4

Score

Contribute

Research & gather information.

Does not collect any

information that relates to the topic.

Collects very little

information – some

relates to the topic.

Collects some basic

information – most

relates to the topic

Collects a great deal of

information – all relates to the topic

Take Responsibility

Fulfill team role’s duties.

Does not perform any

duties of assigned

team role.

Perform very little duties.

Perform nearly all duties.

Perform all duties of assigned

team role.

Shear equally

Always relies on others to do

the work.

Rarely does the assigned

work – often needs

reminding.

Usually does the assigned work –

rarely needs reminding.

Always does the assigned work – without

having to be reminded.

Value Other’s Viewpoints

Listen to other

teammates.

Is always talking – never allows

anyone else to speak.

Usually doing most of the

talking – rarely allows

others to speak.

Listens, but sometimes

talk too much.

Listen and speaks a

fair amount

Average


Task 8: Identify Assessment Tools (cont.)

Examples of Indirect measures

Questionnaires and written surveys (Alumni and employer surveys)

Senior exit surveys and interviews Student course evaluations Grades ** Focus groups conducted by trained

moderator Students internship evaluations (if based on

opinion)

** Refer to Sub-Committee Report or Aggarwal Workshop Notes.


TASK 9: Development Assessment Plan

Selection of Courses Data Collection Cycle Identify Data To be Collected


Mapping: Core Courses & Program Outcomes


Selection of Courses for Outcome Assessment


Selected Courses for Outcome Assessment


Data Collection Cycle

Outcome

Year 1

Year 2 Year 3 Year 4 Year 5 Year 6

(a) X X

(b) X X

(c) X X

(d) X X

(e) X X

(f)

(g) X X

(h) X X

(i) X X

(j) X X

(k) X X


Data Collection Cycle

Outcome

Year 1

Year 2 Year 3 Year 4 Year 5 Year 6

(a) X X

(b) X X

(c) X X

(d) X X

(e) X X

(f)

(g) X X

(h) X X

(i) X X

(j) X X

(k) X X

Learning Outcomes are measured on a continuous basis (3 years cycle is acceptable). You do not have to measure every student for every outcome.

For PEOs, typically data is collected every three years from all constituents for every objective.


Data to be collected

For Program Learning Outcomes Students Course Evaluation using rubric Student Course Portfolios Grades Performance Appraisal Senior exit surveys and interviews Locally developed exams

For Program Educational Objectives Alumni Survey Employers Survey Industrial Advisory Board’s Feedback Placement Data


TASK 10: Collection of Data & Evaluation




Curriculum Mapping




Evaluation is one or more processes for interpreting the data and evidence accumulated through assessment practices. Evaluation determines the extent to which program outcomes or program educational objectives are being achieved, and results in decisions and actions to improve the program.


Example of Evaluation

Evaluation using EXCEL file


TASK 11: Closing the LOOP




Curriculum Mapping




The feedback process is critical to creating and maintaining a systematic quality assurance system. When successfully implemented, all elements of the quality assurance process interact with one another.




Curriculum Mapping



Program Educational Objectives EXAMPLE of

Assessment Plan, Report & Rubrics


ABET Accreditation: key Dates

Task Responsibility Date

1. Complete the form for Programs Evaluation

Academic Departments 31-12-2008

2 Send Request for Programs Evaluation To ABET HQ

Dean Of Engineering Office

15-1-2009

3 Draft Self-Study Reports Ready


4 Forward Draft Self-Study Reports to Quality Assurance Committee & English Editing


9-4-2009

5 Receipt of Reviewed and Edited Reports

Quality Assurance Committee & ELC

9-5-2009

6 Final Self Study Report Ready


7 Preparation of the final Reports


15-6-2009

8 Dispatch The Final Reports to ABET HQ


20-6-2009


Tasks to be done

Task 1: Review Program Educational Objectives (PEO)

Task 2: Review Program Learning OutcomesTask 3: Align Learning outcomes with PEOTask 4: Present to ConstituentsTask 5: Get Approval & PublishTask 6: Mapping Course Outcomes to Program

OutcomesTask 7: Develop Performance Indicators Task 8: Identify Assessment Tools/MethodsTask 9: Develop Assessment PlanTask 10: Data Collection and EvaluationTask 11: Closing the Loop


Task 3: Draft Self-Study Reports ReadyBy: Academic DepartmentsSubmission date: 1-4-2009

Task 16-30 Nov

1-15 Dec

16-31 Dec

1-15 Jan

16-31 Jan

1-15 Feb

16-28 Feb

1-15 Mar

16-31 Mar

1

2

3

4

5

6

7

8

9

10

11


Sources of Information

http://www.abet.org/forms.shtml Aggarwal’s workshop # 1 lectures Aggarwal’s workshop # 2 lectures ABET Sub-Committee Report


THANK YOU

Documents

Program Assessment Process for Continuous Quality Improvement (CQI)