Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

1

June 10, 2009

University of XYZ

• ?? Programs to be evaluated

• ?? member team– ?? observers, ?? co-chairs, ?? PEV’s

• We’ll need to stay focused!

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

2

June 10, 2009

Important Team Expectations• Evaluators represent the EAC of ABET• We are accrediting programs, certifying that

they satisfy the criteria• Team effort—team decisions• Confidentiality• Conflict of interest—every visitor should have

signed a conflict-of-interest statement• Observers

– no evaluative statements– exit interview—thank-you only

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

3

June 10, 2009

Confidentiality• Do not discuss conclusions with faculty,

students, and others• Keep all materials until the July 2010 EAC

meeting. At conclusion of accreditation process (July 2010) materials are to be destroyed

• Information specific to the institution is to remain confidential without time limit

• Institutional data are confidential except with written authorization of institution

• ABET materials only released by ABET staff

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

4

June 10, 2009

Communication

• Maintain open line of communication with the department head

• Identify deficiencies as soon as possible• Discuss all issues with the department head at

the debriefing • Do not discuss the recommended accreditation

action with anyone except team members

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

5

June 10, 2009

General Visit Outline• Today –

– Visit campus, review curriculum materials– Tonight share what we’ve learned, plan

• Tomorrow – – AM: Dean’s overview, PEV’s in departments– Lunch with institutional officials– PM: Support areas, PEV’s in departments– Evening: Share what we’ve learned, tentative conclusions;

draft exit statements• Tuesday –

– Tie up loose ends– Draft exit statements by late morning– Debrief department heads before lunch– Exit meeting at 2PM

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

6

June 10, 2009

Support Program AssignmentsBiology 1:30 Mon

Chemistry 1:30 Mon

Comp Sci 1:30 Mon

English 1:30 Mon

Math 1:30 Mon

Physics 1:30 Mon

Statistics 1:30 Mon

Gen Ed 1:30 Mon

Library 1:30 Mon

Office of Info Tech 1:30 Mon

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

7

June 10, 2009

Visit details• Important forms – done by Tues AM

– Transcript & curriculum analyses – Program Audit Form + Explanation of Shortcoming (two

hard copies to Team Chair—one will be left with Dean after Summary Meeting with Administration—one soft copy to Team Chair)

– Exit Statement—soft copy to Team Chair– Program Evaluator Worksheet—soft copy to Team Chair– Form A2a - Short form—recommended actions—

prepared by Team Chair on basis of information from evaluators

– Program Evaluator Report—soft copy to Team Chair

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

8

June 10, 2009

Terminology• Deficiency – criterion, policy, or procedure is

NOT satisfied.• Weakness – criterion, policy, or procedure is

satisfied, but lacks strength of compliance to assure the quality of the program will not be compromised prior to next general review.

• Concern – criterion policy, or procedure is satisfied, but potential exists for non-satisfaction in the near future.

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

9

June 10, 2009

Working Definition of Key Terms

• Deficiency: assigned to any criterion, policy, or procedure that is totally or largely unmet

• Weakness: criterion, policy, or procedure is met to some meaningful extent, but compliance is insufficient to fully satisfy requirements

• Concern: criterion, policy, or procedure is fully met, but there is potential for non-compliance in the near future

• Observation: general commentary possibly, but not necessarily, related to criteria

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

10

June 10, 2009

Limit Use of Key Terms• Use Key Term only in reference to overall

evaluation of each criterion

• The Key Term (defined in the previous slides) is the overall assessment for the criterion as a whole, not the worst finding among the sub-areas on the worksheet. Do not, for example, give a deficiency to a program that lacks only a measurement for outcome 3(e)

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

11

June 10, 2009

Consistency Issues for the Team

• Depth and completeness of the evaluation from program to program

• Consistency across all programs in an institution

• Assignment of appropriate key terms (deficiency, weakness, concern) to describe shortcomings

• For weaknesses, consistency on interim recommendations—IR vs. IV

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

12

June 10, 2009

Applicable Criteria• Use Criteria for 2009-2010 Accreditation

Cycle. These are the same as the renumbered criteria used in 2008-2009

• If this is an interim visit, the criteria in effect at the time the prior shortcomings were identified should be used, unless it is to the program’s benefit to use the currently applicable ones, and it is the program’s choice.

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

13

June 10, 2009

Application & Interpretation of Criteria

In 2008-09, EAC editors observed that:

• Issues arose related to the linkage between Criterion 2, Criterion 3, and Criterion 4

• Criterion 2 and new programs

• Criterion 4 and new programs

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

14

June 10, 2009

What Does Criterion 2 Say?

The program must have in place:• Published PEO’s consistent with mission and

these Criteria • Process that periodically documents and

demonstrates that the PEO’s are based on the needs of the program’s constituencies

• An assessment and evaluation process that periodically documents and demonstrates the extent to which these objectives are attained

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

15

June 10, 2009

Criterion 2 Highlights1. The process needs to document and

demonstrate that the PEO’s are based on constituent needs – NOT “a process based on the constituents needs in which PEO’s are determined and evaluated” (the old language)

2. Requirement for assessment and evaluation is the same, but there is no longer language in Criterion 2 that requires that the results of the assessment and evaluation process be used for program improvement. (Program improvement is now in Criterion 4.)

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

16

June 10, 2009

Consistency IssuesCriterion 2:

– Do the published PEO’s meet the definition?– Does the program convince the team that the PEO’s

are based on constituent needs?• Notice that there is no language that insists on constituent

approval or involvement! • The program does need to convince the team that it has a

way of determining what the needs of its constituencies are.

– Is there an assessment and evaluation process in place that gives info about the extent to which PEO’s are attained by grads

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

17

June 10, 2009

Criterion 2 Problem in 08-09• Several instances in which programs were being

required to show how results of C2 or C3 assessment and evaluation processes were being used to improve the program.

• This reflects application of old criteria.

• That's not in C2 or C3 any more. C4 refers to results of C2 and C3 processes as possible sources of information upon which continuous improvement could be based.

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

18

June 10, 2009

Criterion 2 FAQ’s• What if the PEO’s really sound like outcomes

(instead of objectives?– If PEO’s are not PEO’s, there is a C2 shortcoming.

• What if PEO’s are ambiguous or reflect outcomes retooled to apply after graduation?– Team judgment – do they meet the intent of the

Criterion?• Is an assessment process for PEO’s that

considers predominately data based on accomplishments of current undergraduates adequate? – Probably not

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

19

June 10, 2009

What Does Criterion 3 Say?

• The program must demonstrate that (a) – (k) are attained

• Program outcomes are defined as (a) – (k) plus any additional ones articulated by the program

• Program outcomes must foster attainment of the PEO’s

• There must be an assessment and evaluation process that periodically documents and demonstrates the degree to which outcomes are attained

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

20

June 10, 2009

Important – for Criterion 3• The definition of program outcomes as being (a) – (k)

plus locally articulated ones– The program may not have its outcomes expressed as (a) – (k)

plus others. It may have just identified a set of outcomes. As long as the program has demonstrated attainment of (a) – (k) and its own outcomes, this element of the criterion is met.

• Requirement for assessment and evaluation is the same, but there is no language in Criterion 3 that results of the assessment process be applied to further development of the program. (Program improvement is now in Criterion 4.)

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

21

June 10, 2009

Criterion 3 Consistency Issues• Be sure to apply this criterion in a holistic sense• The process of assessment and evaluation needs to

demonstrate the degree to which outcomes are attained, but …– There is no language that says all outcomes must be

attained to the same degree– There is no language that says anything about a numeric

scale measuring degree of attainment– There is no language that says the outcomes must be

measured– There is nothing in Criterion 3 that says anything about

use of the assessment and evaluation information for program improvement

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

22

June 10, 2009

Criterion 3 FAQ’s

• What about assessment data? What is adequate data? – Does it all have to be objective/direct? (NO)– Can it be subjective? (Some of it may be; nothing

says it has to be)– Is the observation or conclusion of course instructor

adequate? (What was his or her basis for the observation?)

– Does evidence for each outcome have to be in the form of work the student has produced? (No, but the team needs to be convinced that outcome attainment has been demonstrated.)

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

23

June 10, 2009

What Does Criterion 4 Say?

“Each program must show evidence of actions to improve the program. These actions should be based on available information, such as results from Criterion 2 and Criterion 3 processes.”

• The improvements can be based on any available information!

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

24

June 10, 2009

Consistency Issues• The language of Criterion 4 simply insists on

evidence of action to improve the program.– Such actions could be stimulated by results of the C2

and C3 assessment and evaluation processes– But they could also be stimulated by other information

• The language of this Criterion does not require that the C2 and C3 information be used as the basis for program improvement. It suggests use of the results of C2 and C3 processes as sources of information for program improvement.

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

25

June 10, 2009

Deliverables

• From each PEV, the team chair must have:– A recommended action relative to the program under

review that is consistent with the team’s conclusions (remember, it is a TEAM recommendation)

– A PAF that accurately reflects the team findings– An Exit Statement that will:

• Be read verbatim in the Exit Meeting• Be used as the basis for construction of the draft statement

to the institution

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

26

June 10, 2009

Program Audit Form (for GR or New)(We provide a copy to the institution at the Exit Meeting)

If heading doesn’t have the current visit year, you are using an old form!

Do not use old forms!!

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

27

June 10, 2009

Program Audit Form (for Interim Report Evaluations)

We provide a copy to the institution at the Exit Meeting for IV only

If the heading does not say

INTERIM REPORT

do not use it for an IR evaluation

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

28

June 10, 2009

Program Audit Form (for Interim Visit Evaluations)

We provide a copy to the institution at the Exit Meeting for IV only

If the heading does not say

INTERIM VISIT

do not use it for an IV evaluation

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

29

June 10, 2009

Exit Statement Format• INTRODUCTION—USEFUL

PROGRAM STATISTICS• PROGRAM ISSUES

– Strengths (special, unique or particularly conspicuous strengths standing above the norm)

1.

2.

– Deficiencies (In order, only for those criteria where deficiencies exist)

1. XXX

2. etc.

– Weaknesses (In order, only for those criteria where weaknesses exist)

1. YYY

2. etc.

– Concerns (In order, where concerns exist)

1. ZZZ

2. etc.

– Observations (do not have to relate to criteria)

1. etc.

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

30

June 10, 2009

Writing the Exit Statement• For each shortcoming – use the following

structure for your statement of finding:– What is required:

• Describe what criterion or policy is applicable• Describe what that criterion or policy requires

– What was observed:• Describe what was observed• Describe how your finding violates criterion or policy

– Negative Impact• Describe the negative impact it has on the program (for D

or W) or the current or potential future impact on the program (C)

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

31

June 10, 2009

What does your TC need in the exit statement you write?

• The language must be clear:– “There is inadequate assessment of outcome 3(j).”

• How does the program know why its assessment of student knowledge of contemporary issues is inadequate?

– “There is incomplete evidence that students attain outcome 3(i).”• What is it that is incomplete about the evidence that students

have a recognition of the need for and the ability to engage in life-long learning.

– “It appears that outcome 3(h) is not fully assessed.”• Why did the team find that the broad education necessary to

understand the impact of engineering solutions in a global, economic, environmental, and societal context is not fully assessed? What does “full assessment” mean?

• The program has to understand what the shortcoming really is in order to fix it, the TC has to be able to explain the team recommendation, and later teams have to understand the issues

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

32

June 10, 2009

What Would Make it Clearer?• Instead of “There is inadequate assessment of outcome 3(j).”

– The only mechanism used for assessment of student knowledge of contemporary issues is through administration of a survey instrument asking students whether they have knowledge of contemporary issues. This mechanism does not provide adequate information for determining the degree to which this program outcome is attained.

• Instead of “There is incomplete evidence that students attain outcome 3(i).”– Although student grades in the “Introduction to the Profession” course

were claimed as documentation and demonstration that students have recognition of the need for and an ability to engage in life-long learning, no clear links between the student grades and this outcome were established.

• Instead of “It appears that outcome 3(h) is not fully assessed.”– Even though some anecdotal assessment of whether students do have

the requisite broad general education was described in the self-study report, evidence shows only preliminary plans for the development of a systematic process for assessment and evaluation of this program outcome.

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

33

June 10, 2009

The language must support the team recommendation

• If it is a D, the words have to clearly say that the criterion is totally or largely unmet

• “there is no evidence that”• “there is no assessment and evaluation process”• “not all students are required to engage in a major

design experience”

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

34

June 10, 2009

The language must support the team recommendation

• If it is a W, the words have to clearly say that the criterion is met but that strength of compliance is lacking, and how it is lacking needs to be clear– “Assessment of outcomes 3(d), 3(f), 3(g), 3(h), and 3(i) appears

to be ad hoc. Systematic assessment and evaluation of these outcomes would strengthen compliance with this criterion.” [supporting a recommended citation of a Criterion 3 weakness relative to strength of assessment and evaluation process]

– “While the program has demonstrated that graduates are able to engage in most of the activities required by the Civil Engineering program criteria, the program has not clearly demonstrated that graduates are able to apply knowledge of physics, chemistry, and at least one additional area of science.” [supporting a recommended citation of a Criterion 9 weakness for a Civil program]

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

35

June 10, 2009

The language must support the team recommendation

• If it is a C, the words have to clearly say that the criterion is met and indicate exactly what leads to the concern about the potential for future noncompliance– “It is clear that all of the criterion 3 outcomes are being assessed,

but evaluation of some of the assessment data appears to be inconsistent. Unless evaluation of the extent to which outcomes are attained is carried out on a consistent basis, future compliance with this criterion may be jeopardized.” [supporting a recommended citation of a criterion 3 concern]

– “Although all of the transcripts examined provided evidence that students meet all graduation requirements, the processes by which graduation requirements are audited prior to graduation appears to be ad hoc. This leads to concern that future compliance with this criterion may be jeopardized.” [supporting a recommended citation of a criterion 1 concern]

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

36

June 10, 2009

The PEV Competency Modeland its Application

• Your performance as a PEV will be evaluated against the PEV Competency Model. You are expected to be:– Technically current– Effective at communicating– Interpersonally skilled– Team-oriented– Professional– Organized

• The evaluation forms are found on the ABET web sitehttp://www.abet.org/pev.shtml

Cop

yrig

ht ©

200

9 by

AB

ET

, In

c.

37

June 10, 2009

Last Words

• Remember – this team is “the face of ABET”

• The 4 big C’s go a long way in promoting a productive evaluation:– Courtesy – Consistency– Clarity– Confidentiality