Graduate Attributes Assessment and Reporting

For 2016 Accreditation

October 2015

Accreditation 2016

• 10 of 11 Engineering programs up for accreditation in 2016

• We “must demonstrate that the graduates of a program possess

the [12] attributes” and “there must be processes in place that

demonstrate that program outcomes are being assessed in the

context of graduate attributes, and that the results are applied to

the further development of the program.”

2

Graduate Attributes

3

Graduate attributes are the qualities, skills and understandings a

university community agrees its students should develop during

their time with the institution and consequently shape the

contribution they are able to make to their profession and

society. They are qualities that also prepare graduates as agents

of social good in an unknown future. (Bowden et al., “Generic Capabilities of Australian

Technology Network University Graduates”, http:/www.clt.uts.edu.au/ATN.grad.cap.project.index.html, 2000)

Graduate Attributes - CEAB “Generic characteristics, specified by the Accreditation Board, expected to be exhibited by graduates of accredited Canadian engineering programs at the time of graduation.”

The graduate attributes broadly define the required outcomes, but they are difficult to measure directly. 4

1. KB: Knowledge base for engineering 2. PA: Problem analysis 3. IN: Investigation 4. DE: Design 5. ET: Use of Engineering tools 6. IT: Individual and teamwork 7. CS: Communication skills 8. PR: Professionalism 9. IE: Impact of engineering on society and the environment 10. EE: Ethics and equity 11. EP: Economics and project management 12. LL: Life-long learning

Indicators

• To measure graduate attributes, all universities must define their own measurable descriptors called “indicators”:

o “Descriptors of what students must do to be considered competent in

the attribute; the measurable and pre-determined standards used to

evaluate learning (i.e. measureable characteristics of attributes or

components of attributes).”

Note: We are required to “measure” indicators, not GAs

5

Complete Process

Defining Graduate Attribute

Indicators

Curriculum Mapping

Collecting Data of Student Learning

Analysis and Interpretation

Data-informed

Curriculum Improvement

6

• Core Graduate Attribute indicators are defined o Some graduate attribute indicators will ultimately be customized for each department

• Pilot projects have been undertaken to evaluate how best to report

results of Graduate Attribute assessment

• Graduate Attribute curriculum maps have been prepared for each

program

• Graduate Attribute indicators must be assessed and reported for

2015-2016 academic year (as a minimum)

Objectives

• Two principal objectives in developing accreditation

procedures for the Engineering programs:

1) Maintain the accreditation of our programs in the new CEAB

accreditation context

2) Do so with as little additional work imposed on instructors

7

Procedure • Quantitatively assess all GAs at the level of “D” (Developed) or “A”

(Applied) in a courses, in all years, for all students, by correlating

performance in “assessment tools” with that in GAs / indicators

• However, to get satisfactory data for all GAs in a given program,

departments must also o use other approaches to evaluate “soft skills” GAs (e.g., IT, CS, PR, IE, EE, EP,

LL) o employ other approaches to evaluate GAs other than just correlating GAs with

graded work

• A special emphasis on obtaining data for the “soft skills” GAs will be

undertaken in FACC 100 and FACC 400 (which are common to all

programs).

8

Procedure (cont.) • The faculty is providing general support to assist departments in

meeting the necessary requirements to obtain, analyze and

present their GA data.

• However, every course, program and department is different, so

the process cannot satisfy all situations

o Adaptation and customization will be required by each department

o CEAB guidelines and expectations are very vague

Although the faculty can provide guidance, professors and departments will sometimes need to use their judgment, as many questions do not have clear answers!

9

Overview Department

Program

General approach - infer GAs from assessment tools in courses (i.e. correlating grades with individual GAs)

Alternate approach - ensure good coverage of GAs is obtained - used to evaluate GAs by other means

Excel (analysis is the responsibility of the prof. / dept.)

myCourses (most courses, Faculty does the analysis)

10

Reporting Assessments

myCourses Gradebook Setup

Excel Format

General Approach Use myCourses for the majority of courses to correlate individual student performance in certain GAs with the various assessment tools used by the instructor

• The gradebook setup can be done centrally, if requested by the instructor

• Professors (only) to associate graduate attribute indicators with each graded item

• Professors (or TAs) are responsible for entering data (grades) in myCourses at a level that is sufficiently refined to distinguish between different graduate attribute indicators

• Analysis of the data will be done centrally

12

myCourses Setup Options Two options to setup myCourses gradebook

1) The gradebook setup can be done centrally, if requested by the instructor

o Information is submitted through a general questionnaire

2) The gradebook setup can be done by the instructor or TAs

o One-on-one consultation with professors will be available

o Hands-on TA training will be offered in mid-October

13

14

Pre-Setup Tasks • Define the role of your TA:

o http://kb.mcgill.ca/#tab:homeTab:crumb:8:artId:4131:src:article

15

Pre-Setup Tasks (cont.) • Define the gradebook rules:

Student View Display Options How to treat ungraded items

16

Information for General Setup • The assessment tools for your course • The grade distribution

17

Information for General Setup (cont.) • The grade distribution within each category:

Example:

18

Question#1 Graded out of 20 10% 4.50%Question#2 Graded out of 20 10% 4.50%Question#3 Graded out of 20 10% 4.50%Question#4 Graded out of 60 30% 13.50%Question#5 Graded out of 80 40% 18%

100% 45%

Final exam45%

Distribution based on grade points

• Even distribution

• Distribution by grade point

Assignment#1 Graded out of 20 20% 1.80%Assignment#2 Graded out of 80 20% 1.80%Assignment#3 Graded out of 80 20% 1.80%Assignment#4 Graded out of 50 20% 1.80%Assignment#5 Graded out of 100 20% 1.80%

100% 9%

Even distribution of gradesAssignments

9%

Information for General Setup (cont.)

19

Determine the Graduate Attribute indicators for each component within a graded assessment tool

General Setup

20

Step 1 – Create the assessment tool categories

Step 2 – Create the sub-categories (items) & linking the Graduate Attribute indicators

Graduate attribute indicators

Step 1 – Create Assessment Tools Categories

21

Example

22

Step 2 – Create the Sub-categories (items) & Linking the Graduate Attribute Indicators

23

Example Case1: The grade distribution for the Final Exam is based on the grade points

Case2: The grade for the Assignments category is evenly distributed between the items

24

Final Exam Question 1: • Graded out of 50 • Graduate attribute indicator: KB.7 • KB.7 - Recalls and defines information, first principles

and concepts in specialized engineering science

Question 2: • Graded out of 100 • Graduate attribute indicator: PA.2 • PA.2 - Develops models from first principles to analyze

complex engineering problems

Question 3: • Graded out of 80 • Graduate attribute indicator: PA.4 • PA.4 - Critically evaluates the validity and accuracy of

existing solution methods

Assignments Assignment 1: • Graded out of 10 • Graduate attribute indicator: KB.3 • KB.3 - Recalls and defines information, first principles

and concepts in the natural sciences

Assignment 2: • Graded out of 100 • Graduate attribute indicator: PA.1 • PA.1 - Identifies and formulates complex engineering

problems

25

Assignments Assignment 1: • Graded out of 10 • Graduate attribute indicator: KB.3 Assignment 2: • Graded out of 100 • Graduate attribute indicator: PA.1

Final Exam Question 1: • Graded out of 50 • Graduate attribute indicator: KB.7 Question 2: • Graded out of 100 • Graduate attribute indicator: PA.2 Question 3: • Graded out of 80 • Graduate attribute indicator: PA.4

Gradebook

26

Remarks

• Every course will be different, this is just an example

• Inclusion of question weights is critical to the analysis

• Prof/TA enters grades on myCourses

• Analysis of the data will be done centrally

27

Exceptional Cases - Using Excel

Excel can be used exceptionally • Courses that require more flexibility than myCourses can offer,

e.g. FACC 100, Capstone design courses

o Proper formatting is required

o Professors are responsible for calculation of grades for each

graduate attribute indicator

o Professors are responsible for the analysis of data

28

Excel Setup

• Example for the proper structure o First spreadsheet shows grades for all “assessment tools” and

the associated indicators • Similar structure as in myCourses is expected, e.g.

o Each attribute is summarized in its own individual worksheet • Each indicator is recorded and presented individually • An overall calculation for the graduate attribute is recorded

29

Format – Excel Excel spreadsheet showing grades for all “assessment tools” and the associated indicators

Different indicators are evaluated on 1 assessment

Different indicators are associated with different assessments

Different questions on the exam are associated with different indicators

Each attribute is summarized in its own individual worksheet

30

Excel - Data Calculation

A typical Graduate Attribute

Each indicator is recorded and presented individually An overall calculation

for the graduate attribute is recorded.

31

Excel – Data Analysis

32

Additional approaches • Departments will need to add to the general approach to:

o Acquire sufficient quality data for the “soft skills” graduate

attributes

o Use alternate GA evaluation approaches to satisfy (the unclear)

CEAB requirements regarding types and variety of measurement

tools

o Maria Orjuela-Laverde (Academic Associate, Pedagogical

Coordinator) is available for pedagogical assistance to develop

alternate GA evaluation methods

33

Additional approaches (cont.)

http://www.engineerscanada.ca/sites/default/files/intro_team_chair_presentation_english-sep19.pptx 34

Additional approaches (cont.)

http://www.engineerscanada.ca/sites/default/files/intro_team_chair_presentation_english-sep19.pptx 35

Recommendations • Recommendations to departments:

o Examine your curriculum maps to look for “holes” in the data

o Start by focussing on “Intro to XXX Engineering” course (or its closest

approximation) and capstone design courses to complement your general

approach for GA data acquisition

o List GA indicator codes (e.g. “KB.1,” “PA.2,” etc.) for each question/exam

sheet for all graded items (assignments, labs, exams)

o List learning outcomes on all course outlines (a requirement for the

revised course information sheets!)

o Remember to collect appropriate sample materials for your courses.

36

Overview Department

Program

General approach - infer GAs from assessment tools in courses (i.e. correlating grades with individual GAs)

Alternate approach - ensure good coverage of GAs is obtained - used to evaluate GAs by other means

Excel (analysis is the responsibility of the prof. / dept.)

myCourses (most courses, Faculty does the analysis)

37

Learning Outcomes Course Information Sheet

39 http://www.engineerscanada.ca/sites/default/files/intro_team_chair_presentation_english-sep19.pptx

Course/Lab Learning Outcomes

• “What the student is expected to have learned - These should be analogous to indicators and may have been used for GA assessment purposes. A small number of outcomes (general measurable skills acquired) are expected - not a list of content.”

• “Laboratory learning outcomes - These should be analogous to indicators and may have been used for GA assessment purposes. What are the important specific skills and techniques the student is expected to have acquired from the laboratory experience - not a list of content.”

40

41

Faculty assistance • Setting up gradebook templates in myCourses • Analyzing and processing the data acquired via myCourses

• Providing support (Nasim and Maria) to both professors and TAs • Preparing a survey for students in capstone design courses • Obtaining GA data

o in common FACC courses o via exit survey of graduating students

• Establish a faculty-wide “program improvement” procedure based on GA data

42

Misc. comments

• Some courses not on myCourses

• For a “standard” course, the process has been designed so that the

additional work required by a professor is small (as low as 2 hours

per course)

• Subsequent years will require less work (e.g. myCourses

gradebook template can be imported from previous years)

43

Resources

• Website: (https://www.mcgill.ca/engineering/faculty-staff/accreditation)

o List of indicators

• Video clips o How to setup your gradebook

• Instructions

o How to setup your gradebook

• TA Workshops

o Two sessions were offered in September o Hands-on sessions will be offered in mid-October

44

Thank you!

Keywords for Learning Outcomes

• In–Depth Knowledge • Complex Engineering Problems • First Principles • Research

Questions to consider for these GAs: • Does the course or components of the course:

o address complex engineering problems/open-ended problems? o require the use of in-depth knowledge of mathematics, natural sciences, or

engineering fundamentals? o require the use of in-depth knowledge of discipline-specific engineering? o require the development of solutions from first-principles? o require research for the development of solutions? o require research as an investigative tool?

46

The Affected Graduate attributes: KB: Knowledge Base PA: Problem Analysis IN: Problem Investigation DE: Engineering Design CS: Communication Skills