Preparing for PBL

Donald R. WoodsMcMaster UniversityHamilton, ON, Canada

May 2003;

Second edition, Nov 2004

Third edition, Mar 2006

Preparing for PBL 1. Introduction to PBL:

1.1 What is PBL? 1-11.2 Why select PBL? the claims, 1-21.3 Experience PBL, 1-2 1.4a MRIQ, 1-41.4b Selecting a form of PBL: are you ready? 1-61.5 Selecting a form of PBL: background for selecting , 1-6

1.5-1 Thirty ideas for active engagement, 1-7 1.5-2 Seventeen ideas for cooperation, 1-71.5-3 Fifteen ideas for prompt feedback, 1-81.5-4 Fourteen ideas for empowering students, 1-9

1.6 Selecting a form of PBL: moving gentling from lecture toward PBL, 1-9 1.7 Implementing PBL: possible issues, 1-91.8 How to create problems: example, and criteria, 1-101. 9 Options for assessment of student performance, 1-111.10 Developing process skills, 1-111.11 Perry’s check on student attitude, 1-111.12 Plan for success, 1-111.13 Practical downsides.. and how to solve them, 1-11

2. Example PBL: Linda’s complaint2.1 What are the issues in this scenario?, 2-12.2 Reporting back on selected issues, 2-42.3 Use the new knowledge to solve a problem, 2-112.4 Elaborate the knowledge, 2-122.5 Summarize, 2-12Forms: Learning goals for “team skills”, 2-16

Target skills for group skills, 2-17Quality of self assessment, 2-18Reflective self rating of ideas, 2-19

3. Adopting PBL for classes with > 5 students: a “large class”3.1 What is PBL?3.2 Options: general & Options especially appropriate for large classes 5 - 5003.3 Osterman feedback lecture3.4 Branda’s PBEE3.5 Socratic3.6 Guided design3.7 Tutorless groups: challenges in working with tutorless groups3.8 Making students accountable3.9 Developing the process skillsExample feedback Forms:

Self monitoring skill development, 3-11Quality of self assessment, 3-13 (2-18)Target skills for problem solving, 3-14Self rating of stress management, 3-15Target skills for group work, 3-16 (2-17)Quality of group work, 3-17Quality of individual contributions to group, self and peer, 3-18Target skills for lifelong learning, 3-20

Quality of student teaching to peers, 3-21Quality of identified goals for “goals meeting”, 3-22Tabulation of each person’s learning preference, 3-23

4. Selecting a version of PBL for you4.1 Profile of your graduates from your program, 4-14.2 Skills important to develop for your graduates, 4-14.3 Specific skill development, 4-14.4 Ideas to improve future recall of knowledge, 4-44.5 Ideas to improve student learning, 4-5

comparing lecture vs PBL, 4-54.6 Shift from rote to deep learning, 4-74.7 Use PBL, 4-74.8 Student’s attitude, 4-84.9 Help student’s understand PBL, 4-84-10 Faculty ready? 4-8

MRIQ, 4-9 (1-4)4-11 Plan curriculum, 4-84-12 Curriculum full program, 4-124-13 Part of individual course, 4-124-14 Systems for full program, 4-124-15 Resources: enough tutors for full program? 4-134-16 Weekly schedule for full program, 4-144-17 Learning objectives and problems for full program, 4-154-18 Goals, problems for individual course, 4-164-19 Resources: tutors for single course, 4-174-20 What process skills, 4-184-21 When to develop process skills for full program, 4-194-22 When to develop process skills for single course, 4-214-23 Being a tutor in a group, 4-214-24 Working with tutorless groups, 4-234-25 Assessment and accountability, 4-234-26 Looking after the nitty gritty for tutorless or tutored groups, 4-24

5. Implementation:5.1 List outcomes for graduates, 5-15.2 For process skills: agree on definitions, learning objectives and forms of evidence.5.3 For full program, identify “course” where each skill is developed5.4 Reduce subject knowledge5.5 Arrange class times5.6 Create the problems, 5-25.7 Visualize the timing and the meetings, 5-35.8 Organize the student groups, 5-35-9 Create the resources, 5-35-10 Options to assess knowledge acquisition & process skills, 5-35-11 Develop process skills, 5-35-12 Plan for success, 5-35-13 Add structure for assessment, 5-35-14 Add structure to develop self confidence, 5-45-15 Include reflection and elaboration, 5-45-16 Consider how to evaluate your program’s effectiveness, 5-4Examples: of problems and objectives, 5-5 to -10.

6. Example workshops to develop process skills:

6-1 The need, 6-46-2 How to develop, 6-4Examples: MPS-4: Strategies for problem solving

MPS-28: Group skills

7. Plan for success; how to manage the transition to PBL7-1 Teachers ready? 7-47-2 Are the students ready? 7-57-3 Was an appropriate version of PBL used? 7-97-4 Is the version well-designed? 7-107-5 Are the students accountable? 7-107-6 Are lifelong learning skills developed? 7-107-7 Do the problems mirror professional practice? 7-107-8 Are the process skills applied, extended? 7-117-9 Is the assessment done astutely? 7-117-10 Helping students gain the most from their PBL experience, 7-13

8. How to assess student performance8-1 What is assessment? 8-48-2 What to assess? 8-58-3 Set goals and criteria, 8-58-4 Forms of evidence, 8-78-5 Who does the assessment? 8-8Papers: Assessing team work, 8-10

Assessing student performance in PBL, 8-49They just don’t pull their weight, 8-85

9. Evaluation of the program’s effectiveness.9-1 Evaluation to monitor a course while it is in progress, 9-49-2 Evaluation of the overall effectiveness of a course, 9-59-3 Example, 9-6

10. Myths and misconceptions about PBL10.1. Myth? Prior knowledge is needed before students can solve PBL problems 10.2 Myth? I’m doing PBL already10.3. Myth? There is only authentic version for PBL10.4. Myth? Dilution of Content10.5. Myth? PBL can only be used for certain disciplines10.6 Myth? My Students cannot do PBL10.7. Myth? All students will love it.10.8. Myth? It’s easy for teachers to use it10.9 Myth? you’re giving students licence to study whatever they want. What’s happening to standards?

A: Self assessment journalsMarking/feedback on self assessment journals for process skills, A-2Example journals:

Deirdre Schroder’s PBL, A-Fahim Ahmed’s PBLKathryn Grundy’s Chairperson

Index

PrefaceDec 2006

PBL is the best thing since sliced bread!With the success of the method has come a spin off of all kinds of variations on the theme. Indeed, many seem tohave renamed what they used to do PBL because they use Aproblems@. That=s OK. Some use problems to synthesizeprevious knowledge. This is characteristic of engineering design projects and has led to the term Aproject basedlearning@.

This book is not about problem-based synthesis; this book is not about primarily the use of problems. This book isabout small group, self-directed, interdependent, self assessed problem based learning. The key features are that a)students learn material on a need to know basis and b) students are empowered with most of the learning activities:having been given a problem, they identify pertinent learning goals, learn and teach each other the knowledge andskills, use the new knowledge to solve the initial problem and elaborate the new knowledge.

The biggest difficulties faced in implementing this form of PBL seem to be:! the mistaken attitude that teachers just pose a problem and then wonderful things happen.! the mistaken attitude that students can=t learn a subject correctly on their own; “I need to lecture them first andthen pose the problem”.! the mistaken attitude that students will enthusiastically embrace this approach; teachers fail to prepare studentswell for the transition.

Small group, self-directed, interdependent, self assessed problem based learning was formally introduced as alearning option by McMaster Medical School in the late 1960s. But in emulating this model, some fail toacknowledge some of the conditions that made their approach so successful. This include that in the McMastermedical school program:! students were mature; they had already completed at least three years of university - yet we are considering usingit with students in year 1 university; at least three years younger. ! each group of five to six students had a trained faculty member as tutor - yet we may provide very little trainingfor the tutor and, in many instances, are forced to have one faculty for a group of three to ten groups.! students seeking admission wanted to use this new form of learning; they were ready for learning environmentsdifferent from the traditional lecture - yet we are considering using this approach with students who really prefer thelectures and don=t want the unfamiliar.! five criteria were used for admission into the MD program at McMaster: marks, letters of recommendation,personal essay on why the applicant wanted to be an MD, a personal interview and demonstrated skill in the PBLprocess (especially problem solving, interpersonal skills and group process) through participation in the simulatedtutorial.

Most don’t realize that the students entering the Medical school program had these five unique characteristics. Ithink it is important that we realize that part of the success of PBL is because these McMaster medical school students possessed these characteristics. When we started PBL in Chemical Engineering in 1982 we put in placetraining elements to acquire the five elements medical students had. This helped our students get ready for PBL.Furthermore, we had to develop methods, especially related to assessment, that would allow us to work withtutorless groups.

In this book we share our experience.

I thank many for their feedback and help. In particular thanks to Lynda Wee, CapitaLand, Singapore, Andy Hrymak, Heather Sheardown, Gord Slater and Lynn Falkiner, Chemical Engineering Department, McMaster University andmy students in the program at McMaster University

Dec 2006Don WoodsWaterdown, Ontario , Canada

1-1

1. What is PBL? How to select a PBl version that works for you: workshop #1.

Donald R. WoodsMcMaster University, Hamilton, CanadaSept 30, 2006

1. What is PBL?Activity 1:The time I really learn a subject is when I ...... ________________________________________________________________________________________________________________________________________________________________________________________________

When a student asks “Why are we learning this?” do you ever have to say “Trust me, you’ll needthis later.” _____________

Characteristics of PBL:

“Just in time” learning: Learn information/skills/attitudes to “solve a problem”Problem posed first (before the students have learned anything)Students empowered with selecting learning goals, resources, assessment.Work cooperatively in small groups (with or without a tutor present in each group).Teacher? maintain standards, “Guide on the side not sage on the stage,” monitors the process, (like a design project).Students actively engaged in the learning process; students teach each other.

1-2

2. Why select PBL? The claims

M Improved learning (active, coop, freedom to select,ownership, prompt feedback, learning styles)

M Less coverage of subject knowledge

M Prefer deep learning

M Better learning environment

M Easier for recall of knowledge in professionalpractice:

M Learn lifelong learning skills

M Learn problem solving, team skills etc.

M Develops self confidence

Tricky: depends on exams.Same or Better knowledge acquisition/understandCommon exam: lecture = PBL for knowledge. [UDel]Physics: pre-post gains Force Concepts inventory:lecture 0.23 [UDel]active learn 0.48 [Hake]PBL 0.45 - 0.65 [UDel]

Medical NBME I: Slightly less but using multiplechoice of primarily memorized knowledge

True: That’s because we include skills and attitudedevelopment. 80% “time” to cover material because20% on process activities. Focus on core fundamentals in each course; eliminatethe 20% “nice” stuff. Not a problem from ChE recruiters or alumni; indeedsay they are better prepared

Yes. Proven: PBL promotes deep; lectures promote rote and surface approach

Yes, Ramsden/ Entwistle CPQ [McMaster, UDel]PBL 35 to 45lecture: 15 to 22mix: 22 to 35

Should but No published evidence other than the“diver” experiment. Divers memorized nonsense listsof words under water and on land; then tested underwater and on land. Recall best where memorized.

Yes, Perry, resources used, alumni

No, opportunity to develop. Evidence of more empathetic and whole person MDs

Yes, anecdotal

3. Experiencing PBLCase 3: Letter from the Director Your Director has just returned from a conference on PBL. TheDirector asks you to convert your course to the PBL format

Activity: In small group of 5 or 6, with chair _______________________________; reporter ________________________, brainstorm the issues this case raises, and prioritize the issues: criterion:what do you want to gain from this workshop in the context of the issues raised.__________________________________________________________________________________________________________________________________________________________________________ Feedback about the group work. Form 2802

1-3

Form 2802Task: Problem defined, many issues and hypotheses explored, criteria listed and the issuesprioritized. Refrained from early closure. Task carried out and looked back at the result to assessit. Group agreement as to goals. Process was active with monitoring. Completed task on time.The accuracy in the group’s answer matched the time available. Group avoided contributingexcessive information. None of Few of these Most features All of thesethese behaviours but demonstrated behavioursbehaviours major omissions

G G G G G G G1 2 3 4 5 6 7

Morale: Group relaxed; enjoyed working together. They gave emotional support to each otherand were able to express disagreement or disappointment directly. Seven fundamental rightspreserved. Members are enthusiastic and involved. None of Few of these Most features All of thesethese behaviours but demonstrated behavioursbehaviours major omissions

G G G G G G G1 2 3 4 5 6 7

Individual Contribution to Task and Morale

____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Group Strengths Group Areas to work on_____________________________________ ________________________________ ____________________________________ _____________________________________________________________________________________________________________________________________________

from D.R. Woods (1995)

1-4

My role is Questionnaire (MRIQ) ©copyright Donald R. Woods June 1997

The following 18 items are arranged with options (a and b or a,b and c). Each option represents apreference you may or may not hold. Rate your preferences for each item by giving a score from 0 to 5. 0means you strongly disagree and strongly agree with the other option. 5 Means you strongly agree andstrongly disagree with the other option. The scores for a and b, or a, b and c MUST ADD UP to 5 (0 and 5,1 and 4, 2 and 3. Etc..) Place your rating in the box ‘R’ next too the statement.

I think my role as a teacher is….

Statement R Statement R

la. I have a basic conviction that I can make adifference

1b People come to me with basic attitudes andwon't change-

2a. My role is to maintain high standards andfail those who do not make the standards.

2b. My role is to help each succeed and make themost of his/her abilities.

3a. My role is to uncover material so thatstudents understand.

3b. My role is to cover the material in thecurriculum.

4a. My role is to make learning fun. 4b. Learning is serious business. My role is to bewell prepared.

5a. My responsibility is to teach subjects. 5b. My responsibility is to teach people6a. Students must grow personally as well

as intellectually6b. The sole purpose of university is intellectualgrowth

7a. Teaching. research, consulting are allopportunities to help others learn. The onlydifference is the client and the "class size".Teaching and research are a seamlesscontinuum of learning.

7b. Teaching is the burden I must bear to allowme to do research

7c. Research is the burden I must bear to allowme to teach in university.

8a. Teaching and learning are a two-wayresponsibility If students fail it is partly myfault.

8b. Learning is one-way; I do my thing, and it'sup to the students to learn.

9a. If students understand my presentation.they will automatically remember thematerial. Learning is rote memorisation andrecall of facts.

9b. Understanding is not remembering. Studentsand I need opportunities to see new concepts inperspective to understand their limitations and toreach conclusions. Learning is active,independent and self-directed

lOa. Students should learn knowledge andthe processes for working with thatknowledge. Knowledge cannot be separatedfrom thinking.

lOb. All students need to learn in college isknowledge.

11a. The development of values is an integralpart of my instructional plan. Values play asignificant role in my student's futuresuccess.

11b. The development of values is theresponsibility of the home and/or the religiouscomponent of the student's life. You can'tmeasure "value" development; therefore, it isinappropriate to include this area in one's goals.

12 a. Students should self-assess. My role isto ensure that the assessment process usedby the students is valid. I consider the goals,

12b. Assessment of students is my responsibility.I create and mark all the exams that are used tomeasure the quality of student learning.

1-5

criteria and the evidence.

I think my role as a teacher is …….

Statement R Statement R13a. My role is to design the whole learningprocess. Students just have to follow mydesign.

13b. My role is to empower students with allelements in the learning process: goals, choice oftext, assessment...

l4a. I am a resource to help students learn;students have the principal responsibility formaking and carrying out their own plans.

l4b. I am the source of knowledge. I have theadvanced training to be shared with them.

15a. My role is to help students withacademic and intellectual issues. It's not myresponsibility to get involved with theirpersonal and social life.

15b. My role is to help students with academicand intellectual issues and to help them withpersonal problems15c My role is to help students with academicand intellectual issues and to informally socialiseand attend student events

16a. I prepare the detailed learning objectives,the assessment criteria but publish generalguidelines for the students; to do otherwiseprovides too much detail; it's overwhelming forthe students.

16b. I publish detailed learning objectives andassessment criteria.

l6c. Students should prepare detailed learningobjectives and assessment criteria. I monitor theprocess to ensure the standards are met.

17a. My role is to help them solve problems similarto those they will encounter in professionalpractice.

l7b. My role is to ensure that they know thefundamentals. I use problems that help develop andtest that understanding.

l8a. I teach new knowledge. My role is to presentwell organized explanations expressed to match thestudent's learning style.

l8b. All new knowledge bears some relationship to past.My role is to activate the past knowledge and helpstudents see the relationship between the new and theold.

A.1,2,6,9

1a ______ 1b. _____ 5b ______ 5a. _____ 6a ______ 6b _____10a ______ 10b _____ 11a ______ 11b _____15b+c ______ 15a _____

TOTAL ________ ________

B.1,9 2b ______ 2a ______16b+c____ 16a _____

TOTAL _________ ________

C2,3,6 9b ______ 9a _____10a ______ 10b _____17a____ 17b _____

TOTAL _________ ________

D2,3,6 3a ______ 3b ______ 8a ______ 8b ______ 9b ______ 9a ______10a ______ 10b ______14a _______ 14b ______16b+c _____ 16a ______17a _____ 17b _____18b _____ 18a _____

TOTAL ________ ________

E2,6,7,9 12a _____ 12b. _____13b _____ 13a ______14a _____ 14b ______16c _____ 16a+b ____

TOTAL _________ _________

F2,6,8,9 16b+c _____ 16a _____

1-6

Reflections:____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

4. Issues related to selecting a form of PBLYour comfort zone: Complete MRIQA: ____________________________ scale 0 to 30; 23.7 with standard deviation of 3.6B: ____________________________ scale 0 to 10; 7.5 with standard deviation of 2.0.C: ____________________________ scale 0 to 15; 12.0 with standard deviation of 1.9.D:____________________________ scale 0 to 40; 29.4 with standard deviation of 5.6E: ____________________________ scale 0 to 30; 10.3 with standard deviation of 3.5.F: ____________________________ scale 0 to 5; 3.7 with standard deviation of 1.6.

************************************************************************************5. Background for selecting a form of PBLReminder: On improving Learning: What educational research tells us: (Chickering & Gamson + others)

already do this will try possible not forme

1. active not passive 0 0 0 02. cooperative not competitive 0 0 0 03. quality of teacher-learner interaction 0 0 0 04. provide prompt feedback 0 0 0 05. time on task 0 0 0 06. environment expects success 0 0 0 07. account for personal learn style 0 0 0 08. assessment owned by students 0 0 0 09. published goals/criteria (about 30 - 60 per course) 0 0 0 010. attention span = 20 min 0 0 0 011. wait time before answering 20 s 0 0 0 012. knowledge/ skills/ attitude 0 0 0 013. high quality social environment 0 0 0 014. use Bloom’s taxonomy 0 0 0 015. evaluate what you do 0 0 0 0

*****************Background:Straight lecturing is the least effective way to improve student learning. Here is the evidence. 1. Dale's (1967) cone of learning relates our ability to recall to the type of learning environment:

Dale suggests that we tend to remember 10 to 50% from "passive" involvement in the learning process (about 10% from what we read; 20% ofwhat we hear; 30% of what we see; 50% of what we hear and see) 70 and 90% by "active" involvement (we remember about 70% of what we say and 90% of what we sayand do).

2. Students in learning environments where lecturing dominates become more “rote learners”; students learning inPBL or cooperative learning environments become more “deep learners”.

So if I currently use straight lectures, what might I do? Simple ways to change from straightlecturing to more effective learning environments include:1. Use active not passive2. Use cooperation not competition3. Provide prompt feedback about student performance

1-7

4. Empower the students with parts of the learning process.

5.1. Details about how to actively engage students in the learning process1. Individuals write reflections (2 min) then discuss with neighbour (90 sec) 2. “Turn to your neighbour and say.. Did you understand that?” .. Summarize what’s happened so far?” .... Do you believe that?” .. The key point so far is...” ... A practical application of this stuff is....” or small group. [Described in Chapter 3.]3. “Turn to a neighbour and compare or rework notes.4. Talk Aloud Pairs Problem Solve, TAPPS. 5. Students prepare prompt for class discussion.6. Students present.7. Marks for participation.8. Form a line (take a stand): discuss.9. Discussion diads or groups.10. Mass discussion between large collection of students.11. Students pose problems and discuss.12. Socratic discussion.13. Teacher-directed brainstorming/discussion.14. Socratic or teacher-directed Problem Solving with teacher uncovering information after students have done thetask.15. Socratic “learning”.16. Brainstorm: collect ideas without discussion 17. Bidwell’s “quescussion” where brainstorm questions the situation poses 18. Rounds: circle with each 30 s19. Demo group with substitution20. Demo group21. Use Osterman feedback lecture: described in Chapter 322. Projects-report; displays, exhibitions23. Cooperative learning in-class24. Cooperative learning; mentor groups25. Cooperative learning: 1 Stay and 3 Stray26. Cooperative learning: Student peer teach/assess27. Cooperative learning: Jigsaw: 28. Problem-based Learning 29. Structured controversy30. Workshops

5.2. Details about how to use cooperative not competitive environmentsPositive interdependence:1. Teacher create environment to promote.2. Require “norms” meetings.3. Use contracts for expectations.Consider various forms of face-to-face cooperative groups: 4. Set up Informal groups: students stay in groups 2 to 10 min as diads, triads; teacher gives explicit, precise

instructions; require groups to produce a product. This could be Active learning, Table 15-1, items #1 to 4,9, 19- 21: "Turn to a neighbour..." through to Osterman Feedback Lecture.

5. Set up Formal cooperative groups: students stay in groups 2 to 4 weeks; teacher sets the objectives,decides who is in group, explains task & goals, monitors effectiveness, assesses. This happens often in labgroups.

6. Set up Formal self-directed learning groups: students stay in groups 10 to 12 weeks; students setslearning objectives that are monitored by teacher, students research and teach each other the subjectknowledge. Use Problem-based Learning

7. Set up Projects groups: students stay together 1 year.8. Set up Base cooperative groups: students stay in groups 1 semester, year, four years. They provide

informal support, encourage & assist each other in completing assignments and in succeeding.

1-8

Individual accountability:9. Report/assess: (for options 4-7) the teacher randomly select the individual who will complete the task on

behalf of the group: that individual mark for that work is given to all group members. For example, assign anumber to each group member and roll a dice [draw a number from a hat] to identify the group'srepresentative.

10. Report/assess: (for options 5-7) as in #9, but the students select the individual, subject to the constraint thatfor the five tasks being assessed, each group member is selected once and only once.

11. Report/assess: (for options 4-7) teacher publishes a list of those who will represent the group. Thatindividual's mark is the mark of the group.

12. Report/assess: (for options 4-7) all individuals do a task; all will receive the arithmetic average of theindividual's marks for the group.

13. Report/assess: (for options 4-7) group provides a single response. All members receive the same mark fortask. Individuals report accountability through peer and self-assessment of "contribution".

14. Report/assess: (for options 4-7) group provides a single response. All members receive two marks: the samemark for the response plus a peer/team leader response that notes "individual's contribution".

15. Report/assess: (for options 4-7) individuals do task and get individual mark. Students receive writtenfeedback from peers about their cooperative contribution. Each submits a reflective journal reportdiscussing that feedback and receive individual mark for collaborative effort and skill.

16. Report/assess: (for option 8) twice a year the group reports progress via a poster about their approach; markfor the poster is added for 5% to marks of group members in some identified course.

Positive attitude17. Use Perry’s model and reflection

5.3. Details of how to provide prompt feedback:1. You mark fewer: fewer total;2. You mark fewer: spaced: 1/3 then 1/3; then 1/3.3. You mark fewer: sampling: mark 3 out of 5.4. You mark less: single cover page summary.5. You mark structured: format student presentation for ease in marking.6. You mark structured: format for marking/feedback.

Use peer or self 7. Peer/self: mark assignments in-class with teacher-facilitated. 8. Peer/self: in-class diad. 9. Peer/self: in-class triad.10. Peer/self: in-class feedback. 11. Use group reports with ratings.

Use cooperative learning groups: 12. You mark fewer: cooperative: group's mark is result of individual effort via random selection or other means.13. You mark fewer: cooperative group work; mark group. 14. You mark fewer: cooperative group, one person is responsible for the assignment; completes it and peers ingroup mark and give feedback so that it can be improved before it formally gets submitted to you.15. Use PBL.

5.4. Details of how to empower students with parts of the learning process

1-9

1. Teacher can asssign all the problems in the text for homework and then the exam is a selected set fromtheproblems. Students organize selves so that all of the problems are worked and handed in to be marked. Open bookexam.Students can:2. Set learning goals.3. Set criteria. 4. Select pertinent resources: people, books, articles.5. Select type of evidence to show achievement toward goals.6. Decide on the weighting for different parts of the evidence toward the “final” grade. Example, best of a 60/40 splitbetween term work and final exam or contract for a percentage between 20 and 80 %.7. Self-assess based on teacher’s goals, criteria and forms of evidence.8. Create goals, criteria, forms of evidence and assess “participation”. 9. Submit “first two questions for the final”; share copies and answers with all class members. Teacher selects two.Open book exam.10. Create test questions; questions given to another group; students mark, teacher monitors. 11. Create the exam. Teacher marks.12. Peer assess.13. Do the whole task: goals, criteria, evidence, assess. Teacher validates the process.14. Use PBL.

6. Select an option that suits you: see also Decisions

1. No PBL: improve learning: a. know their names; call them by name.b. use “ombudspeople”: class representatives who tell you how the teaching-learning is going.c. limit teacher talk to 20 min; then have some activity. ex “Turn to a neighbour and...” d. communicate to them that you expect them to succeed..e. explain why you are doing this.

2. No PBL but willing to empower them with part of the assessment:Do #1 and.. let them submit questions (and answers) from which you select 2 or 3 to put directly on the

exam.

3. Comfortable lecturing: Do #1 and.. pose a problem first, then lecture, then return and use the knowledge to solvethe problem.

4. Comfortable with group work: Do #2, pose the problem and ask groups to list “things they need to know to solvethis problem”, list these, then lecture on their learning needs, use the knowledge to solve the problem.

5. PBL for 3 weeks of your course.

6. PBL for a full course

7. PBL for all courses in the program

*************************************************************************7. Possible Issues when you implement PBL: ( suggested answers in article called “Helping Your Students Gainthe Most from PBL: Linda’s complaint”)Teachers ready?Students ready?Created an environment where you will succeed Appropriate version of PBL used: 5 -6 groups of students, 75 minute sessions instead of three 60 min/ weekPBL well -designed? you know the reason for selecting: active, coop, prompt feedback, expect success, studentownership.Students accountable?Lifelong learning developed?

1-10

Problems well crafted?Process skills applied, extended?Assessment done astutely?

8. How to create problemsExample

! Case 6 Paul’s decision

Context: Lifelong learning course to first yearstudents at City University of Hong Kong

Target Objectives:Setting prioritiesManaging timeBudgeting timeClearer understanding of the expectations inuniversity for time to studyLong and short term planningProcrastinationLearning to say No!

Paul’s decision (Ed Ko, City University of Hong Kong)Paul has been persuaded by his friends to run for a position onthe Departmental Society. He really would like to, but he isafraid that doing so might take time away from his otheractivities. He is already on the university swimming team andhas to work five hours a week in order to earn some money topay back his credit card loans. With six courses that he is takingthis semester, he feels that he is constantly behind in his work.

8. How to create problems: criteria for effective cases or scenarios:About the goals:1. the chosen learning goals achievable. For single courses (for example in hybrid or conventional programs) about3 to 5 hours of study for an individual student; about 6 to 10 objectives for a group of 6 students so that each willresearch/teach the others.2. the learning outcomes are consistent with the stage of development and builds on and activates prior knowledge.3. goals integrate knowledge, skills and attitudes across subjects and disciplines.*********************About the scenario created. (can be a single scenario, or you could build a sequence of scenarios but each wouldexpect the same 3 to 5 hours of student study)

4. the scenario contains “cues” that will trigger the desired search for learning objectives; the learning outcomesexpected by the teacher are identified correctly by the students.5. the scenario includes an appropriate level of complexity. 6. the scenario allows an openness.7. the scenario is motivational and relevant. 8. the scenario is similar to one we might encounter in professional practice; (for example, in Engineering this mightinclude rating, debottlenecking, design, trouble shooting, labour relationships, team work, public, monitoringcompliance with legislative regulations).9. promotes student activity.10. any data given should be raw data (like we encounter in practice).11. the scenario identifies the context, gives a concrete scenario and clearly identifies the expected task withoutspelling out specifics.

___________________________________________________________________________________________9. Some options for assessment of knowledge learned.M Student summary of the quality of knowledge learned. M Group solution to the problem.

1-11

M Individual concept maps of the knowledge. M Individual Test and Exams of the knowledge, TETK. created by teacher, peers, groups, individuals, self.M Individual teach notes and learning contract. M Peer assessment of the quality of the knowledge brought to the teaching task.

Assessment of process skills: see series of papers in Chemical Engineering Education “Assessing problem solvingskills”, “Assessing team skills”, “Assessing lifelong learning skills.”

10. Developing the process skills see http://www.chemeng.mcmaster.ca/innov1.htm and click on PBL for overviewand click on MPS for details. see also paper PBL: Decisions for Planning and Action: why? when? who? where?what? how? see also the workshops given in Woods “PBL” Resources to gain the most from PBL” and “LargeClass” workshop.

11. Perry’s check on student attitude; grieving process for students and for you

12. Plan for Success!!! students ready? teachers ready? see Woods, “PBL: how to gain the most from PBL”Chapt 1 and PBL: Decisions with Perry inventory (topic 12) and My Role Is Questionnaire (Topic 5).

13. Practical downsides and how to solve them

Both students and teachers must value the “process skills” and “lifelong learning skills”. PBL must be seen as beingmore than learning subject knowledge! Structure is needed on your part and careful preparation of the students andtheir attitudes is important; not all students will embrace this approach nor gain the same amount from theexperience. Students self report that the goals meeting is the most difficult.

Feedback for PBL/SDL a120

1-12

Feedback to ___________________________ for Unit ____ Date ___________ Present & on time: G Present but late by ___ min. Absent G

Quality of Knowledge: good intellectual understanding of the topic, the material supplied was complete andappropriate.

None of A few but Most of these. All ofthese. major omissions. these. O__________O_________O_____________O_____________O_____________O_____________O1 2 3 4 5 6 7

Quality of Instruction: he/she was here on time, the presentation was focused on the new knowledge; goodchoice of material and medium with effective communication and resource material supplied.

None of A few but Most of these. All ofthese. major omissions. these. O__________O_________O_____________O_____________O_____________O_____________O

Followup: from this presentation I will have to:

Must study subject Major self-study Some No self-on my own; needed. I have some self- study of I learned starting references study of the basics. nothing from from your presentation. the basics. I want to your presentation. reflect

about the ideas.

O____________________O_____________O_____________O_____________O_____________O Strengths Areas to Improve on_____________________________ _____________________________________________________ ____________________________________________________________________________________________________________________________________________ from D.R. Woods, "How to Gain the Most from PBL," (1994)

Feedback for interdependent, self-directed learning Form 3601

1-13

Feedback from the Goals Meeting

Situation$1IssuesNumber identified: 1 2 3 4 5 6 7 >7

Agreement with tutor <50% 50% 60% 70% 80% 90% 100%

Knowledge/skills to be learned Consensus among group little some a lot complete

Agreement with tutor's list little some a lot complete

Learning objectivesQuality poor fair OK good excellent

LearningQuality of questions asked during the teach session none some astute excellent

Willingness to continue to contribute <50% 50% 60% 70% 80% 90% 100%

Your AttitudePerry shift 2 3 3.5 4 4.5 5******************************************************************************

1-14

MPS 36 Learning preferences and attitudes a120

Name Attitude: Perryscale

Learn style Jungian

before now strategic

rote meaning

S value and implicationsfor learning

T value: combine with Sand implications on testquestions

value implication value implication

You

a120

2-1

2. Example PBL: Linda’s complaint Plenary presentation

2 Asia-Pacific Conference on PBLnd

4 to 7 Dec, 2000, Singapore

Helping your students gain the most from PBLDonald R. Woods

Chemical Engineering Department

McMaster University

Hamilton, ON L8S 4L7

Canada

Abstract:

PBL is more than an extremely effective environment to learn subject knowledge. It can be used to help students

develop skill in lifetime learning, change management, teamwork, conflict resolution and problem solving. However,

for this to be effective, there is much that a teacher can do to help the students acquire these abilities.

These skills need to be identified as valued outcomes of the program. Learning objectives and criteria should be

created. Journal writing provides an excellent way to help students gather evidence of accomplishment. Options for

assessment will be discussed.

****************************************************************************

Linda’s complaint

You, a teacher in a PBL program, are talking to Linda, a student graduating from your PBL program. She

confides in you, “I believe I know the expected subject knowledge but I really don’t think that my problem

solving skill or group skills have improved. Perhaps the curriculum planners can look more closely at the

skill development being claimed in the program.”

In this paper, I try to model elements of a version of PBL where

- a problem is posed, Linda’s complaint;

- we identify issues pertinent to the problem by identifying what we know already from similar scenarios and

experiences and what new information we need to know;

- we consider information and ideas about selected issues raised by the case,

- we use that information to “pose a solution” to Linda’s complaint and then

- we elaborate that knowledge.

Consider each in turn.

1 What are the issues in this scenario?

In conventional lecture programs, the teacher will have created a set of learning objectives that he/she wishes the

participants will achieve. The teacher may have used this scenario to introduce a “lecture” on the development of

skill in lifetime learning, change management, teamwork, conflict resolution and problem solving.

In the context of PBL, we discuss how well the scenario or case problem is written, activating prior knowledge, ways

that students can be empowered to create and prioritize the issues and some example issues.

M Is the scenario well-crafted?

In most PBL programs, the goal is to empower the students with the task of creating the learning objectives that are

important to them. The scenario should be posed such that (Dolmans et al., 1997, Alverno College, 1977) and

Drummond-Young and Mohide, 2000)

2-2

1. the learning outcomes expected by the teacher are identified correctly by the students; the scenario contains

“cues” that will trigger the desired search for learning objectives;

2. the learning outcomes are consistent with the stage of development and builds on and activates prior

knowledge;

3. an appropriate level of complexity is included;

4. the scenario requires integration of knowledge, skills and attitudes across topics;

5. the scenario allows an openness;

6. the scenario is motivational and relevant; the scenario is similar to one we might encounter in professional

practice;

7. the scenario promotes student activity;

8. the scenario identifies the context, gives a concrete scenario and clearly identifies the expected task.

Hopefully for this conference this scenario addresses issues that are important in your application of PBL in your

context. I hope it builds on and activates past knowledge. I selected this case because it resembles scenarios that I

encounter frequently when I consult with different groups about PBL. This case addresses the assumption that if we

use a small group, self-directed, self-assessed, problem-based learning model then this should develop skill in such

“process skills” as change management, teamwork, conflict resolution and problem solving. Researchers (Norman,

1988, Norman and Schmidt, 1992, 1993, Woods, 1989, 1993a,b) report that this is a faulty assumption.

Now let’s use the case to activate prior knowledge similar to that presented in the case, Linda’s complaint.

M Activating prior knowledge/experience. Have you seen something similar before?

To facilitate learning, Schmidt (1983) emphasizes the importance of activating previous knowledge gained from

“similar situations.” Have you encountered anything like this before? Perhaps you worked as a volunteer for the Red

Cross in the community and the expected goals for the Red Cross program had not been achieved. An issue in

Linda’s scenario is that a person in a program, Linda, feels that her expectations had not been met. Is this educational

situation similar to one you have encountered previously? Maybe you felt as Linda felt. What did you do? What

turned out to be the major issue?

Now that we have helped relate the case to the known, what new information is needed to resolve the problem posed

in the case Linda’s complaint?

M How might we empower student groups to generate the learning issues?

The options we might use to generate the learning issues include:

- the teacher gives the student groups a list of “teacher-generated” learning objectives.

- large groups (>6 students) of students generate and prioritize a list via a tutor-facilitated discussion.

- small, tutorless groups of students generate and prioritize a list as a small group (<7 students) and their results

are validated by the tutor or www program.

- small groups students with a tutor generate and prioritize a list with the tutor gently guiding the activity.

M What are some issues that this scenario might generate?

Some issues that might be generated by teachers with/without experience in PBL are given below. We find it useful

to consider the problem from different perspectives and to focus on different cue words used in the scenario.

From the perspective of the teacher: “cues” from the case include such words as: you, teacher in PBL, confides,

graduating, claim in the program.

S Does the student have the skill but doesn’t realize it? Is her skill in self assessment weak? perhaps Linda didn’t

receive feedback?

S Is Linda’s expectation justified? Have we said we would develop this skill? or is she equating PBL = problem

solving? small group = skill in teamwork?

S Is this a widespread concern? (Linda confides only in you; did she tell others?)

2-3

S Is this a major frustration for Linda? or is she just giving suggestions to improve the program?

S Why does she express this now instead of earlier in the program? have we poor methods of gathering student

comments throughout the program? Is the issue one of monitoring the program continually?

S What actions do I, as a tutor, take to try to develop my student’s skill and confidence in these skills? Should I

take any?

S Do I have skills in coaching students in the development of these skills?

S Have I received training in coaching?

S What do I mean by “problem solving”? “teamwork”? what other skills and attitudes besides these are addressed

in our program?

S How do I assess these skills? How do I provide feedback to the students about their skill development?

From the perspective of the student: “cues”: confides, improved, I know.... but I don’t think,

S What skills did Linda possess when she entered the program? should she expect to see an improvement?

S Has there been an improvement in skill but she lacks confidence?

S Has she personally benchmarked her behaviour, set personal goals, criteria and selected forms of evidence to

improve? Should she be expected to do this?

S What is Linda’s role and obligation? and what is the tutor’s role and obligation? and what is the program’s role

and obligation ?

S Is this an issue of skill? or confidence in the skill? or both?

S What is Linda’s definition of “problem solving”? “teamwork”? What other skills should she expect to be

developed in the version of PBL she experienced?

S Linda identifies that she “knows the subject knowledge.” This suggests some skill in self assessment. Can the

same forms of feedback, that told her she knew the subject knowledge, be effective in telling her that she is

skilled in problem solving and team work?

S If Linda feels these are important, why didn’t she let us know earlier? An issue might be one of trust and

openness in the learning environment or one of monitoring.

From the perspective of PBL learning process: “cues”: PBL program.

S Which form of PBL was used? Is the form likely to explicitly develop the target skill?

S Class size?

S Assessment?

S Role of tutor? subject specialist? skill development specialist?

S Do all - students and staff - agree that skill development is important?

From the perspective of the program: “cues”: in the program, curriculum planners

S What skills do students have entering the program? How do we know?

S What are the expected outcomes from our program? are these published? do they include confidence and skill?

S Do the tutors, teachers, students, and administration all understand the expected outcomes?

S Have we defined places, times and activities in the PBL program (or before the PBL program) for the

development of the confidence and skill?

S Have we trained the tutors to be effective coaches?

S Have we trained the students?

S How do we assess the student skill?

S How do we evaluate the program effectiveness?

S How do we monitor the program?

S How do we obtain student input continually?

S Do we, as teachers, know the skill is developed but the students don’t?

S How do you develop self confidence?

From the perspective of the skills: “cues”: skill

S What is the definition of the skill?

S Have published learning objectives and measurable criteria been developed for this skill? How are the outcomes

2-4

generated? How can we create measurable criteria?

S Is the skill identified as an outcome of the program?

S How is the skill assessed?

S Have we really helped the student see the skill development or have we been ineffective in communicating?

From the perspective of learning/acquiring skill: “cues”: skill, skill development

S Is developing skill established by methods similar to learning knowledge?

S How do you teach skill?

S How do you assess skill?

S How do students self assess skill?

Synthesis and prioritization of issues:

Many issues were repeated when seen from the different perspectives. However, some main issues are:

A. issues related to developing process skills: how, when and how to assess.

B. issues related to embedding the process skill development into and throughout the program.

C. issues related to assessment especially as it applies to process skill and “intuitive” behaviour.

D. issues related to developing student awareness, confidence and trust.

E. issues related to selecting options for your approach to PBL and your culture.

F. issues related to gathering student feedback, monitoring and program evaluation.

Lots of issues from this case! You have probably identified others.

2 Reporting back on selected issues

Here we summarize ideas for each of the six issues. Consider each in turn.

A. Developing process skills

Using small group, self-directed, self assessed, interdependent PBL assumes that the participants use (and are skilled

in that use) skill in problem solving, group work, self-directedness, teaching, self assessment and communication

(Woods, 1994).

How to develop: For problem solving, what doesn’t work seems to be giving the students many problems to solve,

using challenging problems, watching the instructor/tutor solve the problem, watching others solve problems

(Woods, 1993b). We might generalize these conclusions to the development of any skill or attitude.

What seems to be effective is to use research to identify target behaviours (see how the experts do the task); convert

these into learning objectives with measurable criteria, identify forms of evidence that students can collect that show

their performance, give students opportunities to try the skill, provide feedback, more practice, feedback and

continue until students “believe” they have mastered the skill (Bandura, 1982, Schon, 1987, Woods et al., 1997,

Woods et al., 2000a, Alverno, 1977).

Example evidence-based target skills, learning objectives, criteria, forms of evidence, and assessment are listed

for problem solving, change management, group work, self assessment, conflict resolution (Alverno, 1977,

Woods, 1999a,b; Woods et al. 2000a, Woods et al. 2000 b, and Woods et al. 2000c,d). Example levels of

development for team skills are given in Table 1; target skills, in Table 2.

**************

Insert Tables 1 and 2

**************

We have used a workshop style learning environment to develop process skills (Woods et al., 1997, Woods, 1999d).

A workshop is designed to:

- help individuals become aware of how they do the skill,

- provide target skills for the effective application of the skill,

- give an opportunity to reflect and self assess,

2-5

- gives students a chance to gather evidence about individual use of the skill,

- provide benchmarks and encourage students to create goals for growth of the skill development.

Example transparencies, timing outlines and descriptions of some of the workshops are available (Woods,

1999d).

An alternative to the use of workshops is given by Alverno College. Alverno (1977) integrates the development

throughout all of its courses with published lists of objectives and levels of development for each course. They have

a separate assessment department to assess some skills.

When to develop: Four options that have been used include:

- prescreen and only admit students into your program who have the skills already and continue with monitoring,

assessment and feedback (McMaster MD approach);

- prePBL workshops. Provide workshops before students start the PBL activities and continue with monitoring,

assessment and feedback (our approach in engineering);

- integrate the workshop skill development with the early PBL cases and continue with monitoring, assessment and

feedback (McMaster Theme school approach);

- integrate the development throughout the whole program with a published progression through four levels of skill

development as the students move through the program (Alverno College model).

Assessment of student skill: If the objectives, criteria and forms of evidence are clearly developed and published,

then assessment is easy. More on assessment is given in section C.

B. Embedding the development of process skills into PBL

To embed the skill development into the program requires action at the program or departmental level, general

considerations and at the individual PBL unit level.

M Program actions:

1. Make the process skill a valued and published outcome for your program.

Example:

“Graduates of this program will think rigorously and critically and solve problems effectively and efficiently.”

2. Build student assessment and program evaluation into the program right from the start. Don’t tack this on as an

afterthought. How are you going to test skill in teamwork? How will an individual be graded? How do we evaluate

whether our efforts are successful? How do we benchmark and set goals for growth to improve our program?

For example, all students could complete the Heppner PSI (Heppner, 1986) and the Billings-Moos inventories

(Billings and Moos, 1981) at the beginning of the program to establish benchmark data. The first PBL case

could address issues of teamwork, the second addresses assessment and self assessment and so on.

3. The goals, criteria and methods of assessment should be consistent across the whole program. Publish details of

where and how the skill will be developed. Alverno College provides an excellent example (Alverno, 1977).

For example, “Problem solving is defined in the course pack. The focus in the first semester will be on

developing...”

M General Principles

4. Treat the development of process skills with the same rigor and scholarship that you use in the development of

subject knowledge.

5. Create the framework for assessment and evaluation. Details are given in Section C.

6. Make the implicit behaviour explicit. So much of the processing occurs automatically in our heads and in the

2-6

heads of other skilled practitioners. When asked “How do you do that?” she replies “I don’t know; it just

happens.” Our task is to take the skill and behaviour apart; discover what really is important based on research,

make goals and criteria and then present the experiences in bits and bites that can be mastered by our students.

This provides the context for skill development.

7. Encourage monitoring. Provide a checklist of questions the students can use to monitor their process skill

development. This provides the student with one form of evidence.

8. Ask students to reflect on the process. For each team meeting held, ask them to write out their reflections of how

they did the task. This provides the student with one form of evidence.

M For your PBL activity.

9. In your syllabus, restate the program outcomes, list the outcomes that will be addressed explicitly in your course.

For example,

In these next PBL units, you will learn new knowledge and synthesize previous knowledge to solve problems

related to the cardiovascular system [the subject-specific skill development or new subject knowledge gained]

and you will develop skill in working in teams, (or critical thinking or problem solving or communication).

10. Know what previous training in the target process skills the students have had and build on those.

11. Use terminology, assessment forms and standards of assessment consistent with the overall program.

12. Gather benchmarking data to aid in the program evaluation.

13. Usually assign students to the groups unless there is a compelling reason to allow them to select their own

groups. Keep the same groups for at least eight weeks.

14. Assign a chairperson for every meeting. Research has shown that groups function better with a designated

chairperson. Require the chairperson to prepare and circulate an agenda ahead of time. Ask the group to give written

feedback to the chairperson at the end of each meeting. The chairperson uses this input to reflect on his/her skill and

to set targets for development. This provides the student with one form of evidence.

15. Work with students at their stage of need. For example, don’t expect team performance and use team criteria for

assessment when the group is really a fairly effective group. Apply standards of assessment consistent with the level

of development. Secondly, we could help them see the big picture, and nurture, encourage and reward them on their

journey toward effective teamwork.

C. Principles of Assessment

Assessment we define as a judgement based on the degree to which the goals have been achieved based on

measurable criteria and on pertinent evidence. We have found that breaking this definition into five principles assists

in applying this definition. The five principles are: (Alverno, 1985, 1994; Woods, 1994 and Boud 1993)

1. Assessment is a judgement based on performance - not personalities. We need to help a student realize that a poor

mark does not mean he/she is a bad person. The judgement is made about performance in completing a task. It has

nothing to do with his/her value as an individual. This is an issue, especially for students with attitudes characterized

by Perry’s level 2. More details about Perry’s levels and their implications to teaching and learning are given

elsewhere (Perry, 1970; Woods, 1994; Woods et al., 2000a).

2. Assessment is a judgement based on evidence - not feelings. We might intuitively feel that a student is a good

problem solver. However, we need to replace that intuitive feeling with physical evidence such as the written script

2-7

on an exam or in a project report. Help the students (and staff) gather evidence. Provide time for students to write

reflections. The staff responsibility is to create well-designed standardized feedback/assessment forms. Provide

practice understanding the terms and working with the forms before the program.

3. Assessment should be done for a purpose with clearly-defined performance conditions. The student should know

when he/she is being assessed.

4. Assessment is a judgement done in the context of published goals, measurable criteria and pertinent, agreed-upon

forms of evidence. Use research, not personal intuition, to identify the target skills. Publish goals with measurable

criteria for process skill. Such goals should provide clear needs for documented evidence. These should be published

at the start of the program so that there are no surprises for the students and no unwanted student backlash to the

teacher. Examples for some process skills were described in Section A with ideas for team skills given in Tables 1

and 2. Example forms of evidence for team work are given later in this section.

5. Assessment should be based on multidimensional evidence: static and dynamic situations; small assignments and

lengthy projects; academic, social and personal contexts; under a variety of performance conditions (exams and

homework, written and oral, performance as an individual and as a member of a group,) formative and summative

data and with different persons being the assessors (self, peer, teacher and trained external observers).

To remove ambiguity from the assessment the following six issues in practice should be addressed (Alverno, 1985,

Woods, 1994).

1. Goals: What is being assessed? Knowledge in engineering or a discipline subject? Skills? Attitudes? Have the

goals been expressed unambiguously in observable terms? Who creates the goals? Are the goals explicit and

published?

2. Criteria: Are there criteria that relate to the goals? Can each criterion be measured? Who creates the criteria? Are

the criteria explicit and published?

3. Form of evidence: What evidence is consistent with the criteria? Are the checklists used for the assessment asking

questions related to the criteria? Do both the assessor and the student know that this form of evidence is acceptable?

4. Resources: Are the goals and the collection of the evidence possible to achieve in the time and with the resources

available?

5. Assessment process: What is the purpose of the assessment? Under what conditions is the student’s performance

assessed? Who assesses? What type of feedback is given by the assessor? (For example, Pass fail? a grade? five

strengths and two areas to work on?) What is the form of feedback? Verbal? Written? What is the timing of

feedback? Who delivers the feedback?

6. Training in the assessment process: Have both the student and the assessor received training in assessment?

Failures of assessments to accomplish their purpose can usually be traced to violations of any of these five principles

or to the incorrect application of the six issues in practice. The major challenges students have seem to be:

M Students can understand goals and objectives for subject knowledge. “It’s Chapter 3 in the text.” or “It’s like the

assignments we’ve had, only changed a little (at least we hope it is changed only a little).” But how do you assess

teamwork? An answer is that we need to create easy-to-understand and demonstratable goals.

M Students underestimate the importance of evidence. Furthermore, they have trouble seeing the types of evidence

that might be useful.

Example forms of evidence for problem solving are available (Woods, Wood, Sheardown, Kourti and Crowe,

2000b). Here are some examples for team work (Woods, 2000a):

Open 1 Attendance records. No a very useful form of evidence. A person can be present and sleep through the

activity (Woods, 1996).

Option 2. Test and Exams in Process Skills, TEPS: Create examinations that test the achievement of the published

objectives for team skills. Some example questions are given in Resources Chapter D (Woods, 1999a).

2-8

Option 3: Peer and self assessment of individual contributions to the team using a well-designed, standardized

observation/feedback form (Woods, 1994 and Woods et al., 2000d, Woods, 2000a).

Option 4. Reflective journals for individual growth. Individuals gather evidence about his/her own contributions to

the team: reflections, peer feedback, agendas of meetings, documents brought to the group for a task, monitoring

records.

Option 5; Reflective journals for team growth: Individuals gather evidence from the reflections about group

behaviour after each group meeting, add reflections about what they personally did to help the team grow. Examples

are available (Woods, 1993-95; 1999c).

Option 6: Individual assessment of team activity using a standardized form (Woods et al., 2000c).

Option 7: Observer assessment of the team performance.

Option 8: Observer assessment of the individual performance.

Option 9: Portfolio that synthesizes all the above evidence.

Option 10: Published validated questionnaires for component skills: listening, responding trust, conflict resolution.

See Woods, 1999e for a listing of some possible inventories.

Option 11: Feedback or self reflection about the assessment process. Maturing and moving toward teamwork brings

with it skill in self awareness, self acceptance and self assessment. The form in Table 3 can be used to provide

evidence about skill in self assessment and in assessment (Woods, Marshall and Hrymak, 1988). This reminds us of

the Principles of Assessment outlined above.

************

Insert Table 3

***********

D. Issues related to helping students see success, developing confidence and trust.

Here are some suggestions:

M Provide frequent and prompt feedback; work in the context of evidence-based targets; require reflection and

journal writing. Follow the principles of assessment. Encourage personal goal setting.

M Use validated inventories to help individuals identify his/her uniqueness and personal style (such as Kirton’s KAI,

Schutz’s FIRO-B, Jungian Typology, men and women in conversation, the Perry inventory (Woods, 1994) the

Lancaster Approaches to Studying (Ramsden, 1983) and Johnson’s approaches to resolving conflict (Johnson,

1986)). Help students move from self awareness to self acceptance, to acceptance of others with different styles and

to self confidence.

M Alert students to the four keys to effective interpersonal relationships:

Key #1. Honour the seven fundamental rights of individuals, RIGHTS (Woods, 1994)

R to be Respected

I Inform or to have an opinion and express it

G have Goals and needs.

H have feelings and express them

T Trouble and make mistakes and be forgiven

S Select your response to others expectations

2-9

Claim these rights and honour these in others.

Key #2. Avoid the four Killers of relationships (Woods, 1994):

Contempt

Criticism

Withdrawal and stonewalling

Defensiveness

Key #3. Build trust. Trust glues relationships together.

We build trust by

keeping commitments to yourself and others.

clarifying expectations that you have of yourself and of others.

showing personal integrity, honesty and loyalty to others, especially when they are not present.

promptly and sincerely apologizing when you know you are wrong.

honouring the fundamental RIGHTS listed above and avoiding the killers.

taking time to see things from the perspectives of others.

accepting others "warts and all."

We destroy trust by

the reverse of the Builders of trust listed above, and

not meeting commitments.

selectively listening, reading and using material out of context

not accepting experience of others as being valid.

making changes that affect others without consultation.

playing the broken record until you're eventually worn them out or subtly make changes in the

context/issues/wording gradually so that they are unaware of what is happening until it is too late. They

were sideswiped.

asking others to give up their fundamental RIGHTS.

key #4. Give feedback to others to encourage and help them; not for you to get your kicks and put them down

(Woods, 1994).

This approach applies to help build relationships between teacher/tutor and students and among students.

E. Issues related to selecting options for your approach to PBL and your culture.

PBL is an extremely effective environment for learning subject knowledge. But it also is a misunderstood, popular,

flavour-of-the-month option. Some introduce PBL just because “it is the thing to do this year.” In deciding what

version of PBL might be appropriate for you and for your institution let’s remind ourselves of basis behind PBL.

Here’s my version of the basics:

PBL is a learning environment where a problem is posed first before the students have learned the knowledge. PBL

is like research: we have a conundrum to resolve; what do we have to learn/do to resolve it? There are many versions

of PBL. For example, a lecturer could pose a problem first, and then lecture. The challenge is to us this opportunity

effectively to develop problem solving skill.

PBL helps students create knowledge structures that will prompt rapid recall in future practice. The memory patterns

are different from those generated when the subject knowledge is learned structured around typical textbooks. The

challenge is for us to facilitate the development of cues and the appropriate knowledge structure (Schmidt, 1983,

1993).

Self-directed PBL empowers the students with the tasks of learning (many of which are owned by the teacher in

conventional programs). Students identify learning issues, create learning objectives and criteria, contract with

members of the group to learn and teach parts of the unknown stuff, teach others, elaborate and self-assess how well

the learning has progressed. Students are empowered; the challenge is that we need to make them accountable.

2-10

Self-assessed PBL empowers the students with parts of , if not all of, the assessment. Gibbs (undated) and Novak

(1989) say “whoever owns the assessment, owns the learning,” Many other options can be used to empower students

with assessment (see Felder et al., 2000). PBL is one exciting option. The challenge is to develop self-assessment

skills in our students and to shift the teacher’s role so that the teacher monitors the assessment process, rather than

doing the assessment.

Small group PBL uses an active, cooperative learning environment. Research in learning (Chickering and Gamson,

1987) suggests that learning improves if students are active, work cooperatively, have clear time on task, receive

prompt feedback, and are in an environment that expects success and that caters to their personal learning style.

Small group PBL provides a wonderful opportunity to achieve these. The challenge is for use to create a small

group environment that does indeed include these characteristics. Other learning environments can be designed for

active learning; for cooperative learning; to provide prompt feedback (see Woods, 2000b, Felder et al., 2000), but

small group PBL offers an ideal environment that applies most of the principles to improve learning.

Small group, self-directed, self-assessed PBL tends to create a learning environment that nurtures “deep” learning

rather than “surface/rote” learning. The latter tends to be promoted by traditional lecture environments (Ramsden,

1983; Woods, Hrymak and Wright, 2000e).

Small group, self-directed, self-assessed PBL provides probably the best option for the development of lifelong

learning skills. The challenge is to explicitly develop lifelong learning skills following the same guidelines given

Section A.

Small group, self-directed, self-assessed, interdependent PBL may not be the format that fits your culture or your

personal style. What we can do is:

M Use the knowledge of how best to facilitate student learning as the basis for choosing of learning environment.

M If small group, self-directed, self-assessed, interdependent PBL is selected, then address the challenges so that

your students gain the most.

F. Issues related to gathering student feedback, monitoring and program evaluation.

Give PBL the best chance to succeed and be scholarly in your approach to program evaluation,

M Whatever approach to PBL you take, give your interventions the best chance to succeed. Whenever you try

something new, most students resist.

- Students prefer the familiar to something new. They know the “lecture system” and how to make it work

for them to get the highest marks. If we change the familiar “system”, most resist the change (Benvenuto,

1999). When they experience change, students often follow the eight-step grieving process of shock, denial,

strong emotion, resistance, acceptance, struggle, better understanding and integration (Woods et al., 2000a;

Woods, 1994). To help them through the process, run a workshop.

- Explain why you are making the change.

- Help students see their personal benefits of the new approach in the short term and in the long term.

- Explain your role. Students may believe that they are paying for teachers to “lecture.” (Benvenuto, 1999).

Use Perry’s inventory (Perry, 1970, described in Woods, 1994; Woods et al., 2000a) to help students

understand their perceptions; explain your role in terms of the Perry model.

- Monitor the program frequently. Use ombudspersons or one minute papers (described in Felder et al.,

2000) and adjust in response to their feedback.

- Be flexible. If this particular class is vehemently opposed to the new approach, explore options to achieve

your goals and theirs. Perhaps scale down the intervention. Gather data on how to make the intervention

more effective the next time you try it.

- Bring in success stories. Invite recruiters or students, from other programs where the intervention has been

effective, to give testimonials at the start of your course.

- Help students cope with the upheaval when old habits are identified and changed. This is particularly true

2-11

for problem solving. For example, part way through a program to develop problem solving, students

complained that “focussing on the process of problem solving has meant that they could no longer solve

problems.” We used the following analogy to help them through this frustration.

Consider that you are a reasonably good tennis player. You go to an expert to improve your game.

The expert takes your game apart. As you relearn how to do each part, your game is not what it used

to be. Be patient. Gradually you improve and surpass your past skills.

M Be scholarly in your approach to program evaluation. Unfortunately, when it comes to teaching many faculty

“diddle around”. They try one thing in the classroom; then they try another. Their approach evolves like Topsy

with them never sure as to what works and what doesn't. They would never do research in our subject discipline

that way. In "research" they create a hypothesis, create models or experiments and test the hypothesis and draw

conclusions. With just a little more effort, we can bring that same scholarship to teaching. The hypothesis is that

“by making a change your students will learn more effectively and/or they will develop process skills of value to

them as professionals.” Your intervention should make a change. But does it? Consider using some simple and

easy-to-administer pretests and post-tests. Examples for process skills have been cited (Woods, 1999e; Woods et

al., 2000b, Woods et al., 1997). Gather data from exit surveys. Queen’s University has developed an excellent

survey (Queen’s, 1994). Reflect on what went on. Gather evidence. Write journal articles and seek grants.

Being scholarly in your approach means that right from the start you plan how to test and monitor the

effectiveness of what you do.

3. Use the new knowledge to solve a problem

In Linda’s context we find that:

1. Published outcomes of the program are:

“Graduates of our program will be skilled problem solvers and team players.”

2. Skill development is identified but confidence in applying the skill is inferred.

3. Few formal mechanisms are in place for gathering student feedback about the program.

4. The assessment is done by the tutor for each of the seven students in the group. The assessment is done by the

tutor who ticks off boxes related to skill development.

5. Feedback to the student is through the student reading the file that includes all of her tutor’s forms. The overall

grade is a Pass/fail. The student receives no guidance as to how to interpret the forms.

6. No formal training is given to the students on group skills or problem solving.

7. The tutor assumes the role of chairperson at all PBL meetings; the meeting don’t start until the tutor arrives; no

agendas are circulated.

8. The feedback forms used by the tutor to assess the team skills and the problem solving skills do not include the

criteria; the only time the student sees the forms is in the files.

9. The student is not asked to present evidence about her skill in problem solving or team work.

10. Some self assessment is expected but, as Linda says, “It’s a joke. We receive no training. I was surprised when

you showed me the principles of assessment, in Section C. No target skills or detailed objectives and criteria are

published. We just tick “Good” in all categories. If we tick “Excellent” then we know we might be challenged by the

tutor. If we tick “Needs work” or “Poor” then I’m shooting myself in the foot. My rating is wishful thinking. I really

wish I had some concrete evidence about my skill so I could remove the hazy feeling I have about it all. Besides, I

don’t think anyone pays any attention to my rating. The only rating that counts is the tutor’s.”

2-12

Comments: This program violates most of the principles of assessment. Although the program has the development

of problem solving and team skills as an outcome, there does not seem to be any formal structure to help meet that

claim. Consider, in turn, the principles of assessment:

Principles 1 and 3. the assessment should be based on performance. The performance conditions were not clearly

defined so that Linda did not know the conditions when she was being assessed for these skills. She assumes that the

conditions are her day-to-day performance in the tutorial group. The major assessor seems to be the tutor, who plays

many roles including facilitator and chair of the group. Although some self assessment is asked for, Linda is not clear

about the role her self assessment plays in the overall assessment. The form she uses seems poorly designed because

it lacks target skills and measurable criteria.

Principle 2. The student is not assessed based on nor is given time to produce written evidence.

Principle 4. Published goals and criteria (whether they be created by the students or created by the tutors for the

program) are missing. Linda does not know any acceptable forms of evidence.

Principle 5. The many different forms of evidence that could be used are not clearly identified.

Everything seems to be based on the tutor’s use of a checklist. de Stephen (1985) and Swanson et al. (1991) suggest

that one assessor cannot do a reasonable assessment (via observation of a tutorial group) of more than three people at

one time. In Linda’s school one tutor is expected to assess seven people! in addition to facilitating the tutorial

process.

A solution:

Revamp the assessment process based on the principles from Section C.

Provide training in the process skills and in self assessment for Linda and her classmates.

Adding such structure will improve student accountability.

Provide better communication between the tutors, program administrators and the students about the program and its

expectations.

Provide monitoring activities, such as classroom assessment techniques, CATS (Angelo and Cross, 1993),

ombudspersons or one-minute papers to provide the tutors and administrators with a better understanding of how

well the teaching and learning in progressing in the context of PBL.

4 Elaborate the knowledge

Schmidt (1983, 1993) and Coles (1990, 1991) emphasize the importance of asking students to elaborate on the new

knowledge. Expand on the relationship between concepts, explicitly consider how things come together, see tasks as

linking together aspects of knowledge both within and between subjects. Link theory with practice, make

connections. Relate the new to past knowledge. Some example tasks that help students elaborate include: private

study, essay writing, problem solving in groups, preparing and presenting a paper or case, studying for exams,

creating concepts maps, posing and answering questions, taking notes, discussing with others, teaching peers what

they first learned themselves, summarizing, reflecting, formulating and criticizing hypotheses, writing what they

DISCOVERED, extending the application to other scenarios, creating other scenarios, and extending from education

to everyday life.

For Linda’s complaint, we can extend the use of the same principles to improve staff performance reviews, to

improve our interaction with our children, and to guide our goal-setting and self-improvement activities.

5. Summarizing

Instead of me presenting a summary of what I hoped to highlight in this paper, I ask you to use Table 4 and elaborate

and reflect on what you discovered and rate the ideas as they apply to your situation.

2-13

**************

Insert Table 4

*************

6. References

Alverno College (1977) “Faculty Handbook on Learning and Assessment,” Alverno College, 3401 South 39 St.th

Milwaukee WI. 53215-3922.

Alverno College (1985) “Assessment at Alverno College,” Alverno College Publications, 3401 39 St., Box 343922,th

Milwaukee, WI. 53234-3922.

Alverno College , (1994) “Student Assessment-as-Learning at Alverno College,” Alverno College Publications, 3401

39 St., Box 343922, Milwaukee, WI. 53234-3922.th

Angelo, T.A., and K.P. Cross (1993) “Classroom assessment techniques: a handbook for College Teachers,” 2 ed.,nd

Jossey-Bass, San Francisco, CA.

Bandura, A. (1982) “Self-efficacy mechanism in human agency,” American Psychologist, 37, 122-147.

Benvenuto, M.A. (1999)“In an Age of Interactive Learning, Some Students Want the Same Old Song and Dance,”

The Chronicle of Higher Education, June 4, B9.

Billings, A.G., and R.H. Moos (1981) “The Role of Coping Responses and Social Resources in Attenuating the

Stress of Life Events,” J. Behav. Medicine, 4, no. 2, 139-157.

Boud, D. (1993), “Enhancing Learning through Self Assessment,” Kogan Page, London, UK.

Chickering, A.W., and Z.F. Gamson (1987) “Seven Principles of Good Practice in Undergraduate Education,”

AAHE Bulletin, March 3-7.

Coles, C.R. (1990) “Elaborated learning in undergraduate medical education,” Medical Educ., 24, 14-22.

Coles, C.R. ( 1991) “Is problem-based learning the only way?” Chapter 30 in “The Challenge of problem based

learning,” D.Boud and G. Feletti, eds., Kogan Page, London.

de Stephen, R. (1985) personal communication, Workshop on team skills, Teaching Thinking Conference, Hilton

Head, SC.

Dolmans, D.H., et al. (1997) “Seven Principles of Effective Case Design for PBL,” Medical Teacher, 19, 185-189.

Drummond-Young, Michelle and E. A. Mohide (2000) “Developing problems for use in PBL: from concepts to

application,” Chapter 8 in “Transforming nursing education through problem-based learning,” E. Rideout, ed., Jones

and Bartlett, Sudbury, MA.

Felder, R.M et al. (2000) “The Future of Engineering Education: Part 2: teaching methods that work,” Chem. Eng.

Ed., 34 (1), 26 - 39 and downloadable from the www

http://www2.ncsu.edu/effective_teaching/

Gibbs, G. (undated) “A-Z of Student Focused Teaching Strategies,” Educational Methods Unit, Oxford Polytechnic,

Headington, UK.

Heppner, P.P. (1986) “The PSI Manual,” 210 McAlester Hall, University of Missouri-Columbia, Columbia, MO

2-14

65211.

Johnson, D.W., (1986) “Reaching Out: interpersonal effectiveness and self-actualization,” 3 edition, Prentice Hall,rd

Englewood Cliffs, NJ.

Norman, G.R. (1988) “Problem solving skills, solving problems and Problem-based Learning,” Medical Educ, 22,

279-286

Norman, G.R., and H.G. Schmidt (1992) “The Psychological basis of Problem-based Learning: a review of the

evidence,” Academic Medicine, 67, no. 9, 557 to 565

Norman, G.R. and H.G. Schmidt (1993) “Where is the Learning in PBL?” PEDAGOGUE, 4, 2, summer, a

Newsletter published by the Program for Educational Development, Faculty Health Sciences, McMaster University,

Hamilton, ON.

Novak, J.D. (1989) “Helping students learn how to learn: a view from a teacher-researcher,” Third Congress of

Research and teaching in Science and Mathematics, Santiago de Compostela, Spain, Sept.

Perry, W.G., Jr. (1970) “Forms of Intellectual and Ethical Behaviour in the College Years,” Holt Rinehart and

Winston, New York, NY.

Queen’s University (1994) Office of the Registrar, “Undergraduate Learning Experiences at Queen’s: results from

the exit poll,” Queen’s University, Kingston, ON (each year si nce 1994)

Ramsden, P. (1983) “The Lancaster Approaches to Studying and Course Perceptions Questionnaire,” Educational

Methods Unit, Oxford Polytechnic, Oxford, UK.

Schon, Donald (1987) “Educating the Reflective Practitioner: toward a new design for teaching and learning in the

professions,” Jossey-Bass, San Francisco, CA.

Schmidt, H.G. (1983) “Problem-based Learning: rationale and description,” Medical Educ., 17, 1-16.

Schmidt, H.G. (1993) “Foundations of problem-based learning: some explanatory notes,” Medical Educ., 27, 422-

432.

Swanson, D.B., S.M. Case and C.P.M. van der Vleuten (1991) ‘Strategies for Student Assessment”, Chapter 27 in

“The Challenge of PBL,” D. Boud and G. Feletti., ed.s, Kogan Page, London, UK.

Woods, D.R., R.R. Marshall and A.N. Hrymak (1988) “Self-assessment in the Context of the McMaster Problem

Solving Program,” Evaluation and Assessment in Higher Education, 12, 2, 107-127.

Woods, D.R. (1989) “Problem Solving in Practice,” Chapter 7 in “What Research Says to the Science Teacher,” D.

Gabel, ed., National Science Teachers Association, Washington DC.

Woods, D.R. (1993a) Guest Editorial On the Learning in PBL,” PEDAGOGUE, 4, 2, summer, a Newsletter

published by the Program for Educational Development, Faculty Health Sciences, McMaster University, Hamilton

Woods, D.R. (1993b) “Problem solving- what doesn’t seem to work,” J. College Sci Teaching, 23, 157-158.

Woods, D.R. (1993-1995) MPS 3: “Self-assessment”, PS News, 84, p 84-12 to 25 (1993)

MPS 5, Stress Management, PS News, 85, 85-14 to 85-26 (1993)

MPS 6, “Analysis: classification,” PS News, 92, pp 92-14 to 23 (1994)

MPS 11, “Personal Uniqueness,” PS News, 93, pp 93-17 to 25 (1994)

2-15

MPS 17, “Time management”, PS News, 97, p 97-11to 97-25 (1995).

Woods, D.R. (1994) “Problem-based Learning: how to gain the most from PBL” Woods publishing, Waterdown,

ON and distributed by McMaster University Bookstore. Translated into Japanese by S. Shindo and published by

Igaku-Shoin, Ltd., Tokyo (2000)

Woods, D.R. (1996) “Participation is more than attendance,” J. Engng. Educ. 85, 177-182.

Woods, D.R., et al. (1997) “Developing Problem Solving Skill: the McMaster Problem Solving Program,” J

Engineering Education, April, 75-91 (1997) http://www.chemeng.mcmaster.ca/innov1.htm and click on MPS for

a summary of the research findings and some details for many of the units.

Woods, D.R. (1999a) “How to set up courses and course objectives,” Chapter D in “Problem based learning:

resources to gain the most from PBL,” Woods, Waterdown (1999) and downloadable from

http://www.chemeng.mcmaster.ca/innov1.htm and click on PBL and download from the resources book.

Woods, D.R. (1999b) Target skills for team skills http://www.chemeng.mcmaster.ca/innov1.htm and click on MPS

and then on target skills for each MPS unit.

Woods, D.R. (1999c) “How to assess,” Chapter F in “Problem-based Learning: Resources to gain the most from

PBL,” Woods, Waterdown, 2 edition. Distributed by McMaster University Bookstore, Hamilton ON nd

Woods, D.R. (1999d) “Workshops” Chapters B and C “Problem based learning: resources to gain the most from

PBL,” Woods, Waterdown.

Woods, D.R. (1999e) “How to select instruments for assessment and program evaluation,” Chapter E in “Problem

based learning: resources to gain the most from PBL” downloadable from

http://www.chemeng.mcmaster.ca/innov1.htm and click on PBL and download from the resources book.

Woods, D.R. (2000a) “Team Building: how to develop and evaluate individual effectiveness in teams,” Proceedings,

AIChE Annual Meeting, Los Angeles, Nov 2000, http://www.aiche.org

Woods, D.R. (2000b) “Ideas to Improve Learning,” manuscript, McMaster University, Hamilton ON, 270 pp.

Woods, D.R., R.M. Felder, A. Rugarcia, and J.E. Stice (2000a) “The Future of Engineering Education: Part 3.

developing critical skills,” Chem. Eng. Ed., 34 (2), 108-117 and downloadable from the www

http://www2.ncsu.edu/effective_teaching/

Woods, D.R., P.E. Wood, H. Sheardown, T. Kourti and C. M. Crowe (2000b) “Assessing of problem solving skills,”

Chem. Engng. Educ., paper accepted

Woods, D.R., S Taylor and S. Jaffer (2000c) “Assessing of team skills. Part I: successful teams” Chem. Engng.

Educ.

Woods, D.R., S Taylor and S. Jaffer (2000d) “Assessing of team skills. Part II: individual contribution to teams,”

Chem. Engng. Educ.

Woods, D.R., A.N. Hrymak and H.M. Wright (2000e) “Approaches to Learning and Learning Environments in PBL

versus lecture-based learning,” Proceedings, ASEE Conference, June St. Louis, MO., session 2213.

2-16

Table 1 Example list of levels of skill development for “team skills”

M Alverno’s model (Alverno, 1977)

Level 1. Identify own interaction behaviors utilized in a group problem solving situation. (Eleven behaviors for the

task activities used as criteria; observation by peers, strengths and weakness noted; agreement between observer and

client).

Level 2. Analyze behavior of others within two theoretical frameworks: task and morale components..

Level 3. Evaluate behavior of self within two theoretical frameworks for at least three different, videotaped

situations. Self, peer and teacher assessment. Growth contract created.

Level 4. Demonstrate effective social interaction behavior in a variety of situations and circumstances.

Level 5. Demonstrate effective interpersonal and intergroup behaviors in cross-cultural interactions.

Level 6. Facilitate effective interpersonal and intergroup relationships in one’s professional situation.

Forms of evidence and criteria are available.

M MPS model (Woods, 1999b, Woods et al., 1997; Woods, 2000a)

Level 1. Self awareness, self acceptance and acceptance of personal style and preferences of others: MPS 11

Level 2. Awareness of attributes of successful groups/teams for the two theoretical frameworks of task and morale:

part I MPS 28; Groups can self assess, set goals for growth.

Level 3. Self awareness of own contributions to the group/team: MPS 28; 29. Individuals can state his/her

contribution to the group process. Gather evidence about effectiveness in role as chairperson; set goals for growth.

Understand the implications of FIRO-B for personal contribution to group evolution.

Level 4. Being an effective member of a “good group” MPS trust, conflict resolution, problem solving, give receive

feedback, assertiveness

Level 5. Being an effective member of a team. MPS 51

2-17

Table 2 Evidence-based targets for group skills (reprinted from Woods, 2000a)

Evidence-based targets Progress toward internalizing these targets

20% 40% 60% 80% 100%

M Performance improves when we have goals.

M Assessment must be related to the goals

M Both Task (getting the job done) and Morale (feeling good about the

group work and about how you have interacted with the other group

members) are important

M Any group functions better with a chairperson.

M Chairperson and leadership are different; different people may

become leaders at different times.

M Group evolution tends to follow a pattern described as by such

descriptors as “ forming, storming, norming and performing” . Schutz’s

instrument FIRO-B seems to provide reliable insight as to the personal

style of individuals towards other group members during three of these

phases.

M We can list the roles needed in both Task and Morale to make an

effective group.

M When each person has a clear idea of roles and group norms, the

group functions better.

M When groups are functioning effectively, about 75% of the time is

spent on the task; 15% on morale building activities and 15% of task

process activities (how the problem solving process is going;

summarizing ideas, guiding the process).

M The products from groups or teams is improved when members have

different “styles” (in Jungian terminology some members are dominant

S, and some, dominant N). The products from groups tend to be inferior

when all the members “think and behave alike”.

M The quality of decisions, product, task is improved if group members

offer different perspectives, disagree and seem to introduce conflict into

the process. The trick is to manage the apparent conflict well.

M The characteristics of “ meetings of individuals,” “effective groups”

and “teams” fall on a spectrum with sufficient differences that it is

useful to differentiate based on those characteristics

M In a decision-making mode, after 20 minutes of discussion on any

one topic, few new ideas are presented and repetition of previously

stated ideas occurs.

2-18

Goals: Content is well identified, goals are challenging and achievable, goals are written in observable terms,

goals are unambiguous, the "given" conditions are specified.

None of Few of these Most features All of thesethese behaviours but demonstrated behavioursbehaviours major omissions

G G G G G G G1 2 3 4 5 6 7

Criteria: Criteria are consistent with the goals and are measurable and practical. The criteria are challenging

and achievable.

None of Few of these Most features All of thesethese behaviours but demonstrated behavioursbehaviours major omissions

G G G G G G G1 2 3 4 5 6 7

Evidence: The type and quality of evidence gathered is consistent with the goals and criteria. The evidence

has been gathered conscientiously over a long enough period of time. The evidence is well organized. The

quality and extent of evidence is sufficient to allow me to judge the extent to which the goals have been

achieved.

None of Few of these Most features All of thesethese behaviours but demonstrated behavioursbehaviours major omissions

G G G G G G G1 2 3 4 5 6 7

Process: The assessment process has been applied and as an independent assessor I concur with the decision

as to the degree to which the goals have been achieved.

None of Few of these Most features All of thesethese behaviours but demonstrated behavioursbehaviours major omissions

G G G G G G G1 2 3 4 5 6 7

Strengths Areas to work on

________________________________________ ________________________________________

________________________________________ ________________________________________

________________________________________

________________________________________

________________________________________ from D.R. Woods, "How to Gain the Most from Problem-based Learning" (1994)

Table 3 Feedback about assessment

2-19

Table 4. Reflection about and self rating of the ideas in this paper

Reflection and DISCOVERY

____________________________________________________________________________________________

____________________________________________________________________________________________

____________________________________________________________________________________________

____________________________________________________________________________________________

____________________________________________________________________________________________

Rate the ideas already would might not mydo this work work style

Well crafted problems Meets learning objectives O O O O

Contains “cues” O O O O

In context of the program and activates previous knowledge O O O O

Appropriate complexity O O O O

Integrates knowledge, skill attitude across topics O O O O

Open O O O O

Similar to one from future professional practice O O O O

Promotes student activity O O O O

Identifies context and activity, concrete O O O O

Other ____________________________ O O O O

In your use of PBL

Take time for students to activate previous knowledge O O O O

Facilitate and validate the generation of learning objectives O O O O

Ask for elaboration O O O O

Other _____________________ O O O O

Developing process skills

Create target skills based on research. O O O O

Create goals for learning and measurable criteria O O O O

Provide opportunities for students to gather evidence O O O O

Use a workshop to develop skill O O O O

Use Alverno’s model of continual development O O O O

Decide on When to develop: pre-screen entrants? O O O O

develop before PBL O O O O

integrate development with early units O O O O

integrated throughout whole program O O O O

Other ___________________ O O O O

Embedding within the program

Publish outcomes O O O O

Build in assessment and evaluation from the start O O O O

Identify where the skill is developed O O O O

Consistent assessment throughout O O O O

Other _________________ O O O O

2-20

Rate the ideas already would might not mydo this work work style

Principles of Assessment

Based on performance O O O O

Based on evidence O O O O

Has a purpose and performance conditions are defined O O O O

In the context of published goals, criteria, forms of evidence O O O O

Multidimensional evidence O O O O

Share experiences O O O O

Six issues O O O O

Forms of evidence:

Use of TEPS for process skills O O O O

Use reflective journal writings O O O O

Use self-assessment O O O O

Use peer assessment O O O O

U se individual assessment O O O O

Use portfolio writing O O O O

Use trained observers O O O O

Use assessment of the assessment process O O O O

Other ______________________________ O O O O

Developing student confidence and trust

Provide prompt and frequent feedback O O O O

Work in the context of targets O O O O

Use reflections and journal writing O O O O

Use principles of assessment O O O O

Help students identify personal uniqueness O O O O

Honour seven fundamental RIGHTS O O O O

Avoid four killers O O O O

Build trust O O O O

Give feedback that is helpful O O O O

Other ____________________________ O O O O

Select PBL style effective for you

Pose problem before students learn O O O O

Use opportunity to build problem solving skill O O O O

Help student build knowledge structures O O O O

Empower students with elements of the learning process O O O O

Build in student accountability O O O O

Empower students with parts of the assessment O O O O

Train and use self assessment O O O O

See myself as monitor of the assessment process O O O O

Build in the key elements to improve learning O O O O

Try to develop “deep” learning instead of “surface” O O O O

Use PBL effectively to develop lifelong learning skills O O O O

Other _______________ O O O O

2-21

Rate the ideas

already would might not my

do this work work style

Give PBL the best chance for successRationalize to students O O O O

Use the grieving model O O O O

See the “opportunity” in the change O O O O

Explain your role O O O O

Monitor via “one minute paper” or ombudspersons O O O O

Be flexible: change if needed O O O O

Bring in success stories: Use testimonials O O O O

Other __________________________________________ O O O O

Evaluate program: be scholarly

Write out your hypothesis O O O O

Plan and gather data O O O O

Use exit surveys O O O O

Test and monitor effectiveness O O O O

Other _________________ O O O O

Use new knowledge to solve the problem

Give students with the time to solve the problem O O O O

Other ______________ O O O O

Elaborate the knowledge

Rationalize to students why elaboration is important O O O O

Use concept maps O O O O

Provide activities to reflect O O O O

Extend the application to other venues O O O O

Use DISCOVERY O O O O

Use checklists, like this one O O O O

Other ______________________ O O O O

3-1

3. Large class & PBL: copyright 2002 © Donald R. Woods

Donald R. WoodsMcMaster University, Hamilton, Canada c/pbl/lc-wksp-act.

Case 2: I’ve got some good projects! “I want to try PBL. I have some great projects from a local company. Since

my contacts in that company say that “team work” is important, I’ll use groups. I have a class of about 60 students.

Since I think group work is important, I’ll break the class into 10 to 12 groups. To train in students in PBL I’ll hand

out to the students the revised PBL framework . I have modified the generic PBL framework slightly because I

don’t think it appropriate in a large class for them to evaluate the tutor. Here is what I’ll give to the students to guide

them in PBL

Stage 1: Group setting. Introduce members. Establish ground rules. Define roles of tutor and students.

Stage 2: Problem identification: Present the problem. Identify and clarify the problem. Describe the problem.

Stage 3: Idea generation: Inquire the problem, Brainstorm possible cause and effects. Generate ideas.

Stage 4: Learning issues formulation: Determine what students need to find out in order to solve the problem.

Generate learning issues and action plan. Summarize and rank learning issues.

Stage 5: Self-directed learning:. Seek information.

Stage 6: Synthesis and application: Evaluate sources of information for credibility and reliability. Apply relevant

researched knowledge to the problem. Conducting peer sharing of information. Analyze knowledge. Critique

knowledge. Develop more learning issues if necessary. Discuss and develop solution and explanation.

Stage 7: Reflection and feedback: Self and peer feedback on group functioning, individual problem solving process,

knowledge learnt, self-directed learning.

That should work! and it should be easy for me now that I have the good problems.”

“Ok,” says Phan, “But how will you know that they are learning anything in their groups? Or, how will you know if

only one person is doing all the work? and What are you going to be doing while they are working in groups?”

“The students will have to sort out the first two questions. That’s part of their training to work in groups. As for me,

I’ll sit at the front of the room, and probably check my e-mails since there isn’t that much expected of me. My role is

to give them the problem. And what neat problems I’m giving them.”

*************************

Activity 1: In small group of 5 or 6, with chair _______________________________;

reporter ________________________, brainstorm the issues this case raises. Identify what you know already.

Identify what you need to learn.

Prioritize the issues: criterion: what do you want to gain from this workshop in the context of the issues raised.

_____________________________________________________________________________________________

_______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ Feedback about the group work. Form 2802

3-2

Task: Problem defined, many issues and hypotheses explored, criteria listed and the issues prioritized. Refrained

from early closure. Task carried out and looked back at the result to assess it. Group agreement as to goals. Process

was active with monitoring. Completed task on time. The accuracy in the group’s answer matched the time

available. Group avoided contributing excessive information.

None of Few of these M ost features All of these

these behaviours but demonstrated behaviours

behaviours major omissions

G G G G G G G

1 2 3 4 5 6 7

Morale: Group relaxed; enjoyed working together. They gave emotional support to each other and were able to

express disagreement or disappointment directly. Seven fundamental rights preserved. Members are enthusiastic and

involved.

None of Few of these M ost features All of these

these behaviours but demonstrated behaviours

behaviours major omissions

G G G G G G G

1 2 3 4 5 6 7

Individual Contribution to Task and Morale

_____________________________________________________________________________________________

_____________________________________________________________________________________________

_____________________________________________________________________________________________

_____________________________________________________________________________________________

_____________________________________________________________________________________________

___

Group Strengths Group Areas to work on

_____________________________________ ________________________________

____________________________________ ________________________________

_____________________________________

____________________________________

____________________________________

from D.R. Woods (1995)

3-3

Feedback from the Goals Meeting

Case 2:

Issues

Number identified: 1 2 3 4 5 6 7 >7

Agreement with tutor <50% 50% 60% 70% 80% 90% 100%

Knowledge/skills to be learned

Consensus among group little some a lot complete

Agreement with tutor's list little some a lot complete

Learning objectives

Quality poor fair OK good excellent

Learning

Quality of questions asked during the teach session none some astute excellent

Willingness to continue to contribute <50% 50% 60% 70% 80% 90% 100%

Your Attitude

Perry shift 2 3 3.5 4 4.5 5

******************************************************************************

Reflections:

_____________________________________________________________________________________________

_____________________________________________________________________________________________

_____________________________________________________________________________________________

_____________________________________________________________________________________________

_____________________________________________________________________________________________

_____-

3-4

1. What is PBL?

Problem posed first (before the students have learned anything)

Students empowered with selecting learning goals, resources, teaching each other and assessment.

Work cooperatively in small groups (with or without a tutor present in each group).

Teacher? maintain standards, “Guide on the side not sage on the stage,” monitors the process, (like a design project).

2. Options: general

PBL is an extremely effective environment for learning subject knowledge. But it also is a misunderstood, popular,

flavour-of-the-month option. Some introduce PBL just because “it is the thing to do this year.” In deciding what

version of PBL might be appropriate for you and for your institution let’s remind ourselves of basis behind PBL.

Here’s my version of the basics:

PBL is a learning environment where a problem is posed first before the students have learned the knowledge. PBL

is like research: we have a conundrum to resolve; what do we have to learn/do to resolve it? There are many versions

of PBL. For example, a lecturer could pose a problem first, and then lecture. The challenge is to us this opportunity

effectively to develop problem solving skill.

PBL helps students create knowledge structures that will prompt rapid recall in future practice. The memory patterns

are different from those generated when the subject knowledge is learned structured around typical textbooks. The

challenge is for us to facilitate the development of cues and the appropriate knowledge structure (Schmidt, 1983,

1993).

Self-directed PBL (SDL PBL) empowers the students with the tasks of learning (many of which are owned by the

teacher in conventional programs). Students identify learning issues, create learning objectives and criteria, contract

with members of the group to learn and teach parts of the unknown stuff, teach others, elaborate and self-assess how

well the learning has progressed. Students are empowered; the challenge is that we need to make them accountable.

Self-assessed PBL (SA PBL) empowers the students with parts of , if not all of, the assessment. Gibbs (undated)

and Novak (1989) say “whoever owns the assessment, owns the learning,” Many other options can be used to

empower students with assessment (see Felder et al., 2000). PBL is one exciting option. The challenge is to develop

self-assessment skills in our students and to shift the teacher’s role so that the teacher monitors the assessment

process, rather than doing the assessment.

Small group PBL (SG PBL) uses an active, cooperative learning environment. Research in learning (Chickering

and Gamson, 1987) suggests that learning improves if students are active, work cooperatively, have clear time on

task, receive prompt feedback, and are in an environment that expects success and that caters to their personal

learning style. Small group PBL provides a wonderful opportunity to achieve these. The challenge is for use to

create a small group environment that does indeed include these characteristics. Other learning environments can be

designed for active learning; for cooperative learning; to provide prompt feedback (see Woods, 2000b, Felder et al.,

2000), but small group PBL offers an ideal environment that applies most of the principles to improve learning.

SG, SDL, SA PBL tends to create a learning environment that nurtures “deep” learning rather than “surface/rote”

learning. The latter tends to be promoted by traditional lecture environments (Ramsden, 1983; Woods, Hrymak and

Wright, 2000e).

SG, SDL, SA PBL provides probably the best option for the development of lifelong learning skills. The challenge

is to explicitly develop lifelong learning skills by making learning objectives, helping students gather evidence and

requiring assessment journals.

Small group, self-directed, self-assessed, interdependent PBL may not be the format that fits your culture or your

personal style. What we can do is:

3-5

M Use the knowledge of how best to facilitate student learning as the basis for choosing of learning environment.

M If small group, self-directed, self-assessed, interdependent PBL is selected, then address the challenges so that

your students gain the most.

2. Options especially appropriate for large classes 5-500.

For details and examples of these see Chapter A in Woods, “Resources to gain the most from PBL” downloadable

from

http://www.chemeng.mcmaster.ca/innov1.htm and click on PBL

already might not

use work for me

3.Osterman feedback lecture: Pose problem first, lecture in 20 min bursts O O O

4. Branda’s PBEE: Individual SDL PBL O O O

5. Socratic: SG, SDL, PBL O O O

6. Guided Design: SG, instruction directed PBL O O O

7. Tutorless groups: SG,SDL,SA PBL O O O

7. Challenges in working with Tutorless groups

Challenges to making this work:

structure, structure, structure... without interfering with the process.

Specific challenges related to students working in tutorless groups:

- apparent conflict because all members are not seen as pulling their fair share of the work

- attendance

- trust and reliability

- different work goals and standards among different students that leads to conflict.

- the amount of reflection, monitoring and writing they must do in self assessment and accountability.

Our approach has been to focus on making the students accountable (topic 8) and developing their skill and

confidence in the process skills (topic 9).

8. Making students Accountable

Accountable for the learning, the teaching, the problem solving, the goals and goal setting, the critical reasoning, the

location of literature and information, validation of the information, the team work, the conflict resolution, the

communication, change management, lifelong learning skill development, the assessment, the self assessment, the

ownership of the process and skill as chairperson. Accountable to peers, to self and to tutor, but primarily to self and

peers.

M Base the activities and skill development on research - not intuition. See target skills: download from the specific

MPS unit

http://www.chemeng.mcmaster.ca/innov1.htm and click on MPS (McMaster Problem Solving program)

M Use a framework of assessment for the accountability. Five principles and Six issues in practice.

Five Principles:

Principle 1. Assessment is a judgement based on performance - not personalities. We need to help a student realize

that a poor mark does not mean he/she is a bad person. The judgement is made about performance in completing a

task. It has nothing to do with his/her value as an individual. This is an issue, especially for students with attitudes

3-6

characterized by Perry’s level 2.

Principle 2. Assessment is a judgement based on evidence - not feelings. We might intuitively feel that a student is a

good problem solver. However, we need to replace that intuitive feeling with physical evidence such as the written

script on an exam or in a project report. Help the students (and staff) gather evidence. Provide time for students to

write reflections. The staff responsibility is to create well-designed standardized feedback/assessment forms. Provide

practice understanding the terms and working with the forms before the program.

Principle 3. Assessment should be done for a purpose with clearly-defined performance conditions. The student

should know when he/she is being assessed.

Principle 4. Assessment is a judgement done in the context of published goals, measurable criteria and pertinent,

agreed-upon forms of evidence. Use research, not personal intuition, to identify the target skills. Publish goals with

measurable criteria for process skill. Such goals should provide clear needs for documented evidence. These should

be published at the start of the program so that there are no surprises for the students and no unwanted student

backlash to the teacher.

Principle 5. Assessment should be based on multidimensional evidence: static and dynamic situations; small

assignments and lengthy projects; academic, social and personal contexts; under a variety of performance conditions

(exams and homework, written and oral, performance as an individual and as a member of a group,) formative and

summative data and with different persons being the assessors (self, peer, teacher and trained external observers).

To remove ambiguity from the assessment the following six issues in practice should be addressed (Alverno, 1985,

Woods, 1994).

1. Goals: What is being assessed? Knowledge in engineering or a discipline subject? Skills? Attitudes? Have the

goals been expressed unambiguously in observable terms? Who creates the goals? Are the goals explicit and

published?

2. Criteria: Are there criteria that relate to the goals? Can each criterion be measured? Who creates the criteria? Are

the criteria explicit and published?

3. Form of evidence: What evidence is consistent with the criteria? Are the checklists used for the assessment asking

questions related to the criteria? Do both the assessor and the student know that this form of evidence is acceptable?

4. Resources: Are the goals and the collection of the evidence possible to achieve in the time and with the resources

available?

5. Assessment process: What is the purpose of the assessment? Under what conditions is the student’s performance

assessed? Who assesses? What type of feedback is given by the assessor? (For example, Pass fail? a grade? five

strengths and two areas to work on?) What is the form of feedback? Verbal? Written? What is the timing of

feedback? Who delivers the feedback?

6. Training in the assessment process: Have both the student and the assessor received training in assessment?

M Create learning goals for all activities, skills and attitudes being developed. Download examples from Chapter D

in Woods, “Resources to gain the most from PBL” downloadable from

http://www.chemeng.mcmaster.ca/innov1.htm and click on PBL

M Develop forms of evidence. The forms will make performance (or lack thereof) visible. Form are evidence to be

used for assessment. Use assessment journals are the key evidence for assessment.

Example forms of evidence are given as Appendices.

M Especially with tutorless groups, use forms of evidence that make individual performance visible (and

accountable).

9. Developing the process skills

What to develop: skills needed for the learning, the teaching, the problem solving, the goals and goal setting, the

3-7

critical reasoning, the location of literature and information, validation of the information, the team work, the

conflict resolution, the communication, change management, lifelong learning skill development, the assessment, the

self assessment, the ownership of the process and skill as chairperson. The basics include:

problem solving, team work, self assessment, teaching, and lifelong learning.

How to develop: For problem solving, what doesn’t work seems to be giving the students many problems to solve,

using challenging problems, watching the instructor/tutor solve the problem, watching others solve problems

(Woods, 1993b). We might generalize these conclusions to the development of any skill or attitude.

What seems to be effective is to use research to identify target behaviours (see how the experts do the task); convert

these into learning objectives with measurable criteria, identify forms of evidence that students can collect that show

their performance, give students opportunities to try the skill, provide feedback, more practice, feedback and

continue until students “believe” they have mastered the skill (Bandura, 1982, Schon, 1987, Woods et al., 1997,

Woods et al., 2000a, Alverno, 1977).

Example evidence-based target skills, learning objectives, criteria, forms of evidence, and assessment are

listed for problem solving, change management, group work, self assessment, conflict resolution

We have used a workshop style learning environment to develop process skills. A workshop is designed to:

- help individuals become aware of how they do the skill,

- provide target skills for the effective application of the skill,

- give an opportunity to reflect and self assess,

- gives students a chance to gather evidence about individual use of the skill,

- provide benchmarks and encourage students to create goals for growth of the skill development.

Example transparencies, timing outlines and descriptions of some of the workshops are available (Woods,

1999d).

When to develop: Four options that have been used include:

- prescreen and only admit students into your program who have the skills already and continue with monitoring,

assessment and feedback (McMaster MD approach);

- prePBL workshops. Provide workshops before students start the PBL activities and continue with monitoring,

assessment and feedback (our approach in engineering);

- integrate the workshop skill development with the early PBL cases and continue with monitoring, assessment and

feedback (McMaster Theme school approach);

- integrate the development throughout the whole program with a published progression through four levels of skill

development as the students move through the program (Alverno College model).

How to Embed the development of process skills into PBL

To embed the skill development into the program requires action at the program or departmental level, general

considerations and at the individual PBL unit level.

M Program actions:

1. Make the process skill a valued and published outcome for your program.

Example:

“Graduates of this program will think rigorously and critically and solve problems effectively and

efficiently.”

2. Build student assessment and program evaluation into the program right from the start. Don’t tack this on as an

afterthought. How are you going to test skill in teamwork? How will an individual be graded? How do we evaluate

whether our efforts are successful? How do we benchmark and set goals for growth to improve our program?

For example, all students could complete the Heppner PSI (Heppner, 1986) and the Billings-Moos

inventories (Billings and Moos, 1981) at the beginning of the program to establish benchmark data. The

3-8

first PBL case could address issues of teamwork, the second addresses assessment and self assessment and

so on.

3. The goals, criteria and methods of assessment should be consistent across the whole program. Publish details of

where and how the skill will be developed. Alverno College provides an excellent example (Alverno, 1977).

For example, “Problem solving is defined in the course pack. The focus in the first semester will be on

developing...”

M General Principles

4. Treat the development of process skills with the same rigor and scholarship that you use in the development of

subject knowledge.

5. Create the framework for assessment and evaluation.

6. Make the implicit behaviour explicit. So much of the processing occurs automatically in our heads and in

the heads of other skilled practitioners. When asked “How do you do that?” she replies “I don’t know; it

just happens.” Our task is to take the skill and behaviour apart; discover what really is important based on

research, make goals and criteria and then present the experiences in bits and bites that can be mastered by

our students. This provides the context for skill development.

7. Encourage monitoring. Provide a checklist of questions the students can use to monitor their process skill

development. This provides the student with one form of evidence.

8. Ask students to reflect on the process. For each team meeting held, ask them to write out their reflections of

how they did the task. This provides the student with one form of evidence.

M For your PBL activity.

9. In your syllabus, restate the program outcomes, list the outcomes that will be addressed explicitly in your

course.

For example,

In these next PBL units, you will learn new knowledge and synthesize previous knowledge to solve

problems related to the cardiovascular system [the subject-specific skill development or new subject

knowledge gained] and you will develop skill in working in teams, (or critical thinking or problem solving

or communication).

10. Know what previous training in the target process skills the students have had and build on those.

11. Use terminology, assessment forms and standards of assessment consistent with the overall program.

12. Gather benchmarking data to aid in the program evaluation.

13. Usually assign students to the groups unless there is a compelling reason to allow them to select their own

groups. Keep the same groups for at least eight weeks.

14. Assign a chairperson for every meeting. Research has shown that groups function better with a designated

chairperson. Require the chairperson to prepare and circulate an agenda ahead of time. Ask the group to give written

feedback to the chairperson at the end of each meeting. The chairperson uses this input to reflect on his/her skill and

to set targets for development. This provides the student with one form of evidence.

15. Work with students at their stage of need. For example, don’t expect team performance and use team criteria for

assessment when the group is really a fairly effective group. Apply standards of assessment consistent with the level

of development. Secondly, we could help them see the big picture, and nurture, encourage and reward them on their

journey toward effective teamwork.

Example workshops

http://www.chemeng.mcmaster.ca/innov1.htm and click on MPS for the specific materials

Also see Woods “PBL: Resources to gain the most from PBL.” Chapter B for

3-9

What is PBL?

Managing Change MPS 49

Problem solving MPS 4

Interpersonal skills MPS 52

Group skills MPS 28

SDL MPS 36

Self assessment MPS 3

Goal setting, issue clarification MPS 30

Coping creatively with conflict MPS 45

A listing and brief description are given in downloadable form from Chapter 3 in Woods “Helping your students

gain the most from PBL” http://www.chemeng.mcmaster.ca/innov1.htm and click on PBL.

Reflections:__________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ Enrichment:1. Please help me understand how best to help you2. MRIQ3. MPS 4, MPS 28 (goals, targets, workshop, timing) FIRO-B; Norm meeting

References:http://www.chemeng.mcmaster.ca/Innov1.htm and download MPS materialsWoods “Problem-based Learning: resources to gain the most from PBL” Chapter D forobjectives. downloadable from http://www.chemeng.mcmaster.ca/innov1.htm and PBL; andChapters B and C.Woods “Problem-based Learning: helping your students gain the most from PBL” Chapter 3 downloadable from http://www.chemeng.mcmaster.ca/innov1.htm and PBL for an overview ofthe process skills.Woods, D.R. et al., “Tutored versus tutorless groups in PBL,” Am. J. Pharm. Ed., 60, 231-238,1996

3-10

Feedback forms

Some of the major feedback forms are given here for convenience:

M General awareness and skill for the process skills

M Assessment skills Form 301

M Problem solving Form 401

M Self-monitoring of stress management skills Form 501

M Group skills Forms 2800 ff

M Self-directed learning skill Forms 3600 ff

3-11

Form 000 Monitoring progress with processing skills (rate your awareness & skill at the end of each session, or week, on a scale from 0 [very low] to 10

[highest])

Processing skill

rating of Awareness rating of Skill

1 2 3 4 1 2 3 4

Change management and stress: anticipate and describe stages, identify stages of anger

and withdrawal and move through these, identify cause of confusion, explore corrective

actions, implement and reflect and manage impatience; manage stress by worrying only

about elements under your control, use positive self-talk, keep situation in perspective and

has positive activities to relax and maintain health and sparkle.

Problem solving: aware of process, uses a variety of methods, emphasis on accuracy,

active, monitors and reflects, organized and systematic, flexible and sees situation from

many different perspectives, objective and critically assess information, welcomes

challenge, spend time defining problems, uses fundamentals, and uses explicit criteria for

decision-making

Interpersonal and group skills: as a group addresses Task: problem defined, explored,

alternatives considered, criteria identified, task completed and look back and checked.

Monitoring occurred. Task completed on time. Group avoids contributing excessive and

irrelevant information; sticks to main themes without meaningless side tasks; and Morale:

group is relaxed, enjoyed working together, gave emotional support to each other and were

able to express disagreement or disappointment directly. members enthusiastic and

involved.

Team skills: maintenance of group attributes plus evolution of a group to develop team

goals and willingly foregoes personal goals for the benefit of the team, each knows, accepts

and performs role, decisions by consensus, conflicts are resolved so that 2+2 = 7

Chairperson skills: prepares and distributes detailed agenda, does the necessary detail

work ahead of time so as to have a "team event", facilitates the meeting [knowing when to

be neutral and when proactive facilitation skills are needed]

Monitoring progress with processing skills (rate your awareness & skill at the end of each session, or week, on a scale from 0 [very low] to 10 [highest])

3-12

Processing skill

rating of Awareness rating of Skill

1 2 3 4 1 2 3 4

Self-directed, interdependent learning: aware of educational fundamentals and uses these

to teach others, sets explicit learning objectives that have measurable criteria to identify

achievement and that are achievable with the resources available, considers wide range of

learning resources and willing to draw on peers as resources, agonizes through difficult

parts of the subject [instead of skipping these and hoping they are not pertinent], creates

forms of evidence to show accomplishment and applies these to both the knowledge learned

and the process used.

Self-assessment: knows and applies the issues in assessment, explores evidence of current

and of target knowledge/skills, identifies goals for growth and converts these to observable

objectives and measurable criteria, identifies the type of evidence needed to show

accomplishment and systematically and critically gathers that evidence over a sustained

period of time, makes adjustments as needed, searches for and uses both positive and

negative feedback, and makes a judgement based on the evidence, criteria, objectives and

goals that is deemed, by an outside evaluator, to be consistent, objective and valid.

Knowledge structure: embeds new knowledge by applying it to solve the problem and

elaborates and extends, identifies key "pointers" that link the theory to practice, identifies

basic underlying fundamentals and relates new applications and uses to past experience.

Asks "where else?" Knows the limitations of the knowledge.

Experience knowledge: identifies and memorizes a wide range of tacit numerical values,

processing procedures and rules of inference for the knowledge.

3-13

Goals: Content is well identified, goals are challenging and achievable, goals are written in

observable terms, goals are unambiguous, the "given" conditions are specified.

None of Few of these M ost features All of these

these behaviours but demonstrated behaviours

behaviours major omissions

G G G G G G G

1 2 3 4 5 6 7

Criteria: Criteria are consistent with the goals and are measurable and practical. The criteria are

challenging and achievable.

None of Few of these M ost features All of these

these behaviours but demonstrated behaviours

behaviours major omissions

G G G G G G G

1 2 3 4 5 6 7

Evidence: The type and quality of evidence gathered is consistent with the goals and criteria. The

evidence has been gathered conscientiously over a long enough period of time. The evidence is well

organized. The quality and extent of evidence is sufficient to allow me to judge the extent to which

the goals have been achieved.

None of Few of these M ost features All of these

these behaviours but demonstrated behaviours

behaviours major omissions

G G G G G G G

1 2 3 4 5 6 7

Process: The assessment process has been applied and as an independent assessor I concur with the

decision as to the degree to which the goals have been achieved.

None of Few of these M ost features All of these

these behaviours but demonstrated behaviours

behaviours major omissions

G G G G G G G

1 2 3 4 5 6 7

Strengths Areas to work on

______________________________ ________________________________________

____________________________ ________________________________________

________________________________

________________________________

________________________________ from D.R. Woods, "How to Gain the Most from Problem-based Learning" (1994)

Feedback about assessment Form 301

Feedback about problem solving Form 401

3-14

Attribute

Assessment

Awareness + can describe processes, can distinguish "exercise solving" from "PS."

- unaware of process; it's intuitive; cannot define a framework for PS.

Variety of PS

skills

+ can apply a variety of methods and hints.

- knows very few techniques; attempts to use a "one-step" solution.

Emphasis on

accuracy

+ checks, double checks, rechecks; concern for accuracy.

- concern for speed; unwilling to check.

Active + writes things down, m akes lists, prioritizes, m akes tables and sketches;

- thinks in head, can't keep track, stares at paper.

M onitors &

reflects

+ assesses continually, assesses potential of ideas & options; continually evaluates

and curtails; asks "where is this getting me?"

- does not monitor or assess; just does som ething.

Organized &

systematic

+ plans, anticipates, develops and uses a systematic plan.

- trial & error, impulsive, jumps around; no plan.

Flexible & sees

OPV

+ keeps options open; sees different points-of-view, willing to discard.

- quickly becomes fixed on one or two ideas or options even when mounting

evidence proves these to be untenable; unwilling to discard.

Use of

knowledge:

objective &

critically assesses

+ objective, learns from others, critically assesses data.

- fails to draw on past experience, egocentric, assumes & believes everything they

are told; accepts all information without question.

W elcomes

challenge

+ identifies disequilibrium as good; welcomes change and confusion.

- considers confusion to be "bad."

Time allocation + spends most of time in exploring, defining, planning and engage stages.

- spends m ost time doing, calculating, writing.

Overall approach + based on fundamentals, underlying principles, needs & goals.

- searches for sam ple solutions & cooks to try to make them work.

Decision-making + applies criteria, draws conclusions substantiated by evidence.

- makes a selection based on "gut" reaction.

Strengths Areas to work on

________________________________________ _________________________________________________

________________________________________ ________________________________________________

________________________________________

_________________________________________

_________________________________________ from D.R. W oods "How to Gain the M ost from PBL," 1994.

3-15

Monitoring checklist for stress management Form 501

Ideas Not for

me

Might

work

OK Use

now

1. Worry only about things over which you have control

2. Take care of yourself: exercise, eat and sleep regularly

3. Use destimulating activities: deep breathing, muscle relaxation

4. Use positive, not negative, self-talk:

Rate your self-talk:

don't know very negative neutral very positive

5. Plan ahead

6. Rename the stressful event:

don't know use anxious name neutral "thing" positive

7. Have a support system of family and friends.

don't know have none few some many

Have support system of traditions

don't know have none few some many

8. Use positive addictions that take your mind away to another

world: music, hot bath, crafts, hobbies

9. Be decisive

10. Put it into perspective: "It's not the end of the world!"

11. Use role models who have succeeded.

don't know have none few some many

Current stress:

Symptoms

none few some many

As measured by Holmes-Rahe (1967) or Holmes Gmelch (1983) <100 101-

300

301-

500

>501

3-16

Evidence-based targets for group skills (reprinted from references 4,17) Form 2800 .

Evidence-based targets Progress toward internalizing these targets

20% 40% 60% 80% 100%

M Performance improves when we have goals. [18]

M Assessment must be related to the goals [19]

M Both Task (getting the job done) and Morale (feeling good about the

group work and about how you have interacted with the other group

members) are important [19]

M Any group functions better with a chairperson [20]

M Chairperson and leadership are different; different people may

become leaders at different times.[20]

M Group evolution tends to follow a pattern described as by such

descriptors as “ forming, storming, norming and performing” .[19 ,20]

Schutz’s instrument FIRO-B seems to provide reliable insight as to[12,21]

the personal style of individuals towards other group members during

three of these phases.

M We can list the roles needed in both Task and Morale to make an

effective group. [19]

M When each person has a clear idea of roles and group norms, the

group functions better. [19]

M When groups are functioning effectively, about 70% of the time is

spent on the task; 15% on morale building activities and 15% of task

process activities (how the problem solving process is going;

summarizing ideas, guiding the process). [22]

M The products from groups or teams is improved when members have

different “styles” (in Jungian terminology some members are dominant

S, and some, dominant N). The products from groups tend to be inferior

when all the members “think and behave alike”. [23,24, 19,20]

M The quality of decisions, product, task is improved if group members

offer different perspectives, disagree and seem to introduce conflict into

the process. The trick is to manage the apparent conflict well. [19 ,23 ,24]

M The characteristics of “ meetings of individuals,” “effective groups”

and “teams” fall on a spectrum with sufficient differences that it is

useful to differentiate based on those characteristics

M In a decision-making mode, after 20 minutes of discussion on any

one topic, few new ideas are presented and repetition of previously

stated ideas occurs. [25]

Feedback about the group work. Form 2802

3-17

Form 2802Task: Problem defined, many issues and hypotheses explored, criteria listed and the issuesprioritized. Refrained from early closure. Task carried out and looked back at the result to assessit. Group agreement as to goals. Process was active with monitoring. Completed task on time.Group applied successive approximation and optimum sloppiness. Group avoided contributingexcessive information.

None of Few of these M ost features All of these

these behaviours but demonstrated behaviours

behaviours major omissions

G G G G G G G

1 2 3 4 5 6 7

Morale: Group relaxed; enjoyed working together. They gave emotional support to each otherand were able to express disagreement or disappointment directly. Seven fundamental rightspreserved. Members are enthusiastic and involved.

None of Few of these M ost features All of these

these behaviours but demonstrated behaviours

behaviours major omissions

G G G G G G G

1 2 3 4 5 6 7

Individual Contribution to Task and Morale

____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Group Strengths Group Areas to work on_____________________________________ ________________________________ ____________________________________ _____________________________________________________________________________________________________________________________________________ from D.R. Woods (1995)

3-18

Form 2804, Self and peer assessment of contributions to team work (developed by C.M. Crowe and used with his

permission)

______________________________________

Name

Assessment of the group

(10) Group worked well together, making the project more interesting.

(5) Group usually worked well together, with occasional difficulties

(0) Group could not work together at all, making the project more

difficult

Please enter a mark from 0 (worst) to 10 (best) in each row, for yourself and the other members of your team or group. If you cannot make an

assessment, enter U for Unable to Observe. Your assessment should be about performance and should be based on evidence you have seen or

heard.

Enter your initials in the first cell to the right and those of the other members of

your group in the remaining cells

self

Leadership

(10) Provided direction, inspired the others

( 5 ) Willing follower, took direction easily

( 0 ) Frustrated the group, blocked progress, criticized the others

Cooperation

(10) Worked readily with the others, contributed to the group work

( 5 ) Cooperated with occasional prompting

( 0 ) Rarely contributed, worked mostly alone, had to be coerced into

cooperating or would not cooperate.

Initiative

(10) Produced good ideas which helped others, solved problems

( 5 ) Accepted other’s ideas and improved on them

( 0 ) Criticized other’s ideas, never contributed his/her own ideas’

Attitude

(10) Positive, encouraging the others to work better

( 5 ) Neutral, worked well with the group without enthusiasm or grumbling

( 0 ) Negative, complained about the project, worked unwillingly with the

group

Computation

(10) Helped others to understand and use computer tools better

( 5 ) Good but not innovative in using the computer tools

( 0 ) Uninterested or unable to use computer tools effectively; needed

constant help to becom e useful at computing

Effort

(10) Worked very hard on tasks assigned by the group or on his/her own

( 5 ) W orked reasonably hard, given other courses and commitments

( 0 ) Did not work much at all, tasks were either not done or were

unsatisfactory

3-19

Enter your initials in the first cell to the right and those of the other members of

your group in the remaining cells

self

Research

(10) Found m any additional published papers, contacts or internet sites

which greatly helped the project

( 5 ) Read only the assigned material and used that knowledge effectively

( 0 ) Had not read any material about the project, relied on others for

information

Communication and written reports

(10) Organized and wrote major parts of the report

( 5 ) Contributed his/her share of the writing of the report

( 0 ) M ade little or no contribution to writing of reports.

Individual assignments

(10) Did all of them on his/her own, helped others

( 5 ) Did most of them, perhaps with som e help from others

( 0 ) Either did few of them or m ost of them handed in were copied.

Pertinent technical/subject Know ledge

(10) Excellent grasp of the technical fundamentals for this project and

shared and used this effectively

( 5 ) Reasonable understanding of the basics and used this effectively

( 0 ) Little or superficial understanding, relied on others

3-20

Some evidence-based targets for Lifelong learning skills, Form 3600 © copyright, Donald R. Woods, 1999 (10-16)

Lifetime learning we define as learning how to learn so that whatever comes our way we empower ourselves to

master the new knowledge.

Evidence-based targets Progress toward internalizing these targets

20% 40% 60% 80% 100%

M Consider peers and classmates as resources to help me

see my learning needs, to help me plan my learning and to

provide new ideas so that I can learn from them.

M Able to assess learning needs realistically.

M Able to create observable, unambiguous and achievable

learning objectives to match or satisfy my needs. Use these

to assess progress.

M Can relate to teachers and instructors as resources,

facilitators and helpers rather than as the sole source of

knowledge. Have acquired an attitude toward learning

comparable to Perry level 5.

M Able to identify people and material resources needed to

achieve my learning objectives.

M Able to shift from being a dependent learner through

being an independent learner to being an interdependent

learner.

M Able to devise a time plan and stick to it reasonably

well.

M Willing to assume responsibility and ownership for the

tasks in learning: (goal setting, resource identification,

learning, assessment).

M Meets contract commitments to teach others.

M When teaching others, uses the principles of learning

and addresses differences in learning styles (instead of

“reporting information” and expecting the learner to sort it

out).

M When learning a “difficult” topic, willing to accept the

challenge to unravel the complexity (instead of skipping

over it and “hoping it won’t be on the exam”).

3-21

Feedback to ___________________________ for Unit ____ Date ___________

Present & on time: G Present but late by ___ min. Absent G

Quality of Knowledge: good intellectual understanding of the topic, the material supplied was complete and

appropriate.

None of A few but Most of these. All of

these. major omissions. these.

O__________O_________O_____________O_____________O_____________O_____________O

1 2 3 4 5 6 7

Quality of Instruction: he/she was here on time, the presentation was focused on the new knowledge; good

choice of material and medium with effective communication and resource material supplied.

None of A few but Most of these. All of

these. major omissions. these.

O__________O_________O_____________O_____________O_____________O_____________O

Followup: from this presentation I will have to:

Must study subject Major self-study Some No self-

on my own; needed. I have some self- study of

I learned starting references study of the basics.

nothing from from your presentation. the basics. I want to

your presentation. reflect

about the ideas.

O____________________O_____________O_____________O_____________O_____________O

Strengths Areas to Improve on

_____________________________ ________________________

_____________________________ ________________________

_____________________________

_____________________________

_____________________________

_____________________________ from D.R. Woods, "How to Gain the Most from PBL," (1994)

Feedback for interdependent, self-directed learning Form 3601

3-22

Feedback for PBL/SDL a120 3602

Situation$1

Issues

Number identified: 1 2 3 4 5 6 7 >7

Agreement with tutor <50% 50% 60% 70% 80% 90% 100%

Knowledge/skills to be learned

Consensus among group little some a lot complete

Agreement with tutor's list little some a lot complete

Learning objectives

Quality poor fair OK good excellent

Learning

Quality of questions asked during the teach session none some astute excellent

Willingness to continue to contribute <50% 50% 60% 70% 80% 90% 100%

Your Attitude

Perry shift 2 3 3.5 4 4.5 5

******************************************************************************

3-23

MPS 36 3603 Learning preferences and attitudes a120

Name Attitude: Perry

scale

Learn style Jungian

before now strate

gic

rote mean

ing

S value and implications

for learning

T value: combine with S

and implications on test

questions

value implication value implication

You

a120

3-24

Table 29-1: Date of meeting: _____________ Group: _____ Chairperson: __________________________

Agenda circulated ahead of time, yesG no G; start on time, yes G , no G;. end on time, yes G , no G!Task: Problem defined, many issues and hypotheses explored, criteria listed and the issues prioritized. Refrained from early closure. Task carried

out and looked back at the result to assess it. Group agreement as to goals. Process was active with monitoring. Completed task on time. The

accuracy in the group=s answer matched the time available. Group avoided contributing excessive information.

None Few but M ost features All of these

major omissions demonstrated behaviours

G G G G G G G

1 2 3 4 5 6 7

Total

_______

Agenda: Names, time, place,

purpose, prework, bring, topics,

timing

Use of agenda: Keep on

topic, effectively & not

disruptively keep on time,

20 min rule

Facilitation as needed when team functioning below norms on Task

15 to 20 Excellent & ahead of time Used very well M ega positive: clarified, brought back on task, monitored Task

10 to 14 Poor & ahead of time OK Some positive as needed: clarified, back on task, monitored Task

5 to 9 Excellent but handed out at start Fair Neutral because working well

0 to 4 Poor and handed out at start

0 to -4 Excellent created at start Poorly Neutral when help was needed or unskilled and unable to help

-5 to -9 Poor created at start

-10 to -

15

No agenda Not used Kept dominating as leader, when facilitator need, Solved problem

for group, when they didn=t want that solution

-16 to -

20

M ega negative, imposed own ideas and answers. Dictator

M orale: Group relaxed; enjoyed working together. They gave emotional support to each other and were able to express disagreement or

disappointment directly. Seven fundamental rights preserved. Members are enthusiastic and involved. None Few but M ost features All of these

major omissions demonstrated behaviours

G G G G G G G

1 2 3 4 5 6 7

Total Agenda: Use of agenda: Facilitation as needed when team functioning below norms on

M orale

15 to 20 Excellent & ahead of time Used very well. Comfort

high because know goals

Skilled observer of interpersonal process, praiser, facilitates,

conflict resolution and provides tension relief as needed.

10 to 14 circulated ahead but missing

prework & bring

Comfort moderate, can infer

what to do ahead of time

Some skill and intervention

0 to 9 Fair No intervention because all working well

-1 to -9 Some frustration; feel ill-

prepared because didn=t know

what to prework or bring

Poor No intervention when help was needed or unskilled in facilitation

-10 to -

15

No agenda, High distress Not used, high distress

because unclear as to where

going

Some, causes lower-than-norm morale because of criticism,

defensiveness, contemptuousness, stubborn, conflict

-16 to -

20

M ega causes tension, seeks personal goals, critical, abusive, self

centered and defensive, contemptuous, ignores others

Personal Satisfaction with decisions made:G 0 extremely dissatisfied...5 OK..... 10 very satisfied !Strengths Areas to work on

_____________________________________ _______________________________

____________________________________ _______________________________

_____________________________________

____________________________________

___________________________________

4-1

4. PBL: Decisions for Planning and Action: c/pbl/planWhy? When? Who? Where? What? How? Donald R. Woods copyright 2000 Donald R. WoodsMcMaster University download and use but please acknowledge source. ThanksHamilton Canada

Here is an approach to decision-making. This is biased. This is based on my experience and may not work for you.

Step 1. Profile of your graduates from your program:distribute 100 points among:

subject knowledge: _______skills:

problem solving _______team work _______self assessment _______manage change _______lifelong learning _______other: _______

attitudes:ethics _______environment _______impact on society _______other _______

If 100 points to subject knowledge, then go to Step 4If some points given to skills, then go to Step 2.___________________________________________________________________________Step 2. Skills are important to develop for your graduates. Then: we should2a. Publish a calendar or university catalog description of outcomes. Example Figure 2-1.

2b. Use standard consistent forms and criteria for assessing student performance throughout the whole program (sothat we don’t use Form A in course 202; then form B in course 302; form D in course 444.) ExampleCommunication skills Form Figure 2-2.

2c. Ensure that assessment is consistent with development and goals. For example, 80% on subject knowledge and20% on teamwork. Example syllabus. Down load from Chapter D, “PBL Resources to gain the most from PBL”which is downloadable from http://www.chemeng.mcmaster.ca/innov1.htm and click on PBL

2d. Select methods for program evaluation and benchmarking progress.

Please identify specific skill and go to Step 3.________________________________________________________________________________________Step 3. Specific skill development.Common to the development of most skills are the following features:

3.1. We create observable, published goals and measurable criteria describing the skill explicitly. Examples seeChapter D, “PBL Resources to gain the most from PBL” which is downloadable fromhttp://www.chemeng.mcmaster.ca/innov1.htm and click on PBL

4-2

Figure 2-1 Example University catalog description

“Graduates of this program in Chemical Engineering will:- be technically sophisticated with an empahsis on the fundamentals.- be not only computer literate but also able to program effectively and efficiently and would be familiar with a wide range of process flow-sheeting programs, such as HYSIS.- have enriched experiences through senior electives drawn from our Department’s research expertise.-have skills in lifelong learning, communication, problem solving, interpersonal and group work and - know how to create hypotheses, perform experiments and draw valid conclusions.- have a professional attitude with- concern for safety and the environment- ethical standards.”

Figure 2-2 Example feedback form for the product of written or oral communications© Donald R. Woods 2000

Criterion Oral Written Assess-ment

Audience:answersquestions &needs of theaudience

positive answers my questions, speaks at mylevel, seems interested in me

understandable, writes at a level I canunderstand, define words; main theme isquestion I want answered.

negative talks above me, uses jargon,information dump with no attempt to answer myquestions; tells me what he/she knows; giveschronological history

writes above me, uses jargon, inconsistent,complex ideas are not explained;

Content:providesfactualevidence tosubstantiateclaims

positive supports statements with evidence gives supporting evidence and elaboration;counter arguments given; references given

negative no supporting evidence, want me tobelieve because “I say so.”

no references, makes arguments withoutlogic; uses humor and narrative

Organization:well organized

positive: advance organizers, gains interest,provides summary; gives transitions from onesection to the next

good coordination and subordination; nosingle sub-points. Advance organizers,summary, transitions between sections

negative rambles; jumps from topic to topic, nosummary, no pattern

outline suffers from faulty coordination andfaulty subordination;

Style:interesting

positive: interesting, variety of visual and voice,good eye contact, gestures, quality Power pointor visuals, clarity, quality and good diction

coherent, interesting writing style. Variety insentence length. Deletion of extraneous andambiguous phrases/words.

negative no eye contact, crazy gestures,incoherent sentences, Uummm, Ahhhs

vague and boring words, long involvedsentences

Format/mechanics:followsexpectedstandards

positive. Begin and end on time; correct use oflanguage; credible, visuals legible

correct format and referencing, grammar,spelling & punctuation are correct Neat.

negative runs over time; visuals upside down;grammatical mistakes

did not follow conventions in discipline

4-3

3.2. We agree on forms of evidence that students can use to document achievement. See papers by Woods in thejournal Chemical Engineering Education called “Assessing...” published in 2001. Woods, Taylor, Jaffer “Teams”(2001c and d); Woods et al., “Problem Solving,” (2000b); Woods “Lifelong learning,” (2000a); Woods (2000a).

3.3. We expect students to write assessment journals for the progressive development of the skill. For example seeChapter F, “PBL Resources to gain the most from PBL”

3.4. We select an appropriate learning environment to develop the skill and student confidence in the use of the skill.Table 3-1 lists recommended learning environments to develop different skills.

Table 3-1. Suggested learning environments to develop skills

Skill Learning environment comment and resources

communication, written personal projects with emphasis onboth the process and the product

we have found that the key is to select topics ofinterest to the students plus personal guidance.See Postface, page P-6 in “Helping yourstudents gain the most from PBL” Helping

communication, oral videotape plus personal feedback

communication,interpersonal

workshops In the MPS program, MPS 52 and 26, 27

problem solving workshops In the MPS program, about 75 h of workshopswith a minimum of 4 h, MPS 4. Chapter 3 inHow to Gain the Most from PBL, HTGTM

team work workshops In the MPS program, about 45 h of workshopswith a minimum of 6 h: conflict resolution, MPS45; group process, MPS 28. Chapter 5 in Howto Gain the Most from PBL, HTGTM

change management workshops In the MPS program, unit MPS 49; Chapter 1 inHow to Gain the Most from PBL, HTGTM

self assessment workshops; personal goal project In the MPS program, MPS 3 and in Woods,Marshall and Hrymak (1988) and Chapter 9 inHow to Gain the Most from PBL , HTGTM

lifelong learning small group, self assessed, self-directed PBL see Step 7.

In addition in the MPS program MPS 36 and inHow to Gain the Most from PBL, HTGTM

With small group, self directed, self assessed PBL students have an opportunity to practice problem solving, selfassessment, group skills and communication skills. Our experience has been that the use of PBL does not initiallydevelop or build the skills.

3.5. Calendar or university catalog descriptions should include the development of the skills as explicitly plannedactivities. Example, “Course YYY, The... subject content. Team skills and self assessment skills. 3 lectures perweek, one 2 -h tutorial. Prerequisite.... Corequisite...”

3.6. We need to identify where in the overall curriculum the skills are a) initially developed. b) used. Planner A inTable 3-2. Example from Spencer-Chapman (1996) for a traditional course/subject curriculum.

4-4

Table 3-2 Planner

Outcomes Subjects, courses or PBL cases

A B C D E F G H I J K L M N O

Subject knowledge

Skills:

communication

problem solving

team work

change management

self assessment

lifelong learning

Attitudes:

ethics

At this time we have completed our look at the development of process skills. We will consider some details of howto do this in Steps 18, 19 and 20. But that’s a story for later. If we have not selected small group, self directed, selfassessed PBL as an important option, then consider other issues relating to student learning by going to Step 4.____________________________________________________________________________________________ Step 4. In learning subject knowledge we want to improve future recall by our students of knowledge learnedTo aid future recall, an option is to pose a problem first and help students to learn the new knowledge needed tosolve a practical problem . One learning environment to use, but not the only one, is small group, self directed, selfassessed PBL see Step 7. Table 4-1 gives suggestions on how to aid future recall for the students.

Table 4-1 Knowledge structure in Lecture versus PBL

Conventional “lecture” style small group, self directed, self assessed, PBL

knowledgestructure

- pose problem first- teacher provides structure- use concept maps-use cause-effect diagrams- use elaboration

- occurs naturally in this context

cues to connectproblems torequiredknowledge

- need to emphasize this and spendtime developing- concrete practical examples- bring in hardware, real problems

- occurs naturally in this context

4-5

That addresses the development of knowledge structure. But what about improving student learning? Please go toStep 5, or to Step 7 if you have selected small group, self directed, self assessed PBL_________________________________________________________________________________________Step 5. In learning subject knowledge we want to improve student learning. Principles to improve learning include:1. Chickering & Gamson (1987) have given the following suggestions to improve learning:1. students are active2. students work cooperatively3. quality of teach-learn interaction4. students receive prompt feedback5. students have a clear time on task6. students work in an environment that expects success7. the learning environment caters to the student’s personal learning styleand some additional suggestions include8. assessment should be owned by the students9. there must be published goals/criteria10. attention span = 20 min11. students will wait 20 s before attempting to answer an question posed by the teacher12. we need to address knowledge, skills and attitude.

From a slightly different perspective Ramsden & Entwistle (1983, 1985, 1981) suggest that if we want to promotestudent’s “Deep learning” instead of “surface learning”, then the learning environment should be:1. good teaching2. openness to students3. freedom to choose4. clear goals & standards5. vocational relevance6. social climate

7. (-workload)8. (-use of formal lectures)Table 5-1 considers some of these suggestions and gives options to achieve these in the learning environment.

Table 5-1 Comparison between “Lecture” and PBL

Ideas to improve learning and tonurture “deep learning”.

Conventional “lecture” style small group, self directed, selfassessed, PBL

Active Possible. Use - feedback lecture- cooperative learning,- “turn to a neighbour and...”- Talk Aloud Pairs Problem Solving

Occurs naturally

Cooperative Possible. Use- informal cooperative groups- formal cooperative groups- study groups- criterion-referenced assessment

Occurs naturally

4-6

Ideas to improve learning and tonurture “deep learning”.

Conventional “lecture” style small group, self directed, selfassessed, PBL

Quality of teacher-studentsinteraction

Challenging but- know and call them by names- take an interest in them as persons- attend student events- arrange in-class celebrations ofstudent achievements.

For tutored groups, this is easier.For tutorless groups, this is morechallenging than for “lecture”.

Prompt feedback Possible but more difficult as classsize increases.perhaps use peer and selfassessment.

Occurs naturally.

Time on task Easier because “you” are themanager.

Motivation helps to keep them ontask.

Environment expects success Needs special effort. Needs special effort.

Personal style of learning Possible. Easier because of the small size ofthe group.

Students own assessment Options available for students toown some or part of theassessment:- contracts;- students create part or all of theexam- self and peer assessment countsfor part of the grade.

Occurs naturally in self-assessed.

Clear goals and criteria Teacher provides; easy to do butthe teacher must publish.

Training needed but should occurnaturally; challenging because thestudent’s own it; monitoringneeded.

Attention span 20 min Teacher needs to limit “teachertalk” to 20 minute bursts.

Occurs naturally.

20 s response time Teacher needs to help studentsovercome.

Overcome naturally because ofmotivation and small size of group.

Addresses knowledge, skills andattitude

Usually focuses on knowledge:need to provide development andopportunity to further develop.Self assessment, conflict resolution and change management areusually not included.

Opportunity to develop/ furtherapply the skills and attitudes. Oftenthe tutor supplies these and studentsdon’t acquire skills.

Freedom to choose: Possible -use Branda’s PBEE (1993-95)

students empowered with theprocess; tutor guides and monitors.

Motivation Possible. High.

4-7

Ideas to improve learning and tonurture “deep learning”.

Conventional “lecture” style small group, self directed, selfassessed, PBL

Vocational relevance Possible. Occurs naturally if problems areselected with care.

Social climate Possible to create. Occurs naturally.

It is easier to achieve most of the characteristics that improve learning by selecting small group, self directed, selfassessed PBL see Step 7. If lecturing is the preferred option, then each characteristic can be addressed separately.It’s possible. It is just more work. please go to Step 6 or directly to Step 7.____________________________________________________________________________________________Step 6. In learning knowledge, we want to minimize student’s preference for rote or surface learning and shift tomore emphasis on “search for meaning” or deep learning. Research evidence suggests that learning environments that are primarily lecture (with teacher control) tend toencourage rote learning; the preference for deep learning decreases. Such environments are characterized by a lowvalue of the Course Perceptions Questionnaire (or equivalent) with values 15 to 22. The use of small group, self -directed PBL environments that are student centered tend to encourage deep learning with rote learning decreasing.The Course Perceptions Questionnaire has values of 35 ro 45. Evidence from Woods (2000b), Coles (1985), Regan-Smith et al. (1994) and Woods Hrymak and Wright (2000e).To encourage students to search for meaning and use deep learning practices, use small group, self directed, selfassessed PBL see Step 7.___________________________________________________________________________________________Step 7: We have selected small group, self directed, self assessed PBL as a learning environment toa) develop lifelong learning (from Step 3). This approach does not do much to build skill in problem solving,communication, team work and self assessment (Woods, 1993a and b; Norman 1988, Norman and Schmidt, 1993,and Woods et al., 1997) However, this learning environment provides an excellent opportunity to bridge and extendthe skills.b) to aid future recall (from Step 4).c) to improve student learning (from Step 5).d) to encourage students to abandon a surface learning approach in favour of a preferred deep learning approach(from Step 6).M But are the students ready? Perhaps use Perry’s instrument (see How to gain the most from PBL, HTGTM,Chapter 1, Perry, 1970, Felder et al., 2000 and Woods et al., 2000c). For small group, self directed, self assessedPBL, we would like all students to be characterized by Perry level 4 to 5. Students who are not ready may becharacterized as being at levels 2 to 3.Are the students:- ready to accept the teacher’s role as coach instead of lecturer? see Woods (1998). If not, see Step 8.- ready to accept their role of ownership of the learning process? If not, see Step 8.- confident that they can succeed in student-centered activities? If not, see Step 8.- skilled and confident in problem solving, team work, self-assessment, conflict resolution and communication? Ifnot, see Step 2.- knowledgeable about PBL? If not, then see Step 9.

If yes, then go to Step 10.____________________________________________________________________________________________

Step 8: Preparing the student’s attitude about their role in small group, self directed, self assessed PBL .

4-8

Use the Perry inventory as a diagnostic.- run a workshop on managing change addressing the grieving process, stress and time management and building selfconfidence. see “Problem-based Learning: Resources to gain the most from PBL” Chapter B. MPS 5, MPS 17, MPS11, MPS 49 (see also Woods et al. 1997a). - use Chapter 1 in HTGTM.

Please go to Step 10.____________________________________________________________________________________________Step 9: Helping students understand small group, self directed, self assessed PBL.Some options include:- show a video of students in small group, self directed, self assessed PBL - Socratically run a session for 5 to 500 students to demonstrate the process.- use HTGTM as a required text and work through it. - run a workshop on “What is PBL?” see “Problem-based Learning: Resources to gain the most from PBL” Chpt B.Please go to step 10. ____________________________________________________________________________________________Step 10: Are the faculty ready for small group, self directed, self assessed PBL?Perhaps use Woods’ “My Role is..” questionnaire, and consider dimension E in particular. This is reproduced inTable 10.1.

10.1 Are the faculty ready to become student-centered and empower students with tasks in the learning process?Data for college and university teachers in conventional programs show responses of 10.3 with standard deviation of3.5 for item E, page 11, on a scale from 0 to 20. Ideally, we would want faculty responses > 10. If your value is not> 10, perhaps reconsider whether small group, self-directed, self-assessed PBL is an option for you. Return toTable 5.1 and consider how you might modify the “lecture” style to improve learning.

10.2 Are faculty ready to assume the role of coach? “Guide on the side” and not “sage on the stage”? If not, perhapsreconsider whether small group, self-directed, self-assessed PBL is an option for you. Return to Table 5.1 andconsider how you might modify the “lecture” style to improve learning.

10.3 Are faculty ready to maintain standards by assessing the process instead of traditionally being the assessor? Ifnot, perhaps reconsider whether small group, self-directed, self-assessed PBL is an option for you. Return to Table5.1 and consider how you might modify the “lecture” style to improve learning.

10.4 Do faculty have confidence, skill and coaching ability in the process skills of problem solving, team work,conflict resolution, change management, astute questioning and critical thinking? If not, go to Step 3.1, 3.2 3.4 andconsider having a series of workshops to train tutors. A possible outline for training sessions is given in Table 21.1.More on tutor training will be discussed in Step 23.If students are ready (from Steps 8 and 9) and the faculty are ready, then please go to Step 11.____________________________________________________________________________________________Step 11: Plan the curriculum for small group, self directed, self assessed PBL For your context you might:

- convert the whole program to PBL, see Step 12.- use PBL in one course in an otherwise conventional program, see Step 13.- use PBL as part of one course in an otherwise conventional program, see Step 13.

____________________________________________________________________________________________

4-9

Table 10-1: My role is .... Questionnaire, MRIQ (based on Magnesen, 1985; Paul, 1992 andWoods, 1991) Donald R. Woods, June 1997The following 18 items are arranged with options (a and b or a,b and c). Each option represents a preference youmay or may not hold. Rate your preferences for each item by giving a score from 0 to 5. (0 means you really feelnegative about it or strongly about the other options. The scores for a and b, or a, b and c MUST ADD UP to 5 (0and 5, 1 and 4, 2 and 3, 1, 2 and 2) Please do not use fractions.I think my role as a teacher is:1a. I have a basic conviction that I can make adifference.+),.)-2a. My role is to maintain high standards and failthose who do not make the standards.+),.)-3a. My role is to uncover material so that studentsunderstand. +),.)-4a. My role is to make learning fun.+),.)-

5a. My responsibility is to teach subjects.+),.)-

6a. Students must grow personally as well asintellectually+),.)-

7a. Teaching, research, consulting are allopportunities to help others learn. The only differenceis the client and the "class size". Undergraduateteaching and graduate research are a seamlesscontinuum of learning.+),.)-

8a. Teaching and learning are a two-wayresponsibility. If students fail its partly my fault.+),.)-

9a. If students understand my presentation, they willautomatically remember the material. Learning is rotememorization and recall of facts.+),.)-

1b. People come to me with basic attitudes and theywon't change.+),.)-2b. My role is to help each succeed and make themost of his/her abilities.+),.)-3b. My role is to cover the material in the curriculum.+),.)-4b. Learning is serious business. My role is to be wellprepared. +),.)-5b. My responsibility is to teach people.+),.)-

6b. The sole purpose of college is intellectualgrowth.+),.)-

7b. Undergraduate teaching is the burden I must bearto allow me to do research.+),.)-7c. Graduate research is the burden I must bear toallow me to teach undergraduates.+),.)-

8b. Learning is one-way; I do my thing, and it's up tothe students to learn. +),.)-

9b. Understanding is not remembering. Students and Ineed opportunities to see new concepts inperspective, to understand their limitations and toreach conclusions. Learning is active, independentand self-directed. +),.)-

4-10

10a. Students should learn knowledge and theprocesses for working with that knowledge.Knowledge cannot be separated from thinking.+),.)-

11a. The development of values is an integral part ofmy instructional plan. Values play a significant rolein my student's future success.+),.)-

12 a. Students should self-assess. My role is toensure that the assessment process used by thestudents is valid. I consider the goals, criteria and thequality of the evidence.+),.)-

13a. My role is design the whole learning process.The students just have to follow my design.+),.)-

14a. I am a resource to help students learn; studentshave the principal responsibility for making andcarrying out their own plans.+),.)-

15a. My role is to help students with academic andintellectual issues. It's not my responsibility to getinvolved with their personal and social life.responsibility.+),.)-

16a. I prepare the detailed learning objectives, theassessment criteria but publish general guidelines forthe students; to do otherwise provides too muchdetail; it's overwhelming for the students.+),.)-

10b. All students need to learn in college isknowledge.+),.)-

11b. The development of values is the responsibilityof the home and/or the religious component of thestudent's life. You can't measure "value"development; therefore, it is inappropriate to includethis area in one's goals.+),.)-

12b. Assessment of students is my responsibility. Icreate and mark all the exams that are used tomeasure student learning.+),.)-

13b. My role is to empower students with allelements in the learning process: goals, choice oftext, assessment...+),.)-

14b. I am the source of knowledge. I have theadvanced training to be shared with them.+),.)-

15b. My role is to help students with academic andintellectual issues and to help them with personalproblems.+),.)-

15c. My role is to help students with academic andintellectual issues and to informally socialize andattend student events. +),.)-

16b. I publish detailed learning objectives andassessment criteria.+),.)-16c. Students should prepare detailed learningobjectives and assessment criteria. I monitor theprocess to ensure the standards are met.+),.)-

17a. My role is to help them solve problems similar

4-11

to those they will encounter in professional practice.+),.)-

18a. I teach new knowledge. My role is to presentwell-organized explanations expressed to match thestudent's learning style.+),.)-

17b. My role is to ensure that they know thefundamentals. I use problems that help develop andtest that understanding.+),.)-

18b. All new knowledge bears some relationship topast knowledge. My role is to activate the pastknowledge and help students see the relationshipbetween the new and the old.+),.)-

***********************************************************************************Scoring:

A.1,2,6,9

1a ______ 1b. _____ 5b ______ 5a. _____ 6a ______ 6b _____10a ______ 10b _____ 11a ______ 11b _____15b+c ______ 15a _____

TOTAL ________ ________

B.1,9 2b ______ 2a ______16b+c____ 16a _____

TOTAL _________ ________

C2,3,6 9b ______ 9a _____10a ______ 10b _____17a____ 17b _____

TOTAL _________ ________

D2,3,6 3a ______ 3b ______ 8a ______ 8b ______ 9b ______ 9a ______10a ______ 10b ______14a _______ 14b ______16b+c _____ 16a ______17a _____ 17b _____18b _____ 18a _____

TOTAL ________ ________

E2,6,7,9 12a _____ 12b. _____13b _____ 13a ______14a _____ 14b ______16c _____ 16a+b ____

TOTAL _________ _________

F2,6,8,9 16b+c _____ 16a _____

Step 12: Congratulations! Everyone in your department is shifting to small group, self directed, self assessed PBL.

4-12

This provides an opportunity to make the greatest impact. - Check first that all faculty have bought in and are ready, revisit Step 10.With the students experiencing all subjects/courses in this format,

12.1 We have great chance to integrate the student’s learning across subjects.

12.2 We usually have sufficient faculty resources to have one tutor with each group of five to six students.

12.3 We can do a better job of formally preparing students to capitalize on their experience.

12.4 We can timetable student groups around limited experimental and laboratory equipment.

Please go to Step 14. ____________________________________________________________________________________________Step 13: Congratulations! You are not waiting for others. You are going ahead with something over which you havecontrol- your course. You can introduce small group, self directed, self assessed PBL as a pilot project for aminimum of three weeks or three cases or you might convert your whole course to this format. With the students experiencing PBL only in your course,

13.1 We have a limited chance to integrate the student’s learning across subjects in our cases. (The implications arediscussed later in Step 15)

13.2 Since we have classes of size > 5 students we either work Socratically with many groups of five students or wework with tutorless groups. (This will be discussed later in Step 17.)

13.3 We have to be creative in preparing students to capitalize on their experience. (This will be discussed later inStep 23.)At this stage, however, we should start by identifying systems, setting goals, creating the problems and resources. Please go to Step 18.____________________________________________________________________________________________Step 14: Identifying systems and setting learning goals for Phases in the context of a full PBL program.

M 14.1 Identify systems: Conventional programs consist of a series of subjects or courses that are taken is somesequence: Chemistry, Physics, Mathematics and so on. Small group, self-directed, self-assessed PBL consists of aseries of problems (that require the application of some Chemistry, Physics, Mathematics and so on as needed). Wecan eliminate the narrow focus on subjects and integrate the knowledge around practical systems. The practicalsystems, and the learning objectives for these, are selected so that within the program the students will achieve thesame knowledge objectives as they would have achieved in conventional programs. However, the selection of thetopics in a subject is on a need-to-know basis; rather than on a learn it now and you’ll see later why you need this.

For example, in Medicine the knowledge addressed might be (Hamilton, 1976):Phase I: 10 weeks (10 problems) to introduce health care problems, universal concepts of structure, function andhuman behaviour, interviewing and clinical skills. (see p 9-5 in HTGTM )Phase II: 12 weeks. the reaction of the body to stimuli and injury, how cells, tissues and the whole organismresponds to inflamation, neolasia, metabolic homeostasis, ischemia and behaviour.Phase III: four session of each 10 weeks on each of the major body systems: 1) blood and gastrointestinal system, 2)cardio respiratory system, 3) neuroscience, locomotor system, psychiatry and 4) renal physiology and electrolytes,reproduction system and endocrinology. Phase IV: clinical clerkship: 8 weeks medicine; 8 weeks surgery; 4 weeks family medicine; 4 weeks psychiatry,; 4weeks obstetrics and gynecology; 4 weeks pediatrics and 16 weeks electives.

Detailed examples have not been published in engineering but some illustrative themes might be

Phase I: 10 weeks (10 problems) chemical engineer functioning in society. problems of design, processimprovement, product design, trouble shooting in a range of processes (to introduce mass balances, finances,

4-13

chemistry, analytical chemistry, biochemistry, statistics, sampling, environmental impact, legislation, safety,sampling, measurement and process control, mathematics, asking questions, data resources, consulting and teamwork).Phase II: 12 weeks (12 problems) energy mechanical and thermal; idealized behavior. Phase III: 40 weeks (40 problems): non-idealized behavior and a focus on separation systems, mixing/size change,reactions. Phase IV: design, process improvement, trouble shooting apprenticeships.

From such an integrated view, we can then identify overall learning goals (for knowledge, skills and attitudes) forthe Phases.

An example matrix worksheet is given in Table 14-1 with columns for the traditional “subjects” or “courses” in aprogram and rows for the “problems”. For each column we have a conventional listing of the learning objectives foreach course. The various goals for “subject content” or for “essential concepts” are scattered throughout the columnso that we address each objective in some problem.

Table 14-1 Matrix of traditional subjects and problems

Learn-inggoals

Case Traditional subjects or courses

organicchem

investment;finance

samplingstatistics

environmentalchem.

massbalances

analyticalchem

physics math

Goalsfor Phase

1 x, y x y y y y

2

3

4

5

6

M 14.2 Decide on a suitable cluster of goals for a Phase or two-semester period.

Consider all the knowledge, skills and attitudes and express these in relatively large clumps that relate to the Phasesoutlined in Step 14.1

Please go to Step 15.__________________________________________________________________________________________Step 15. Do we have the tutors for a full PBL program?

Even though you have the whole program converted to PBL, sometimes we find we don’t have enough facultyresources to have a tutor for each group of 5 to 6 students. A temptation might be to increase the size of the studentgroup to 7, 8 11, 15. Please resist the temptation. Work with tutorless groups. (More will be said about tutorlessgroups in Step 24.)

Hopefully since the whole program is all PBL we can have one tutor for each group of five to six students.

Here are some guidelines to decide on the number of tutors available and needed.In a conventional program, the typical teaching load for faculty with extensive research/clinical responsibilitiesmight be two, three-credit courses per semester (10 to 15 weeks). These might be one graduate course and one

4-14

undergraduate or two undergraduate courses. This would be about six contact hours per week. Therefore in a fullPBL program, the expectation would be to lead one or two tutorial groups with three to six contact hours per week.

In a conventional program, the typical teaching load for faulty with no research or clinical responsibilities might befive, three-credit courses per semester or fifteen contact hours per week. Therefore, in a full PBL program theexpectation would be four or five tutorial groups.

For example, in our Chemical Engineering Department about 12 full time equivalent faculty in a research intensiveprogram would he available to handle our usual classes of 45, 40 and 30 students in our second, third and fourthyears. For groups of five or six students, ten faculty would be needed to handle the undergraduate load.

As another example, with 100 students in each of three years would need about 30 research/clinical faculty for theundergraduate program; or 4 “teaching” faculty and 20 research/clinical faculty or, in the extreme, 12 “teachingfaculty and no research/clinical faculty.

(Tutor training will be discussed in Steps 18 and 21 with an example outline given in Table 19.1. We will look at thedetails of tutors, tutor training (and even tutorless groups) later.)

At this stage we have confirmed that we either can work with tutorless groups (because our faculty resources areinsufficient) or tutored groups.

Now consider the weekly load for the students (and for the faculty) in a full PBL program. Consider this in Step 16. ___________________________________________________________________________________________Step 16. Exploring the weekly schedule in a full PBL program.

To make the discussion concrete, consider two options A and B.

In Option A, the students have 4 to 6 hours of contact per week that are typically spread out as a 2 h “Goals”meeting early in the week (to be introduced to the case, identify learning needs, contract with each other on topics)followed later in the week by a 2 to 3 h “Teach” meeting. Between the Goals and Teach meetings the students havetwo to three days to research, learn and prepare to teach. With this approach, the learning objectives per case wouldbe modest. The group will handle about eight different cases in 10 to 12 weeks. Several cases may lead to sufficientlearning goals that student groups will elect to use the same case for several cycles of goals-teach.

In Option A, the “teaching” tutor who handles five groups a week has 25 to 30 h of contact per week and theschedule might be as shown in Table 16.1.

Table 16.1 Option A with a “teaching” tutor handling five groups a week; the “research/clinical” tutor is shown initalics

Monday Tuesday Wednesday

Thursday Friday Saturday Sunday

am Teach 1 Teach 4 Goals 1 Goals 4

pm Teach 2 Teach5 Goals 2 Goals 5

evening Teach 3 Goals 3

In Option B, the students have 2 to 3 hours of contact per week with “Goals” meetings alternating with the “Teach”meeting. Between the Goals and Teach meetings the students have a week to research, learn and prepare to teach.With this approach, learning objectives per case will be more ambitious. The group will handle three to five differentcases in 10 to 12 weeks. Again, several cases may be used for several cycles of Goals-teach.

In Option B, the “teaching” tutor who handles five groups a week has 10 to 15 h of contact per week; the“research/clinical” tutor handles two groups a week with 2 to 3 hours of contact per week. The schedule might be asshown in Table 16.2.

4-15

Table 16.2 Option B schedule for a “teaching” tutor; for a “research tutor in italics

Monday Tuesday Wednesday

Thursday Friday Saturday Sunday

am group #1 group #2 group #4

pm group #3 group #5

evening

Option B with an in-depth consideration of fewer cases may be more attractive later in the program when studentsare familiar with the approach. Option A may be more approach early in the program when students need moreprompting, guidance and frequent feedback.

From a faculty viewpoint, option B certainly requires less contact time per week and more closely resembles thecontact time in conventional programs.

The results of this analysis help us decide on the scope of the problems to be created to match the learningobjectives. Please go to Step 17. ____________________________________________________________________________________________Step 17: Clustering the learning objectives and developing the problems for a full PBl program

In Step 14, we identified systems and suitable clusters of essentials goals for a block of 10 to 15 weeks. Based on thedecisions made in Step 16, we may have selected - 8 cases with a ½ week turnaround time between Goals and Teach or- 3 to 4 cases with a one-week turnaround time between Goals and Teach.

Here is a simplified illustration of how we might proceed.

M 17.1 Create the learning objectives for each problem Consider the use to Table 14-1 in the context of Chemical Engineering Phase 1, problem 1. For example, in option“x” the cluster of learning objectives might be: - the organic chemistry for “ethylene”, chemical thermal and environmental characteristics; typical reactions andproducts that can be made from ethylene and the raw materials and reaction routes for its production and - how to read financial statements.

Alternatively, for option” y”, instead of learning how to read financial statements, our learning goals might be tolearn elementary statistics, sampling, fate of chemicals in the environment, environmental legislation, and massbalances. M 17.2 Create the problem case scenario.

A common mistake made by persons just starting PBL is to focus first on collecting or creating the problems. We donot start with the “problems” (whether we create them, get them over the internet or purchase them from otherprograms). We create the problems to satisfy the learning objectives to be achieved. Step 18.1 must be completedbefore we create the problems.

Our initial attempt to create a problem to achieve the learning objectives for case “x” might be:Esso plans to expand their ethylene plant. Some argue that this expansion will have a detrimental effect ontheir profit. You have been asked if you wish to invest in a stock purchase plan offered by the company.

4-16

For option “y” a problem statement might be:Esso plans to expand their ethylene plant. Environmentalists are protesting this expansion. This expansionwill have a detrimental effect on the environment. As a representative of the company you are to go to apublic meeting addressing these issues.

These are possible first attempts. Some criteria for a “good problem” are (Dolmans et al., 1997, Alverno College,1977) and Drummond-Young and Mohide, 2000):

1. the learning outcomes expected by the teacher are identified correctly by the students; the scenariocontains “cues” that will trigger the desired search for learning objectives;2. the learning outcomes are consistent with the stage of development and builds on and activates priorknowledge;3. an appropriate level of complexity is included;4. the scenario requires integration of knowledge, skills and attitudes across subjects;5. the scenario allows an openness;6. the scenario is motivational and relevant; the scenario is similar to one we might encounter inprofessional practice; 7. the scenario promotes student activity; 8. the scenario identifies the context, gives a concrete scenario and clearly identifies the expected task without spelling out specifics.

We use these criteria to polish, expand, alter the problem statement.

M 17.3 Create the resourcesBased on the learning objectives, we can then identify, and make available, pertinent learning resources, laboratoryequipment, resource experts. If we have limited laboratory equipment for the number of student groups, then we cangive different groups different problems so that the laboratory equipment can be available for the students.

Please go to Step 20.

___________________________________________________________________________________________Step 18: Identifying the section, setting goals for the problems, creating the problems and the resources in thecontext of a single course of small group, self directed, self assessed PBL in an otherwise conventionalcurriculum.

M 18.1 Identify the course or part of the course

Consider, perhaps a three-week section of the course. Choosing less than three weeks provides insufficient time forstudents to adjust to and appreciate the new learning environment. Give your pilot program a chance to succeed!

Explore how we might overcome limitation 13.1 (limited opportunity to integrate across subjects). Is there anylaboratory work, tutorial or project work that can be included to enrich the subject knowledge? Although we arelimited in how extensively we can integrate the subject knowledge across the case, we can still try to bring in issuesother than those we might traditionally have considered. For example, if we selected a three-week section on heat exchanger characteristics from a traditional course of heatexchange, then we might want to:

- ensure that the knowledge is used in a context typical of what a professional encounters in practice. The“characteristics” of heat exchangers might be replaced by “rate” a heat exchanger, “select” a heatexchanger, “trouble shoot” unexpected behaviour on a heat exchanger.

- broaden the viewpoint to include operability of heat exchangers, thermal integration of systems, energyreduction, safety, ethics, engineering economics related to, operation of heat exchangers integrated withreactors, integrated with distillation columns and so on.

M 18.2 Identify the learning objectives

We have already published learning objectives and criteria for our traditional course. Reconsider these in the light ofactions taken in Step 18.1. Cluster these into sets of objectives that can be achieved in about one week of PBL.

4-17

For example, for PBL activities in one course in an otherwise traditional curriculum, we need to work with shortcases that can be achieved in a one week turn around. The first time I tried this I created a long case problem thatwould take three weeks to address. The students didn’t do anything on it until the last week and then complainedthey ran out of time; the case was too big! Typically I now use one hour on Thursday for the “Goal setting” meeting.The students read the case, identify what they need to know to solve the problem, create learning objectives andcontract with each other on the topics they will learn and teach. The “Teach” meeting is on Monday or Tues, for oneto two hour. This means that each student has only from Thursday to Tuesday to research, learn and prepare to teachthe knowledge. Considering the other commitments the students have to the other courses they take concurrently in aconventional program, I can expect them to do about three to five hours of “homework” per week. There will befour to five other people working on other topics. Since each problem has about five or six learning objectives, onestudent is expected to achieve one learning objective. Hence, cluster learning objectives in packages of about six toseven for each week of PBL activity.

M 18.3 Create the problem case scenario

Some criteria for a “good problem” are (Dolmans et al., 1997, Alverno College, 1977) and Drummond-Young andMohide, 2000):

1. the learning outcomes expected by the teacher are identified correctly by the students; the scenariocontains “cues” that will trigger the desired search for learning objectives;2. the learning outcomes are consistent with the stage of development and builds on and activates priorknowledge;3. an appropriate level of complexity is included;4. the scenario requires integration of knowledge, skills and attitudes across subjects;5. the scenario allows an openness;6. the scenario is motivational and relevant; the scenario is similar to one we might encounter inprofessional practice; 7. the scenario promotes student activity; 8. the scenario identifies the context, gives a concrete scenario and clearly identifies the expected task without spelling out specifics

We usually have to relax criterion #4 “the scenario requires integration of knowledge, skills and attitudes acrosssubjects” unless we have been able to overcome this limitation in step 18.1.

M 18.4 Create the resources

This may be your published critique of the major texts, copies of key articles made available in the reference sectionof the library and copies of your transparencies or internet materials you used when you taught the course in atraditional format.

Now consider how best to handle the tutorial task.

Please go to Step 19.____________________________________________________________________________________________Step 19: Tutors for PBL in a single course in an otherwise conventional curriculum.

As pointed out in 13.2 since we have one instructor with classes of size > 5 students we either work Socraticallywith many groups of five students or we work with tutorless groups.

19.1 Work Socratically. Assign students to groups of five. Lead all the groups in one large room through the processby alternately assigning a task, gathering the results, responding to the results and reassigning a task. This is stillsmall group, self-directed, self assessed PBL but the tutor is shared with all groups. A description of this approachis given in “Problem based Learning: Resources to gain the most from PBL,” p A-19 to A-23. The challenge withthis approach is monitoring the self-assessment. Some ideas on how this might be done will be given in Step 25.

19.2 Use tutorless groups. (This will be discussed in Step 24.)

4-18

We will look at the details of working with tutorless groups later. Now we need to consider the development of theprocess skills.Please go to Step 20. ____________________________________________________________________________________________Step 20: For both partial and full PBL programs, select what process skills are needed for small group, selfdirected, self assessed PBL

In Steps 2 and 3 we discussed skill development as dictated by the desired outcomes fo the program. However, for small group, self directed, self assessed PBL our needs for process skills are clear. Table 20-1 summarizes theskills, an idealized approach to development and my estimate of the minimum.

Table 20-1 Process skills for small group, self directed, self assessed PBL

For skill needed idealized number of workshops minimum commitment

smallgroup,

interpersonal skillgroup skillconflict resolutionassertivenesschairperson

MPS 52 ( 2 h)MPS 28 (4 h)MPS 45 (3 h)MPS 47 (2 h)MPS 29 (15 h)

Modified MPS 52 (30 to 50 min)p. B-24 ff. ResourcesModified MPS 28 (60 to 90 min)p. B-30 ff. Resources

selfdirected,

self confidencegoal setting(self assessment)teach skillcritical thinking

MPS 1, (2 h) MPS 11 (3 h)MPS 31 (3 h)MPS 3 (6 h)*MPS 36 (3 h), MPS 12 (4 h)MPS 30 (4 h)

Modified MPS 36 (90 min)p B-38 ff. Resources

selfassessed

self assessment MPS 3 (6 h)* Modified MPS 3 (70 to 90 min)p B-48 ff. Resources

PBL problem solving:strategies, awareness,creativityelaboration

MPS 1 (2 h)MPS 4 (4 h)MPS 7 (8 h)MPS 14 (2 h)

Modified MPS 4: (90 min)p B-16 ff, Resources and p. 3-23 activity 3.2 in HTGTM

The MPS program is described inhttp://www.chemeng.mcmaster.ca/innov1.htm and clickon MPS.

The goals, transparencies and timingsheets for the modified workshops aregiven in “Problem Based Learning:Resources to gain the most from PBL”

How to use the data in Table 20-1 depends on whether we work with full PBL programs (discussed first) or withPBL as a course in an otherwise conventional program (discussed second).

M 20.1 For full small group, self directed, self assessed PBL programs:

I recommend that tutors receive the full training as outlined in column 3. Tutors usually possess these skillsintuitively but most lack confidence in how to coach others and how to assess the skill of the students. This remindsme of a call I received. “We have failed a student based on her “poor” problem solving skill. We know she lacks theskills but we are unsure as to how to provide feedback to her about her deficiencies.” Once tutors have receivedthese workshops (or even mentally worked their way through the workshops) tutors know what the skill is, how todescribe, how to assess the skill and how to give feedback to students about their skill.

In addition tutors should gain confidence in how to prompt students. This is discussed in Chapter 2 in Helping,“Problem based Learning: helping your students gain the most from PBL” that can be downloaded free from http://www.chemeng.mcmaster.ca/innov1.htm and click on PBL.

4-19

For students in full PBL programs, I recommend that the students receive at least the minimum skill developmentprogram listed in column 4. Actually, I prefer to give students as much training as possible from column 3.

For tutored groups, that are possible in full PBL programs, the tutor can play a vital role in the skill development.Indeed, some options used in different programs include:

a) Train the tutor and depend on the tutor to supply the process skills. Do not expect skill acquisition by the students.This is contrary to Step 1 and I do not recommend this approach.

b) Train the tutor and trust that through feedback and modeling the students will acquire the process skill. I believethat more structure is needed to develop the skill and confidence in the students.

c) Train the tutor, use HTGTM or equivalent as a required text and work through the text.

d) Train the tutor and recommend HTGTM or equivalent as a resource.

Although a to d are options that have been used, my bias is that students should explicitly experience the minimumskill development by workshops, as outlined in column 4 of Table 20.1.

Please go to Step 21 to address the question of when to develop these skills for students in a full PBL program. ________________M 20.2 For small group, self directed, self assessed PBL in one course in an otherwise conventional program

The teacher/tutor should receive the full training described in column 3 of Table 20.1.

The students should receive the minimum in column 4 and as much of column 3 as possible. Empowering thestudents with these process skills in very important because the student groups will be working autonomouslywithout a tutor. They cannot count on a tutor to help them resolve conflict, create learning goals, have an effectivegroup, systematically solve the problem; they should have the skills to do this themselves.

Please go to step 22 to address the question of when to develop these skills for students in a single courses in anotherwise conventional program.____________________________________________________________________________________________Step 21: For a full PBL program, select when to address the process skills and getting students ready for smallgroup, self directed, self assessed PBL. Since all students in the program need “process skills” for PBL, we can formally prepare students as part of theprogram goals and outcomes. Here are some options:

21.1 Pre-screen students and admit only those who can demonstrate skill. This approach is taken by McMasterMedical School. The pre-screening is done with applicants working on a problem in a simulated tutorial. Threetrained observers rate the problem solving and group skill of each of the seven applicants being observed.

Table 21-1 Suggested outline for a week-long, pre-PBL workshop (from Helping, p. 6-5)

Time Topic Comments and resources

4-20

Day 1: am 20 min Overview to process skills; importance inlife and for PBL;

List names of skills: problem solving, group process,interpersonal skills, self-directed learning, shift fromdependent to independent to interdependent learning, self-assessment; rationale for why skills are important, thechallenge in developing them; Resource: Section 3.4 of thisbook. MPS unit 2 and "The MPS Strategy Book".

Day 1: am 3 h rest ofmorning

Awareness of "what goes on in our mind" Workshop using Talk Aloud Pairs Problem Solving;sometimes may be omitted and participants go directly to thetalk-aloud strategy workshop. Otherwise, this is a prerequisiteactivity for the Strategy workshop. Resource: MPS Unit 1 and"the MPS Strategy Book". Resources Section C.3.

Day 1: pm 4 h Strategy workshop Workshop using Talk Aloud Pairs Problem Solving with theproblem solver moving a marker to show the "stage' that theyare addressing. An extremely simple and powerful workshop.Resource: Exercise 3.2 in HTGTM. MPS Unit 4 and "theMPS Strategy Book". Resources Section B.3.

Day 2: am and pm 7½ h Creativity workshop Helps students brainstorm issues, create hypotheses. Oftenhandled in other programs through a 10 to 30 min introductionto the principles of brainstorming. We have found this to beinsufficient to actually develop confidence and personal skills.Resource: mentioned in Chapter 3 in HTGTM. MPS unit 7and the "MPS Idea Book".

Day 3: am 3½ h Change and stress management Addresses one of the most important attitudinal issues in PBL.Can be tied to topics of "time management". Can be handled astopic by itself or intimately connected to the short-term goal ofPBL. Resource: contact your local mental health associationand ask them to facilitate the workshop. Or use Chapter 1,HTGTM; Others resources include MPS 5 and 17 on timemanagement.

Day 3: pm 4 h Criteria and decision-making Needed for assessment, for goal setting, for defining problemsand for any evaluation or judgement. Resource: Chapters 3and 9, HTGTM. MPS 23 and 24 and "The MPS Idea Book".

Day 4: am 3½ h Self-assessment Even if the tutor will do much of the assessment, helping thestudents understand the issues of assessment is vital. This alsois a nice extension of the previous topics of criteria anddecision-making. Resource: Chapter 9, HTGTM. MPS 3 andChapter 5 in "The MPS Idea Book".

Day 4: pm 4 h Group skills Important set of workshops in which one includes as muchexperience as the students have time for. You might wish tostart with activities on interpersonal skills and listening.Although you will not be able to move the group all the way toteam behaviour in the limited amount of time, you will be ableto plant seeds that will guide future behaviour. Resource:Chapter 5. In particular Exercise 5.5, HTGTM. MPS 27, 29,45 and 52.

Day 5: am 3½ h Group skills, chairperson skills

Day 5: pm 3 h Teams and team building

Day 5: pm 1 h Wrap-up, summary

21.2 Develop the skills in explicit skill development workshops before the first semester begins. An example outlineof such a training program is given in Table 21-1. This training can be for tutors and we recommend that it be used

4-21

for all students in the program. (The Nursing program at McMaster University uses a half-day workshop on changemanagement, stress management, understanding PBL and problem solving strategies on the first day for all students.An outline of these workshops is given in “Problem-based Learning: Resources to gain the most from PBL,”Chapter B.)

21.3 Develop of the student’s skills concurrently with the first PBL cases but the emphasis is on “process skill”development. Kozu (2000) describes a PBL program in Health Sciences in Japan where the emphasis at the first isabout 80% on process skill development. This tapers to about 20% emphasis on process skill development near theend of the program.

21.4 Develop the skills of the students concurrently but with equal emphasis on process skills and the knowledgebeing learned. This was the approach taken in McMaster University’s Theme School program on “New Materialsand Their Impact on Society” (Woods, Hrymak and Wright, 2000). This is illustrated in Table 21-2 and 21-3. Twoversions are given that depend on the extent of skill development selected.

Please go to Step 23 if in Step 15 we have sufficient faculty for a tutor for every group.

Otherwise with tutorless groups please go to Step 24.____________________________________________________________________________________________Step 22: For an individual course, select when to address the process skills and getting students ready for smallgroup, self directed, self assessed PBL. This also applies for Socratically facilitated PBL (discussed in Step 19.1).

With a single course, we are restricted in how much time is available to develop the process skills and get thestudents ready (as discussed in Steps 8 and 9). The options are:

22.1 Use some tutorials ahead of time to get the student’s attitude ready (as discussed in Steps 8 and 9) and todevelop problem solving and group skills (modified MPS 28 and modified MPS 4 listed in Table 20-1, column 4).

22.2 Build the skills concurrently. Blend the skill development with the PBL activity as outlined in Table 21-2 or21-3. In this approach we start a PBL case problem in the subject domain and expect the students to have a Goalsmeeting followed by a Teach meeting. While this is happening, we take some of the in-class time and run a processskill development activity. Each PBL unit takes about a week. The version shown in Table 21-2 takes seven weeksof elapsed time. The version shown in Table 21-3 is less ambitious and takes four weeks of elapsed time.

Please go to Step 24 to see how to handle tutorless groups.____________________________________________________________________________________________Step 23: Being a tutor in a group.

Having a tutor in each group of five to six students is usually only possible for a full PBL program where there aresufficient faculty resources.

Small group, self-directed, self assessed PBL requires skill in interpersonal communication, problem solving, teamwork, conflict resolution, stress management, change management, critical thinking, elaboration, assessment andteaching. Tutor training in the process skills has been discussed already in Step 20.1.

Tutors should gain confidence in how to prompt students. This is discussed in Chapter 2 in Helping, “Problem basedLearning: helping your students gain the most from PBL” that can be downloaded free from http://www.chemeng.mcmaster.ca/innov1.htm and click on PBL.

The main issues to be addressed in tutored groups tend to be (Woods, Hall, Eyles, Hrymak and Duncan-Hewitt,1996):

Table 21-2 Building up the process skills concurrently: long version

4-22

Generalfocus

Skills and attitude addressed with MPS unit

attitude problem solving group skills selfassessment

self directedlearning

PBL1

adjusting Perry, change,MPS 49

PBL2

awareness of processes

strategyworkshops, MPS4

interpersonalMPS 52;groups MPS28

importanceof feedback

PBL3

goal setting how to define.MPS 31

importanceof goals.MPS 3

learninggoals. MPS36

PBL4

self-talk anger, stress, timemanagementMPS 17, 5

creativity. MPS 7 anger, MPS52

PBL5

criteria decision makingMPS 24

criteria MPS3

criteria MPS23

PBL6

self-management

uniqueness, MPS11; timemanagement,MPS 17

exploring issues.MPS 15

cope withconflict.MPS 45

PBL7

knowledgestructure

decision makingby consensus

chair MPS 29 teach vs talk;cues

Table 21-3 Building up the process skills concurrently: short version

Generaltheme

Skill and attitude development with Resource material

attitude problem solving group selfassessment

self directedlearning

PBL1

adjustment mini MPS 49,Perry and what isPBL?

PBL2

problemsolving

mini MPS 4

PBL3

group work mini MPS 28

PBL4

self directedlearning

mini MPS 36

- breadth versus depth. helping the groups select just the right amount of depth to be learned. - addressing emotional issues brought to the problem from personal experience that tends to cloud the search for the

4-23

real issues. - the quality of the critical reasoning: helping the students to develop these skills- dominance/passiveness: being able to create an environment where all students are comfortable sharing andcontributing.- astutely drawing on the background experience that students bring to a case but not allowing one with a strongbackground to dominate.- encouraging the required reflection and elaboration after the problem ha been “solved” - getting students to self assess based on evidence rather than try to negotiate.- lack of closure: - dealing with overly negative or difficult behaviour- skill in using the resources. Some suggestions are given by Woods et al. (1996) about how to cope with these. Please go to Step 25.____________________________________________________________________________________________Step 24: Working with tutorless groups.

Our experience has been that tutorless groups function exceptionally well. The main issues to be addressed in tutorless groups tend to be (Woods, Hall, Eyles, Hrymak and Duncan-Hewitt,1996):- apparent conflict because all members are not seen as pulling their fair share of the work (see also Woods, 2001e)- attendance- trust and reliability- different work goals and standards among different students that leads to conflict.- the amount of reflection, monitoring and writing they must do in self assessment and accountability.Some suggestions are given by Woods et al. (1996) about how to cope with these.Please go to Step 25.___________________________________________________________________________________________Step 25: Making it all work: assessment and accountability.

Whether working in tutored groups or tutorless groups, we are empowering students with the tasks of settingstandards, assessing progress, developing trust and working effectively together.

Our experience has been that by underpinning the whole process with the principles of assessment makes PBLeffective, efficient and hassle-free. Those principles are:

Principle 1: Assessment is a judgement based on performance - not personalities. We need to help a student realizethat a poor mark does not mean he/she is a bad person. The judgement is made about performance in completing atask. It has nothing to do with his/her value as an individual. This is an issue, especially for students with attitudescharacterized by Perry’s level 2. More details about Perry’s levels and their implications to teaching and learning aregiven elsewhere. (see How to gain the most from PBL, HTGTM, Chapter 1, Perry, 1970, Felder et al., 2000 andWoods et al., 2000c.)

Principle 2: Assessment is a judgement based on evidence - not feelings. We might intuitively feel that a student isskilled at team work. However, we need to replace that intuitive feeling with physical written evidence.

Principle 3: Assessment should be done for a purpose with clearly-defined performance conditions.

Principle 4: Assessment is a judgement done in the context of published goals, measurable criteria and pertinent,agreed-upon forms of evidence.

Principle 5: Assessment should be based on multidimensional evidence: static and dynamic situations; smallassignments and lengthy projects; academic, social and personal contexts; under a variety of performance conditions(exams and homework, written and oral, performance as an individual and as a member of a group), formative andsummative data and with different persons being the assessors (self, peer, teacher and trained external observers).

4-24

More about how to use assessment to aid small group, self directed, self assessed PBL is given in:- Chapter 5, of “Helping your students gain the most from PBL” that can be downloaded free from http://www.chemeng.mcmaster.ca/innov1.htm and click on PBL.

- Chapters 8 and 9 of HTGTM - Woods “Assessing lifelong learning skills,” Chem Eng. Education (2001a).

Please go to Step 26.____________________________________________________________________________________________

Step 26: Looking after the nitty gritty for both tutored and tutorless groups.

You’re almost ready to go. Consider the issues in Chapter 4, “Issues in Setting up Small group, self-directed, selfassessed PBL” in Helping your students gain the most from PBL” downloadable free fromhttp://www.chemeng.mcmaster.ca/innov1.htm and click on PBL.

ENJOY!

Acknowledgments:

Thanks to Professors Michele Drummond-Young and Barbara Love, School of Nursing, McMaster University,Hamilton, Canada and Professor Greg Riach, Faculty of Business, Seneca College of Applied Arts and Technology,North York, Canada for their comments and suggestions.

********************************************************************************************References:

Alverno College (1977) “Faculty Handbook on Learning and Assessment,” Alverno College, 3401 South 39th St. Milwaukee WI.53215-3922.

Branda, Luis, (1993-95) “A Problem-based Evaluation Exercise, PBEE,” in Chapter F, of “ Problem-based Learning: Resourcesto gain the most from PBL,” (1999) D.R. Woods, 2nd printing of 2nd edition, Woods, distributed by McMaster UniversityBookstore, see http://www.chemeng.mcmaster.ca/innov1.htm and click on PBL. Chickering, A.W., and Z.F. Gamson (1987) “Seven Principles of Good Practice in Undergraduate Education,” AAHE Bulletin,March 3-7. Coles, C.R. (1985) “Differences between conventional and problem-based curricula in students’ approaches to studying,”Medical Education, 19, 308 to 309

Dolmans, D.H., et al. (1997) “Seven Principles of Effective Case Design for PBL,” Medical Teacher, 19, 185-189.

Drummond-Young, Michele and E. A. Mohide (2000) “Developing problems for use in PBL: from concepts to application,” Chapter 8 in “Transforming nursing education through problem-based learning,” E. Rideout, ed., Jones and Bartlett, Sudbury,MA.

Felder, R.M et al. (2000) “The Future of Engineering Education: Part 2: teaching methods that work,” Chem. Eng. Ed., 34 (1), 26- 39 and downloadable from the wwwhttp://www2.ncsu.edu/effective_teaching/

Hamilton, J.D. (1976) “The McMaster Curriculum: a critique,” British Medical Education, May 15, 1191- 1196.

Kozu, T., Tokyo Women’s Medical University, Japan, (2000) Personal communication at the 2nd Asia-Pacific Conference onPBL, Singapore, Dec, 4 to 7.

Norman, G.R., (1988) “Problem solving skills, solving problems and Problem-based Learning,” Medical Education, 22, 279-286

4-25

Norman, G.R. and H. Schmidt (1993) “Where is the Learning in PBL?” PEDAGOGUE, 4, 2, summer, a Newsletter published bythe Program for Educational Development, Faculty Health Sciences, McMaster University, Hamilton

Perry, W.G., Jr. (1970) " Forms of intellectual and ethical development in the college years," Holt, Rinehart and Winston, 1970.This is a measure of the student's (and teacher's) attitudes toward the learning process. Level 2 sees the student's role toregurgitate; the teacher's role to tell; At Level 5 the student's role is to assess which information is pertinent and the teacher's roleas a resource. Research-quality instruments measure this level by asking participants to describe their most exciting learningexperience. These scripts are scored by trained professionals. Simpler, self-scored questionnaires are available. They show goodinterconsistency. Two such examples (with scoring) are given in Woods [1997].. This can be used to probe instructor's attitudeabout his/her role and to design/select learning environments that promote the higher level attitudes. A one-on-one agreementexists between the Perry level attitudes and student's preferred tasks as measured by Bloom's taxonomy. (see Woods , J. CollegeScience Teaching, 22, Feb., 250-251, 1990.) Related measures have been developed in the Reflective Judgement model byKitchener and King with added dimensions of critical thinking skill development/ problem solving skill. (J. AppliedDevelopmental Psychology, 2, 89-116, 1981.)

Ramsden, P. (1983) “The Lancaster Approaches to Studying and Course Perceptions Questionnaire,” Educational Methods Unit,Oxford Polytechnic, Oxford, UK.

Ramsden, P., (1985) "How Academic Departments Influence Student Learning," HERDSA News, 4, 3-5,Ramsden, P., and N.J. Entwistle, (1981) "Effects of Academic Departments on Student's Approaches to Studying," Brit. J. ofEducational Psychology, 51, 368-383. The emphasis here is on promoting deep learning (or "achieving" learning where thestudent applies deep or rote learning depending on the situation). The research identifies 8 elements in the learningenvironment that enhance student's efforts at deep learning. The six factors that positively promote deep learning are: goodteaching, openness to students, student's freedom in selecting learning, use of clear goals, standards and assessment,vocational relevance and the positive social climate. The two factors that negatively promote deep learning are heavyworkload and high dependence on formal lecturing and teacher dominated delivery. Instruments are available to measure thestudent's preference for learning, LASQ, and the learning environment, Course perceptions questionnaire. (Woods andcolleagues are gathering benchmarking data for engineering.) Similar findings and slightly differentquestionnaires/instruments have been developed by Marton and Saljo (On the qualitative differences in Learning," British J.of Educational Psychology, 46, 4-11, 1976) and Biggs, J.B., "Student Motivation and Study Strategies in University andCAE Populations," Higher Education Research and Development, 1, 33-55, 1982.. Although deep learning is related to the higher levels of Bloom's taxonomy (because unless one is skilled in comprehendingthe context, analysis, synthesis and evaluation will not be possible) the concept of deep learning includes other dimensionsnot included in the taxonomy.

Regan-Smith, M.G., et al. (1994) “Rote Learning in Medical School,” JAMMA, 272, 17 1380 to 1381. Spencer-Chapman, Nick (1996) “The Rough Guide to Problem-based Learning in Engineering,” Oxford Brookes University,Headington, Oxford OX3 OBP, UK

Woods, D.R., R.R. Marshall and A.N. Hrymak (1988) “Self-assessment in the Context of the McMaster Problem SolvingProgram,” Evaluation and Assessment in Higher Education, 12, 2, 107-127.

Woods, D.R. (1993a) Guest Editorial On the Learning in PBL,” PEDAGOGUE, 4, 2, summer, a Newsletter published by theProgram for Educational Development, Faculty Health Sciences, McMaster University, Hamilton Woods, D.R. (1993b) “Problem solving- what doesn’t seem to work,” J. College Sci Teaching, 23, 157-158.

Woods, D.R. , F.L. Hall, C.H. Eyles, A.N. Hrymak and W.C. Duncan-Hewitt, (1996) “Tutored versus Tutorless Groups inProblem-based Learning,” American J. of Pharmaceutical Education, 60, 231 - 238.

Woods, D.R. (1997) “Problem-based Learning: how to gain the most from PBL” 2nd edition, Woods publishing, Waterdown, ONand distributed by McMaster University Bookstore. HTGTM, Translated into Japanese by S. Shindo and published by Igaku-Shoin, Ltd., Tokyo (2000)

Woods, D.R., et al. (1997) “Developing Problem Solving Skill: the McMaster Problem Solving Program,” J EngineeringEducation, April, 75-91 (1997) http://www.chemeng.mcmaster.ca/innov1.htm and click on MPS for a summary of theresearch findings and some details for many of the units.

Woods, D.R., (1997b) "Problem-based Learning: helping your students gain the most from PBL," 2nd ed., Woods,

4-26

Waterdown, distributed by McMaster University, Hamilton, ON, Helping. and downloadable from http://www.chemeng.mcmaster.ca/innov1.htm and click on PBL and download.

Woods, D.R. (1998) “The Case of the Letter to the Dean,” Plenary presentation at the ASEE Summer School for ChemicalEngineering Faculty, Snowbird, Utah

Woods, D.R. (1999) “Problem based learning: resources to gain the most from PBL,” Woods, Waterdown (1999) anddownloadable from http://www.chemeng.mcmaster.ca/innov1.htm and click on PBL and some Chapters are download from theresources book. Resources

Woods, D.R. (2000a) “Team Building: how to develop and evaluate individual effectiveness in teams,” Proceedings, AIChEAnnual Meeting, Los Angeles, Nov 2000, http://www.aiche.org

Woods, D.R. (2000b) “Ideas to Improve Learning,” book manuscript, McMaster University, Hamilton ON, 270 pp.

Woods, D.R., R.M. Felder, A. Rugarcia, and J.E. Stice (2000c) “The Future of Engineering Education: Part 3. developingcritical skills,” Chem. Eng. Ed., 34 (2), 108-117 and downloadable from the wwwhttp://www2.ncsu.edu/effective_teaching/

Woods, D.R. (2000d) “Helping your students gain the most from PBL: the Case of Linda’s Complaint,” in the Proceedings ofthe 2nd Asia-Pacific Conference on PBL, Singapore, Dec 5 to 7, p. 12 - 36.

Woods, D.R., A.N. Hrymak and H.M. Wright (2000e) “Approaches to Learning and Learning Environments in PBL versuslecture-based learning,” Proceedings, ASEE Conference, June St. Louis, MO., session 2213.

Woods, D.R. (2001a) “Assessing Lifelong learning skills,” Chem. Engng. Educ. submitted.

Woods, D.R., P.E. Wood, H. Sheardown, T. Kourti and C. M. Crowe (2001b) “Assessing problem solving skills,” Chem. Engng.Educ., paper accepted

Woods, D.R., S Taylor and S. Jaffer (2001c) “Assessing team skills. Part I: successful teams” Chem. Engng. Educ.

Woods, D.R., S Taylor and S. Jaffer (2001d) “Assessing team skills. Part II: individual contribution to teams,” Chem. Engng.Educ.

Woods, D.R. (2001e) The Case of “They just don’t pull their weight”, in “Case Studies in PBL,” P. Schwartz and S.Mennin, ed., Kogan Page, UK.

4-27

PBL: when? who? where? what? how? c:\pbl Donald R. Woods, copyright 2001

1.Overall purpose of programKnowledge? Skills? Attitudes?

2. How: general4. Aid future recall 3. How: specifics5. Improve learning problem solve, team6. Deep learn lifelong learning

7. Use PBL

8. Students ready? 10. Teachers ready?9. Students understand PBL

11. Which form of PBL?

12. Full program 13. One course14. General goals15. Enough tutors? role of tutor16. Schedules: students & tutors17. Goals, specific 18. Goals, specific

19. Tutor’s role

20. Prep. process skills: what? how?

21. When? 22. When?23. Tutor details 24. Tutor details (tutorless)

25. Assessment basis26. Nitty gritty

5-1

5. Implementing PBL copyright © 2002 Donald R. Woods Donald R. WoodsMcMaster University, Hamilton, Canada

The learning experience must be designed to take advantage of the strengths of PBL: active, empowering yetaccountable, cooperative, just-in-time learning, positive environment expecting success, prompt feedback,developing of maturity and confidence and developing process skills (especially lifelong learning). You will have adifferent role as monitor and mentor (guide on the side) with your major effort being spent creating the learningenvironment to provide structure, structure, structure without interfering with the process.

So how to start?Step 1. For the full program: list the knowledge, attitudes and skills possessed by your graduates. Example, “skill inlifelong learning, team work, self assessment..” Publish these in the calendar and in the course outline, syllabus foreach individual course.For an individual course, list the knowledge, attitudes and skills possessed by the graduates of your course.

Step 2. For the full program, agree on the definitions, learning objectives and forms of evidence for each processskill. Use the same criteria and form throughout the whole program. Example, “The form to assess oral and writtencommunication skills for all activities in our program is....” For an individual course, don’t reinvent the wheel. Select published data. For example, see Chapter 3 of Woods,“PBL: helping your students”, Chapter D of Woods’ “PBL: Resources..” andhttp://www.chemeng.mcmaster.ca/innov1.htm and click on MPS for the process/professional skills.

Step 3. For the full program, identify for each course where those skills will be developed initially and usedconsistently throughout the program. Example, “team skills will be developed in Course 950, used in course1015,...” This information is used by the Department to plan the curriculum.For an individual course, tell your colleagues about what you are doing so that they can use the skills your studentsare acquiring. They can use this information to help other instructors with their own courses.

Step 4. Reduce the amount of subject knowledge “covered” in your traditional lecture course by 1/4 to 1/2. Initiallythis may appear to be an impossible task. However, my experience has been that in any course the task is relativelyeasy when we scrutinize the current “coverage” based on the criteria of

1. What fundamental ideas from this course will still be pertinent in the year 2015? Will technology havechanged?

2. What must the students know to take the subsequent courses? In any course I find that there is the core but there also are "the instructor's favorite ideas". Nothing wrong with thelatter but we need to focus on the fundamentals.

[This is easier to do when the Program is considering all the courses in the program. There usually is a lot of overlapand repetition. You can also take 2 credits of stuff from one course and combine it with 1 credit from another. On theother hand, if you are the only one making the change and it is in your course, then you have more constraints... butyou can do it.]The reason we need to “reduce the subject knowledge” is that time must be allowed to develop, and assess theprocess skills and attitudes being developed.

Step 5: Try to arrange for two 75-minute sessions per week instead of the traditional three 50- minute lectures perweek. The University of Delaware has found this to be extremely helpful. One the other hand, at McMaster we wereconstrained to three 50-minute sessions per week and we could make PBL work. Also, we had two 100 minutesessions per week to develop the process skills. Try to have class in a room with a flat floor, moveable chairs andtables (instead of a traditional banked auditorium). If your class size is 60 students, arrange for a room “with seatingcapacity” for 90 to 100.

5-2

Step 6: Create the problems.First the problems must achieve the target “learning goals”1. the learning goals are achievable. For single courses (for example in hybrid or conventional programs) for eachproblems allow about 3 to 5 hours of study for an individual student; each problem would have about 6 to 10objectives for a group of 6 students so that each will research/teach the others.2. the learning outcomes are consistent with the stage of development and builds on and activates prior knowledge.3. goals might integrate knowledge, skills and attitudes across subjects and disciplines.*********************The problems must represent “professional practice” and must contain “cues” that will direct the student to select“your learning objectives for the problem” About the scenario created. (can be a single scenario, or you could build a sequence of scenarios but each wouldexpect the same 3 to 5 hours of student study).

4. the scenario contains “cues” that will trigger the desired search for learning objectives; the learning outcomesexpected by the teacher are identified correctly by the students.5. the scenario includes an appropriate level of complexity. 6. the scenario allows an openness. This challenges the student’s thinking and expects the student to integrate thenew knowledge with the old. 7. the scenario is motivational and relevant. 8. the scenario is similar to one we might encounter in professional practice; (for example, in Engineering this mightinclude rating, debottlenecking, design, trouble shooting, labour relationships, team work, public, monitoringcompliance with legislative regulations).9. promotes student activity.10. any data given should be raw data (like we encounter in practice).11. the scenario identifies the context, gives a concrete scenario and clearly identifies the expected task withoutspelling out specifics.

! Example Case 6 Paul’s decision

Context: Lifelong learning course to first yearstudents at City University of Hong Kong

Target Objectives:Setting prioritiesManaging timeBudgeting timeClearer understanding of the expectations inuniversity for time to studyLong and short term planningProcrastinationLearning to say No!

Paul’s decision (Ed Ko, City University of Hong Kong)Paul has been persuaded by his friends to run for a position onthe Departmental Society. He really would like to, but he isafraid that doing so might take time away from his otheractivities. He is already on the university swimming team andhas to work five hours a week in order to earn some money topay back his credit card loans. With six courses that he is takingthis semester, he feels that he is constantly behind in his work.

Suggested approach:1. Consider working with 4 week’s worth of “learning” or 12 hours of “lecturing”2. For that portion of your course gather the following materials together:

A. the learning objectives (For example, there should be about 20 to 30).B. your lecture notes for those 12 hours.C. the usual homework assignments you give (For example, this might be about 4 assignments [one perweek] with about 4 questions each assignment. These may not be very helpful because most “textbookassignments” give information that is not usually given in professional practice (criterion 8, p 5-2) and wewill ultimately have to create four PBL “problems” whereas you had 16 homework problems).

3. Try to divide the learning objectives, lecture notes, typical problems into four different package. That is becausewe are replacing the four weeks of lectures with four PBL problems to be handled one problem per week. (Or

5-3

alternatively one series of problems that has four parts, eg. Case 9, Tony LoPresti.)

4. Once you have a package representing one week’s worth of work (or one case problem), write down about 6 to 8learning objectives, each of which would require a student to spend about 5 to 8 hours to research, understand andprepare teach notes.

5. For this subject knowledge, think of several professional situations which your graduates might encounter wherethey would need to know this subject knowledge. For each situation, start to frame a brief story. Often it helps if youhave a conversation between two or three people.

6. Set your description of the situation aside and leave it for several days. Then return and reread it. Think carefullyabout the relationship between the description and the learning objectives. Is it going to be very easy for someone tocreate a learning objective from the story, or do we have to add many cues?

7. The problems should:build on knowledge and activate knowledge they know already (criterion 2)should be motivational, interesting, something a student will want to work on (criterion 7)must be representative of what you would encounter after the student graduates (criterion 8)- List yourlearning objectives.

8. Give your results to a student (or colleague) and ask them to brainstorm the learning issues. Usually they willonly be able to identify 60% of your objectives. Explore what additional cues might have helped them retrieve yourobjectives.

Step 7: Visualize the timing and the meetings.Must have at least Two meetings;Meeting #1: Goals meeting: where students interact with the problem, generate learning objectives, get validationfrom the tutor that their learning goals are “correct”, contract with each other about the learning/teaching each willprovide. Example “validation form” attached. see p 3-22

Meeting #2: Teach meeting: when students teach each other, use the information to solve the problem and elaborateabout how and when this new knowledge can and should be used. Example feedback form attached. see p 3-21.

Woods adds a third meeting (after the Teach meeting) Meeting #3, the Feedback or exam meeting when studentspose exam questions to other groups.

If students, and you, do not have confidence that all the students will be teaching the correct stuff, then try a Jigsawmeeting in between the goals and teach meetings, Meeting #1b. Since all groups have the same core objectives,probably one person from each group will have contracted with the group to teach topic A. Bring together membersfrom all the groups that have researched and will teach topic A to consolidate understanding.

Step 8: Organize the student groups. Assign students to groups of 5 to 6. These groups will remain the same for thesemester (10 to 15 weeks depending on your program).

Step 9: Create the resources. Publish a guide to locations of the information to be learned. Alert the Library.

Step 10: Think about options for assessment of knowledge learned.M Student summary of the quality of knowledge learned. M Group solution to the problem. M Individual concept maps of the knowledge. M Individual Test and Exams of the knowledge, TETK. created by teacher, peers, groups, individuals, self.M Individual teach notes and learning contract. M Peer assessment of the quality of the knowledge brought to the teaching task.

5-4

Assessment of process skills: see series of papers in Chemical Engineering Education “Assessing problem solvingskills”, “Assessing team skills”, “Assessing lifelong learning skills.”

Step 11. Develop the process skills see http://www.chemeng.mcmaster.ca/innov1.htm and click on PBL foroverview and click on MPS for details. see also paper PBL: Decisions for Planning and Action: why? when? who?where? what? how? see also the workshops given in Woods “PBL” Resources to gain the most from PBL”.

Step 12. Plan for Success!

Include activities at the beginning to help the students realize why you selected PBL as the learning environment,bring in success stories, help them see the new roles expected of them and the new role you will be playing. Use Perry’s questionnaire to help students identify their current attitude about the learning environment. (see Woods,“PBL: how to gain the most from PBL” Chapt 1 ). Use class ombudspeople to monitor the learning environmentthroughout the year. Be flexible and respond to their concerns. For more see Workshop #5.

Step 13. Add structure for assessment

Students expect us to be fair. Because cooperative group actitivities are used, students usually feel that their groupwill have slackers who do not do their full share of the work. They seek assurance that somehow you will know whatis going on in their automomous group. You can! By the use of assessment forms for about every task the studentdoes you can gather evidence that will help you understand what is happening and that can be used by the studentsas evidence for assessment.

A range of assessment forms have been developed and are given in the book Woods “Problem-based Learning: howto gain the most from PBL” and as handouts from these workshops.

Step 14. Add structure to develop self confidence.

Bandura (1982) studied self confidence, what it is and how you develop it. He found the following differencesbetween self confident persons and those who had little self confidence. Self confident Low self confidencePrepare AvoidFocus on the “tough stuff” Hope it doesn’t happenPersevere Give upManage stress Stress debilitatesFocus first on the familiar Focus first on the “strange”Positive self talk Negative self talk

He found that an environment to develop self confidence provides rewards for performance; highlights the positivesachieved because those with low self-confidence will tend to become despondent and self-devalue. The presence ofhighly competent individuals undermines the effective use of routine skills by others.

To develop self confidence Bandura (1982) recommended enactive mastery via workshops > observe modelsperform the task > emotive strategy to tell yourself you can do it > cognitive strategy. Stress is related to selfconfidence. We have found that helping individuals identify their “uniqueness” is very helpful. Training students toself assess is another activity that increase self confidence. In general, the stages in developing self confidence mightfollow the following pattern: ! Self awareness.! Aware of what others do.! Acceptance of self and others.! Cognitive and attitudinal shifts related to stress management: positive self talk, learn to say NO; worry about whathave control over; sense of being in control; willingness to risk! Attitude about assessment. Receiving ± and responding to feedback. (Perry level 5)

5-5

! Aware of models of behavior: observe, and become aware of the target skills.! Successfully complete achievable goals posed by others with +ve feedback from others (as is done in workshops).! Set achievable goals and self assess.! Self confidenceUse these ideas in planing your PBL learning environment.

Step 15. Include reflection and elaboration.

Elaboration is the process of creating a new perspective and relating new ideas to previous ones. According toSchmidt (1983) elaboration is one of the more effective ways to “learn”. Elaboration includes activating previousknowledge and searching for connections between previous and new knowledge. Elaboration includes determiningthe details of the new knowledge, creating examples, analogies, restatements, conditions, deductions and simplifyingand generalizing. Elaboration includes searching for similar concepts and noting confusing elements, similarities anddifferences.

Reflection is process of thinking about what you have experienced in the past; include knowledge, attitudes,feelings, reactions of what you did and what others did, felt, responded.

Step 16: Consider how to evaluate your program’s effectiveness

The changes you make should make learning easier and more effective for your students. In the design of yourcourse, think about how you might gather evidence to prove this.

**************************Enrichment:1. Please help me understand how best to help you2. MRIQ3. Feedback forms: 3601 student teaching effectiveness in PBL

- Goals identification; - Learning preference table

4. Ten example goals and problem statements (some excellent, some fair, some poor)

References: Woods, “Problem-based Learning: helping your students gain the most from PBL” Chapter 4 downloadable

5-6

M Case 2: Letter to the Dean:

Context: education in PBL

Target learning objectives:

student-teacher mismatch:- learning style MBTI- Piaget: theory vs

concrete- Perry: student vs teacher

expectation

lack of clear goals and criteriateacher attitudestudent's reaction to change lack of mentor for teacher

criteria for tenure:- solely on student evaluations- 3 peer reviews of course outline- teaching dossier- evidence of 1/3 course revision- evidence of scholarship vs

diddlin' around

need to inform others: dean,students, chair, peers.

The letter to the Dean.

You have been five years in a tenure-track position. You have 15 refereedpublications with 4 others in the mill. You have 2 Master’s students and 2PhD students plus a large series of grants to keep your research programviable.

Your teaching has received “fair” ratings from the students in the past. lastyear the Chair suggested that you might devote a little time to improvingyour teaching. Indeed, the Department sent you on two workshops: one oncooperative learning and the other on PBL. These were very motivating.You saw that you could introduce PBL to part of your course 393. Youadded cooperative learning to each tutorial and you took a four week sectionof the course and converted it to PBL. It is now six weeks into the term. TheDean invites you to see him and gives you a copy of the following letter:

___________________________________________________________Oct 15

Dear Dean Habizz

We the undersigned represent 82% of the students in Course 393. Theprofessor is incompetent! The professor is not doing the job. The professorshould be fired. We are paying big bucks to come to this university. Wedemand that you replace this teacher with one who knows how to teach.

H. Andre; A. Sabina; R. Gottz; F. Goamm; R. Jones; K. Sigvaler K.Armstrong; A. Siggs; P. Scripps; B. Carruthers; Z. Gafter; H. Dominik andmany many other signatures

M Case 3: Letter from the Dean:Context: educationTarget learning objectives:

What is PBL?advantages vs disadvantages:time: can I gradually move towardform of PBL for my situation: how many instructors? how many students?mine the only course & all others lecture?What resources do I need?Am I ready? new role for me? students ready? how to helpProcess skills: group skills, problem solving, stressAssessment of studentsEvaluation of the program

Letter from the Dean

Your Director has just returnedfrom a conference on PBL. TheDirector asks you to convertyour course to the PBL format.

! Case 4: Advice to students

5-7

Context: teacher training

Target learning objectives:

learning stylesrole of teacher vs student in learningdefinition of learningare test results consistent with learningcan predict how well students will do?how to communicate advice so that it will beacceptedstudent preparation prior to courseremedial strategiesstudent developmentgrades & rewards

Advice to students: (from Dale Roy, McMaster, withpermission)You are teaching a large enrollment first year course for thesecond time in a row. Last year it was a matter of getting thebugs worked out but this year you are determined to make thecourse more effective.One thing that you found peculiar last year was that you werenot really able to say who would do well in the course. Somestudents that you thought were doing well, had a tough time bythe end of the year; the final work of several others was apleasant surprise. It seems that different students learned quitedifferent things and in quite different ways.The first month of term is over. you have decided to set asideyour lecture for next week and to talk instead to your studentsabout learning.

! Case 5 They just don’t pull their weight!

Context: third year, week 6, second cycle ofthe PBL process;

Target learning objectivesAssertivenessDealing with conflictIndividual accountability in groupsTeacher anticipating problems and setting uprules for coping ahead of timeHow the teacher can hold individualsaccountable to each otherContracting“Norms” meetings to establish acceptableconductLearning stylesEffectiveness of prep. workshops

They just don’t pull their weight!

You are using PBL in a third year course. This is the first timethe students have experienced this approach in an otherwiseconventional curriculum. You assigned students to groups of fiveor six and tried to ensure that you had a mix of abilities. Indeed,the grade average for all eight groups that you set up was about76%. Before introducing PBL you ran two hour workshops on“group skills”, “being assertive”, “how to teach in the teachmeetings” and “managing conflict”. For the first case problemall of the groups seem to be working OK and the individualreports about the group work didn’t show any anomalies. Wehave just had the second teach meeting. Michelle knocks on yourdoor. After talking briefly about the weather Michelle complains“The other members in my group don’t pull their weightespecially in the teach meeting! In the goals meeting eachcontracts with the group that they will “teach” a topic. Thenwhen we have the teach meeting the others don’t come prepared.They have no notes to hand out. They don’t teach me anything.They tell me that I do a good job of teaching them. But they justdon’t pull their weight! I want to leave this group and joinAndre’s group. Andre says it’s OK.”

5-8

! Case 6 Paul’s decision

Context: Lifelong learning course to first yearstudents at City University of Hong Kong

Target Objectives:Setting priorities; Managing timeBudgeting timeClearer understanding of the expectations inuniversity for time to studyLong and short term planningProcrastination; Learning to say No!

Paul’s decision (Ed Ko, City University of Hong Kong)Paul has been persuaded by his friends to run for a position onthe Departmental Society. He really would like to, but he isafraid that doing so might take time away from his otheractivities. He is already on the university swimming team andhas to work five hours a week in order to earn some money topay back his credit card loans. With six courses that he is takingthis semester, he feels that he is constantly behind in his work.

! Case 7 Mary’s group

Context: Lifelong learning course to first yearstudents at City University of Hong Kong

Target Objectives:

Responsible attitudeCommunication among group membersGroup work/ meeting skillsDiscussion skillsInterpersonal relationships

Mary’s group (Ed Ko, City University of Hong Kong)

Mary has been frustrated over the progress of her group on theirproject. It has always been difficult to schedule meetings. In thefirst meeting, two of the five group members arrived late. In thesecond meeting, one member did not show up at all. Eventhough each meeting lasted several hours, they could neveragree on what needed to be done and who should do it. Nowthey are faced with a deadline for a progress report. Mary ischarged with the responsibility of editing each member’scontribution into a single report but she can’t even locate one ofthe group members let alone get their work.

! Case 8: Bitter Cherries

Context: Clinical clerkship; hybrid program

Target objectives:

Biochemical: electron transport system andoxidation redox reactions.Enzyme bindingFever and temperature regulationConsciousness and different states of alertness,dizzinessPharmacodynamic and drug metabolismInterviewing patients to clarify symptomsDelivery of emergency health care in remoteareas.

This is handled in two scenarios with 3 h ofstudent research-prep work between Goals andteach.

Bitter cherries (Luis Branda and Barbara Ferrier,Biochemistry, McMaster University)

It’s cottage closing time. The last evening of their weekend attheir cottage in the outskirts of the small community ofBancroft, Ontario, the family is relaxing. They remember thatearly in the season they collected choke cherries and put themin a bottle with vodka with the intention of making sort ofcherry brandy. The bottle is brought to the table; the vodka hasnow a strong red color and smells like Amaretto. Two membersof the family drink some of it and find it very palatable. Afterthe short time they feel hot and dizzy; they say they don’t feelwell and want to be taken to the local hospital. Their breath hasthe odour of bitter almonds.

Situation IIThey are rushed to the emergency clinic in the town hospitalwhere they are given amyl nitrite to inhale and an intravenousinjection of 3% sodium nitrite, followed by 25% sodiumthiosulfate. Oxygen is administered.

! Case 9: Tony LoPresti

5-9

Context: Level III, Adult oncology program,tutored group; Hybrid program

Target Objectives

Hodgkin’s diseases,Peripheral stem cell transplantHodgkin’s versus non-Hodgkin’s lymphoma,staging, investigations; diagnosis of Hodgkin’sdisease; short term and long termcomplications therapy, infertility Cancer therapiesYoung adulthood, sexuality, role changeChronic illness Crisis and copingUncertainty and anxietyFamily assessmentSurvivorship issuesAdvocacy issues related to clinical trials,patient decision making, quality of care issuessuch as delay in diagnosisEthicsEpidemiology: etiology, incidence, prevalence,prognosis and treatment

This case will be handled over four weeks asessentially three sequential aspects of the case.Students are expected to spend 3 to 5 h eachweek on this case. Concurrently they haveother courses.

Tony LoPresti (Barb Love, Oncology, McMaster withpermission)Situation ITony LoPresti is a 22 year old man with a history of newlydiagnosed Hodgkin’s Disease. He is being seen in the outpatientOncology clinic today, Dec 15, for further staging of his diseaseand treatment. He has had a six month history of fever, nightsweats and weight loss of 5 kg. He has back pain that is madeworse with alcohol consumption. He was initially diagnosed tohave chronic prostatitis and was treated with high dose septa inOctober and November. However, his symptoms did not resolve.Two weeks ago he found a lump in his left groin; lymph nodeexcision and biopsy last week revealed a diagnosis of Hodgkin’sdisease. His mother has come with him to this appointment.You are the primary care nurse assigned to care for Tonythroughout the course of his treatment.

Situation II: Mr. LoPresti was admitted to hospital yesterday because ofaltered mental status. Over the past week he has been jittery andlast night had one episode of paranoia. He has been admitted forfurther assessment and observation.As this patient’s primary care nurse from the clinic, how can youparticipate in his care during this period of hsopitalization?

Situation IIITony is being seen in the clinic today for recurrence ofHodgkin’s approximately 6 months following the completion of8 cycles of MOPP/ABV chemotherapy. He is being consideredfor a peripheral stem cell transplant.

As his primary care nurse, how will you assist Tony and hisfamily in preparing for this intensive therapy?

Situation IV

Tony is now 1 month following a peripheral stem cell transplantand consolidation radiotherapy for relapsed Hodgkin’s Disease.Phsyically he is recovering fairly well and appears to be inremission. He will require monthly follow up in the clinic for thenext 3 months and then will be seen every 3 months for the next2 years.

As his primary care nurse in the clinic, what needs and issuesmay arise during Tony’s process of recovery and long termfollow up?

! Case 10: Process safetyContext: Chemical process analysis. For the past threeweeks we have been analysing the process to make

maleic anhydride from butane. The students have thedetailed Process & Information Flow Diagram.

5-10

Target learning objectives:

Given the name of a chemical, you will be able toidentify whether the chemical is on the HON list, theHON Section F list.

Given various sources and data for the hazardous natureof chemicals, you will be able to define the terms andinterpret the degree of hazard and the implications.

Given a process, you will be able to use HAZOP (orequivalent procedures) to identify the conditions forunsafe operation and recommend corrective actions.

Ideal but not critical learning objectives:You will be able to describe the Natural Step approachand apply it to this process.

Upcoming visit from Occupational Health & Safety

You are the process engineer for the maleic anhydrideprocess. Recently, a process in the US, similar to oursexploded. Fortunately no one was injured but theensuing fire caused ½ million dollars US damage.Furthermore, new environment legislation is beingproposed that really clamps down on emissions andwater discharge. We also are having a visit, in fourmonths, from the occupational health and safety branchof the government. Your supervisor requests that yousystematically look over your process.

! Case 11: Heat exchanger

Context: Engineering course in heat transfer

Target objectives:

Size a double pipe heat exchanger.

Will the Heat exchanger work?

300,000 lb/h of crude oil, (heat capacity = 0.475 BTU/lb F;viscosity 2.9 mPa.s; thermal conductivity = 0.0789 BTU/ft.h.F;density = 51.5 lb/ft3) are to be heated from 70 to136 F by heatexchange with the bottom product from a distillation column.The product (heat capacity = 0.525 BTU/lb F; viscosity 5.2mPa.s; thermal conductivity = 0.069 BTU/ft.h.F; density = 54.1lb/ft3) at 257,000 lb/h is to be cooled from 295 to 225 F.Available is a tubular exchanger with an inside shell diameter of231/4 in having one pass on the shell side and two passes on thetube side. It has 324 tubes 3/4 in OD of 14 BWG and 12 ft longarranged on a 1in square pitch and supported by baffles with a25% cut spaced at 9 in intervals. Will this exchanger do the job?

5-11

Table 5-1: Case 10 revisited:

Case 10 Criteria

Problem Upcoming visit from Occupational Health & Safety

You are the process engineer for the maleic anhydride process.Recently, a process in the US, similar to ours exploded. Fortunately noone was injured but the ensuing fire caused ½ million dollars USdamage. Furthermore, new environment legislation is being proposedthat really clamps down on emissions and water discharge. We also arehaving a visit, in four months, from the occupational health and safetybranch of the government. Your supervisor requests that yousystematically look over your process. As you are thinking about this assignment, Kim walks by and suggeststhat the HON list would be helpful; Kim suggests that the HAZOPapproach is a good systematic way to solve the problem. “Is sustainability something I should also consider?” Kim thought for amoment and then suggested that this was not a direct concern for thisproblem but the visitors would be impressed if we had at least thoughtabout sustainability.

3. Multidisciplinary6. Openness 7. Motivational8. Authentic professionalpractice10. Only raw data aregiven that are typical ofprofessional practice

Priorknowledge

Context: Chemical process analysis. For the past three weeks we havebeen analysing the process to make maleic anhydride from butane. Thestudents have the detailed Process & Information Flow Diagram.

2. Builds on pastknowledge

Newknowledge

Use of systems thinking, integration of health and safety issues intoconcept of a process

6. Openness andintegration

Cues chemical process, exploded, emissions, water discharge, environmentallegislation, government, health and safety, HON, systematically identifypotential hazards for a process, HAZOP, sustainability

4. Cues given

Learningissues

Given the name of a chemical, you will be able to identify whether thechemical is on the HON list, the HON Section F list.

Given various sources and data for the hazardous nature of chemicals,you will be able to define the terms and interpret the degree of hazardand the implications.

Given a process, you will be able to use HAZOP (or equivalentprocedures) to identify the conditions for unsafe operation.

Given a process and identified conditions that are unsafe, you will beable to recommend corrective actions.

For enrichment about sustainability:Given a process you will be able to apply a systematic approach, suchas the Natural Step approach, to identify areas for potential action

1. Based on learningobjectives1. Achievable with fourmajor learning objectives1. Achievable with eachobjective requiring about3 hours of research plus 2hours of planning how toteach.6. Requires challengingthinking

! Case 12: Investments

5-12

Case 14. For a section of a process, what kinds of hazards to life (because of health, fire and explosivity)might occur? For the present focus on identifying quantitatively the type of hazard and identifying conditionsthat might cause the hazard to exist.

Context: Engineering economics

Target objectives:Concepts and calculations for interest and depreciationHow to read a financial statementInvestment options versus riskPersonal finance and budgeting

Investment opportunityYou have been hired with an annual salary of $54,000.Your company is listed on the stock exchange and haspublically traded shares. They offer a share purchaseplan to all employees. For every share of common stockyou purchase, they will purchase one in your name.How do you respond to the company’s offer. A copy ofthe annual report and its performance on the stockexchange are given.

Case 13: Intro to Chem Eng (based on material of Wayne Seames, Univ of North Dakota)

Context: Introduction to chemeng

Target objectives:Concept of unit operationDifference between a chemicaland physical processThree common systems of unitsConcepts of dimensions andhow to use them

Ollie and Marta’s first day“Welcome to our company,” said Tex Mexan, Ollie and Marta’s new boss.“We’ve been shorthanded and we want you to start your experience with ourcompany working on a small process we need to develop for our Marislandplant. They want to convert their existing Polytoxic plant to accommodate thepolygreeneric process. It was developed by a former employee. However, wehave to find a way to make that polymer safer without increasing costs orreliability. Here’s the file; get to work!”

The file contained 15 process flow diagrams of the existing plant. It was a seriesof drawings documenting the equipment, flow rates, temperatures and pressuresin all the existing process (all given in common units of measurement, lbs, psi,oF). The file also contained a report documenting the polygreeneric process. Itcontained page after page of calculations (all in metric units, g, kg/ cm2, oC) plusone simple generic sketch - boxes really - of the process. “How can we make sense of such a huge problem? I just don’t see how we canrelate these two, disparate documentation sources to each other?” Ollieexclaimed. “Besides, I thought that when you made a polymer you just reactedstuff together and made the polymer. This drawing shows all types of equipmentbut I only see one piece of equipment called reactor. What’s going on here? Arethey just trying to make it complicated because we are new to the company?”

Marta closed her eyes and tried to think of something she had read about thatmight relate to this problem. Suddenly she smiled and said to Ollie “Unit Ops”.

“Unit OPs? What’s that?” Ollie had never heard of this subject before and hecouldn’t see how this could help them make sense out of the two files and linkthe two together to make the new process. Did some of the equipment in the newprocess replace some in the old process or did you just link the two sets ofequipment, or files, together?

Resources

5-13

MPS Unit 22 Broadening Perspectivesexample, p. 215 MEK from D.G. Austin and G.V. Jeffreys (1979) "The Manufacture of methyl EthylKetone from 2-Butanol," Institution of Chemical Engineers, London. p. 167-178 for the reactor sectiononly.

Handout re TLV

W.F. Kenney (1993) "Process Risk Management Systems," VCH Publishers, New York. p. 4-18, *** reading for introduction.Hazard Identification p. 47-59, 64-74; **Hazard necessary? p 75-113. ***Risk Identification. p 115-150 ****Risk Assessment, p. 151-165; 175-184; ***

T.A. Kletz (1992) "HAZOP and HAZAN: Notes on the Identification and Assessment of Hazards," Institution ofChemical Engineers, UK, Rugby, Warwickshire, UK Defines HAZOP, p. 7 as the process of considering none,more of, less of, part of, more than and other than. ** [In MPS 22 we consider "start up," "shut down," "switchover," "power failure," "barrier failure." plus No, Add, +/-, Part, unexpected, p. 2211]

example, p. 14, ***fault trees, p 49 **

T.A. Kletz (1985) "What Went Wrong?” Gulf Publishing Co., Houston TX. Examples, not much theory. *

G.L. Wells and L.M. Rose (1986) "The Art of Chemical Process design," Elsevier, Amsterdam- Chapter 13 Safety and Loss Prevention. Very similar to Wells (1987).

- options: p. 464-467 ***- HAZOP. p. 469-475 **- HAZAN fault tree, 475-483 **- example, p. 496-505. ***

G.L. Wells (1987) "Safety in Process and Plant Design," Chapter 9 in "Recent Developments in Chemical Processand Plant Design, Y.A. Liu, H.A. McGee, Jr., and W.R. Epperly, John Wiley and Sons, New York, NY

- very similar to Wells and Rose- options, p. 333-335. ***- HAZOP, p. 338-339 **- HAZAN p. 341 **- example, p. 348-355 ***

Woods, D.R., transparencies for "lecture notes" **

5-14

Air concentrations

TLV-TWAC (TLV-TWA) Threshold Limit Value-time weighted average

Time weighted average concentrations (or levels) ofan agent for an 8-hour day with a weekly exposure of40 h to which, it is believed, nearly all workers maybe exposed, day after day, without experiencingadverse effects. Note that TLV-TWA states "nearlyall workers" - not "all workers." There will be anoccasional worker who is hypersusceptible by reasonof a peculiar genetic condition or unusualenvironmental factors and therefore may not beincluded in these limits even though the limits containlarge safety factors.

TLV-STEL (Threshold Limit value- short termexposure limit)

Maximum concentration of an agent in the air towhich workers may be exposed from time to time.However, the exposure of a worker to this maximumconcentration should be:

- not greater than 15 min.,- not greater than four times per day- only after 60 min have elapsed from thetime of the last exposure to such a condition.

TLV-C (Threshold Limit Value-ceiling)

Concentration of an agent that should not beexceeded at any time.

Misuse of Criteria

1. Do not use these criteria to "compare toxicity."SO2 TLV-TWA 2 ppmH2S TLV-TWA 10 ppm

SO2 is not 5 times more toxic than H2S. Eachhas an entirely different effect on the humansystem.

2. These data apply to workers. They do not accountfor populations of all ages and states of health or forperiods of exposure > 8 h work day.

Use

Calculate the time-weighted average exposureconcentration for any given set of conditions and seehow this compares with the standards.

Case 14 is similar to Case 10 and Case 10 revisited, Table 5-1. The case problem has different wording. Which onedo you prefer and why?

Case 14 illustrates how one might provide the students, and the librarians, with a resource list. This addresses Step 9in the process of implementation.

Additional examples of Problems in the context of managing people are given p 11 ff in the book Wee, Sun, Boon,Ng and San “A PBL Module on Managing People,” Prentice Hall, 2003

Roles of students and of teacher in PBL

Table 5-2 suggests the roles of the students and the teachers for the following version of small group, self-directed,self-assessed interdependent PBL using tutorless or autonomous groups. Here one problem is considered each week;there are three meetings: the goals meeting, the teach meeting and the exam or feedback meeting.

5-15

Table 5-2 Roles of students and teachers for PBL

Activity Student’s role Activity Teacher’s role

1 Preparation

Prepare learning objectives

2 Create problems

3 Create list of resources & tell library

4 Locate room with flat floor, movable tablesand chairs

5 Try out problems, use cues to create objectives Test out problems

6 Revise problems

7 Assign students to group; prepare list togive to students

8 Create rules about attendance, failure tohand in reports. The letter

9 Make list of chairperson responsibilities;give to students

10 Prepare startup workshops*

11 Students participate in workshop Preparethestudents

Run workshop: What is PBL?*

12 Students participate in workshop* Workshop: are you ready for PBL*

13 Student complete Perry Inventory

14 Students complete Jungian typology, LASQand KAI*

Teacher score as needed (KAI). Jungianand LASQ are self scoring*

15 Students participate in workshop* Teacher prepare process skill training*

16 Student write up journal about the workshop*

17 Teacher mark journal: 50% for skill in selfassessment and 50% for target skill*

18 Students participate in workshop* Workshop: self assessment MPS 3*

19 Student write up journal about the workshop*

20 Teacher mark journal: 50% for skill in selfassessment and 50% for target skill*

21 Students participate in workshop* Workshop: Group skills. MPS 28*

22 Student write up journal about the workshop*

23 Teacher mark journal: 50% for skill in selfassessment and 50% for target skill*

5-16

Activity Student’s role Activity Teacher’s role

24 Students participate in workshop* Workshop: strategy. MPS 4*

25 Student write up journal about the workshop*.

26 Teacher mark journal: 50% for skill in selfassessment and 50% for target skill*

27 Students participate in workshop Workshop: Goals and Teach meeting*

28 Students get in their assigned groups. They willstay with the same group for all semester (12weeks). Students may have a starting “norms”meeting to introduce themselves and to addressthe guidelines of how they want their group towork

Teacher provides list of 17 possible thingsto think about for an effective group (p B-35, Resources book)

29 Chairperson prepares and distributes agenda

30 Goal Meet Students read over the problem, identify whatthey need to learn, write as learning objectives

Teachermonitors

Teacher checks on the list of learningobjectives from each group

31 Teacher signs form 1-13

32 After the students receive the validation fromthe teacher, they formally write up theirlearning objectives, they contract with eachother. Set up rules of behaviour, share learningstyle information with other group members

Teacher monitors to see that all groupmembers are present; makes notes if somemembers are missing

33 Individuals complete the feedback tochairperson form (if chairperson skills are partof your program) and gives form to chair

34 Group completes assessment form about groupprocess; 1-3; will use as evidence about groupskill development

35 Individuals learn about their topic, prepare“teach notes”; think about the learning styles ofall the members

Teacher is a consultant to any student whoasks

36 Chairperson prepares and distributes agenda

37 Teach Meet Each student teaches the others. Monitor Teacher monitors to see that all people arepresent; if some were missing at the firstmeeting, asks the group if they want theteacher to send the letter; sends letter todelinquent student if authorized. Monitorsto see that all members are present for theTeach Meeting

38 Group use the information to solve theproblem. They check that the learningobjectives given in step 32 were achieved.

39 Group elaborates about how the newknowledge could be used in other contexts;reflects on what they learned

5-17

Activity Student’s role Activity Teacher’s role

40 Individuals complete the feedback form p 1-12on quality of the teach for each member(including self) and give forms to everyone

41 Individuals complete the feedback tochairperson form (if chairperson skills are partof your program) and gives form to chair

42 Group completes assessment form about groupprocess; 1-3; will use as evidence about groupskill development

43 Students study on their own or as a group anytopics that they don’t understand.

44 Chairperson prepares and distributes agenda

45 Each student prepares a “good” 10 minquestion (and answers). The question should bein the topic that the student did not teach inactivity 34 and 37.

46 Exam Meet Students critique the various questions andselect the “best” using the criteria of thelearning goals approved by the teacher inactivity

Teacher monitors to see that all people arepresent; if some were missing at the firsttwo meetings, asks the group if they wantthe teacher to send the letter; sends letter todelinquent student if authorized. Monitorsto see that all members are present for theExam Meeting. Keeps track of time!

47 Each team receives a question from anothergroup. The team writes out a group answer.

Teacher collects one question from eachgroup and delivers it to another group suchthat the source of the question is not knownby the students. Keeps track of time!

48 Markers are sent to each group to mark thegroup answer

Teacher reveals the source of the questionand asks that each group send a “marker”

49 Group puts together a copy of its 1) bestquestion and 2) answer and 3) the group’sanswer to the question they created and hadmarked in step 48. Each group submits this toteacher.

Teacher marks the three results and recordsmarks for all group members

50 Individuals complete the feedback tochairperson form (if chairperson skills are partof your program) and gives form to chair

51 Group completes assessment form about groupprocess; 1-3; will use as evidence about groupskill development

5-18

Activity Student’s role Activity Teacher’s role

52 Students study on their own or as a group anytopics that they don’t understand.

53 Individual students prepare report about PBLbased on evidence (31, 32, 35, 38, 39, 40, 43,45, 52) and his/her analysis of the evidence andsubmit to teacher

Teacher mark journal: 50% for skill in selfassessment and 50% for target skill of PBLor lifelong learning

54 Individual students prepare report aboutchairperson skill based on evidence (28 and 33or 36 and 41 or 44 and 50) and submit toteacher

Teacher mark journal: 50% for skill in selfassessment and 50% for target skill ofchairperson

55 Individual students prepare report about groupskills based on evidence (34, 42, 51) andsubmit to teacher

Teacher mark journal: 50% for skill in selfassessment and 50% for target skill ofworking in groups.

56 Cycle repeats starting at 29

* These activities only have to be done once.

The letter is given on page D-24 of the Resources book and can be downloaded fromhttp://www.chemeng.mcmaster.ca/innov1.htm and click on PBL and then download Chapter D from Problem basedLearning: Resources to gain the most from PBL.

For activity 28, the norms meeting, the 17 issues are, 1 need for chairperson, 2 terminology for problem solving, 3.type of brainstorming approach, 4. decision making procedure, 5. role of chairperson in decision making, 6.resources needed, 7. record and distribute minutes of meeting?, 8. roles, 9. meeting agenda, 10. Sandler’s rules, 11.how to handle conflict, 12. how to combat group think, 13. willingness to share personal information about “styles”with all group members, 14. level of intervention desired, 15. how to handle emergencies, 16. how to handleemotional issues brought to the group, and 17. how to ask a member to leave the group.

6-1

6. How to develop processing skills for learners? copyright © 2002, Donald R. Woods Donald R. WoodsMcMaster University, Hamilton, Canada

Case 4: How much is enough? “We have taken six hours of precious class time to run workshops on Problem Solving Strategies and on Groupskills” exclaimed Jon. “We can’t take more time to train them in these “soft skills”. Besides, what additional skillsdo they need? These students are graduating from a Polytechnic! They’ve got to have technical knowledge. Let’s geton with it. Let’s use the first case problem on Monday.”

*********************************

Activity: In small group of 5 or 6, with chair _______________________________; reporter ________________________, brainstorm the issues this case raises. Identify what you know already.Identify what you need to learn.

Prioritize the issues: criterion: what do you want to gain from this workshop in the context of the issues raised._________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

6-2

Feedback about the group work. Form 2802

Task: Problem defined, many issues and hypotheses explored, criteria listed and the issues prioritized. Refrainedfrom early closure. Task carried out and looked back at the result to assess it. Group agreement as to goals. Processwas active with monitoring. Completed task on time. The accuracy in the group’s answer matched the time available.Group avoided contributing excessive information. None of Few of these Most features All of thesethese behaviours but demonstrated behavioursbehaviours major omissions

G G G G G G G1 2 3 4 5 6 7

Morale: Group relaxed; enjoyed working together. They gave emotional support to each other and were able toexpress disagreement or disappointment directly. Seven fundamental rights preserved. Members are enthusiastic andinvolved. None of Few of these Most features All of thesethese behaviours but demonstrated behavioursbehaviours major omissions

G G G G G G G1 2 3 4 5 6 7

Individual Contribution to Task and Morale

____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Group Strengths Group Areas to work on_____________________________________ ________________________________ ____________________________________ _____________________________________________________________________________________________________________________________________________

from D.R. Woods (1995)

6-3

Feedback from the Goals Meeting

Case 4:IssuesNumber identified: 1 2 3 4 5 6 7 >7

Agreement with tutor <50% 50% 60% 70% 80% 90% 100%

Knowledge/skills to be learned Consensus among group little some a lot complete

Agreement with tutor's list little some a lot complete

Learning objectivesQuality poor fair OK good excellent

LearningQuality of questions asked during the teach session none some astute excellent

Willingness to continue to contribute <50% 50% 60% 70% 80% 90% 100%

Your AttitudePerry shift 2 3 3.5 4 4.5 5******************************************************************************

Reflections:______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________-

6-4

1. The need

The processing skills are needed for two reasons.1. The usual set of desired outcomes for programs include expertise in the subject discipline andskills in oral and written communication, problem solving, team work, managing change andlifelong learning.

2. The usual set of skills needed by students so that they perform well in PBL are skills in oraland written communication, problem solving, team work, managing change, self assessment,elaborating and learning and teaching others. A common core of skills include: problem solving (seeing strategies, monitoring, creativity, defining problems, creating the lookback)managing change or making the transition to a new learning environment (stress and timemanagement, understanding personal uniqueness and skills)group skills (interpersonal skills, group work, managing conflict, developing trust)teaching others (learning skills, understanding personal uniqueness, goal setting, learningcontracts) self assessment.

2. How to develop

The graduates need skill and confidence to that they can use the skill effectively. As Bandura hasillustrated, workshops, that give participants a chance to try a skill and get prompt feedback,seem to be the most effective approach.

Step 1. Define the skill base on research, not intuition. Publish target behaviours of successfulpersons performing the skill. Some examples are published:http://www.chemeng.mcmaster.ca/innov1.htm and click on MPS and look for “target skills”

Step 2. Create learning objectives for the skill. Some examples are published:http://www.chemeng.mcmaster.ca/innov1.htm and click on MPS and look for “objectives”

Step 3. Design the worksheets (feedback forms and forms of evidence that participants will useto show performance accomplishments)

Step 4. Work out the “timing sheets” for how you will manage the workshops.An overall template for the workshops

DefineRationalizePretestObjectivesRoute ahead

6-5

Activity with feedbackReflectionsActivity with feedbackDescription of target behavioursActivity with feedbackReflectionsActivity with feedbackContinue until the participants feel they can achieve the objectivesObjectivesPost TestDiscovery

In Large Class-- Workshop #2 -- we introduced the workshops MPS 4 Problem Solvingstrategies and MPS 28 Group skills.In this workshop, we introduce workshops:Managing Change MPS 49Self-directed Learning/teaching MPS 36

Enrichment:1. Please help me understand how best to help you2. MRIQ3. MPS 49 and 36 (goals, targets, workshop, timing)

References:http://www.chemeng.mcmaster.ca/Innov1.htm and download MPS materialsWoods “Problem-based Learning: resources to gain the most from PBL” Chapter D forobjectives. downloadable from http://www.chemeng.mcmaster.ca/innov1.htm and PBL; andChapters B and C.Woods “Problem-based Learning: helping your students gain the most from PBL” Chapter 3 downloadable from http://www.chemeng.mcmaster.ca/innov1.htm and PBL for an overview ofthe process skills.

6-6

a74 ©copyright Donald R. Woods, April 2002

MPS-4 Strategy

Donald R Woods, McMaster University, Hamilton

CANADA

Jan. 200, Apr 2002

A Strategy is an organized approach used in solving a problem. Such an organized approach identifies steps or stagesfor different parts of the process. For example, there is usually a stage called "Read the problem". Another stage mightbe called "Check the answer I obtained." The author George Polya describes a four-step strategy that includes the steps"Define, Plan, Do it, and Look Back". A strategy is important because:

- we all usually use one,

- a strategy helps us to be organized and systematic,

- having a strategy helps to calm us down if we become anxious when we are given a very difficult problemto solve,

- having a strategy helps us to "monitor" our mental processes.

In this UNIT the goal is to help you to discover names for the stages you use, to help you identify the thinking andattitudinal skills needed during different stages, to help you see how many minutes you spend in the different stagesand to help you to monitor your thinking. To help you get this skill, two of you will work together: one will play therole of a "TALKER and marker mover" ; the other plays the role of the "LISTENER and recorder".

6-7

McMaster MPS 4: Strategy Objectives © copyright, Donald R. Woods, 1998MPS 4: Strategy: is the organized and systematic sequence of stages applied when problems are solved. Each stage ischaracterized by its own set of attitudes and thinking skills. Example: the four-step problem solving strategy publishedby Polya: Define, Plan, Do it, Look back.

Skill development:1. More skill in being able to talk about thought processes.2. More practice to focus on accuracy (instead of on time).3. More practice on being active and writing things down.4. Recognizing that others solve problems differently than they do.5. Acquire more skill at listening.6. Acquire more skill in self assessment.7. Acquire more skill in giving and receiving feedback. 8. Through self awareness, to improve self confidence.9. Further emphasis that assessment is based on evidence.10. Continuing to develop an environment of trust where risking is OK.11. Beginning to recognize patterns in the problem solving process.12. To realize that a “strategy” is not applied linearly and sequentially; that it is used flexibly.13. To recognize the difference between problems and exercises.14. Understand the relationship between subject knowledge, past solutions to problems and problem solving.15. To acknowledge the importance of defining problems and to recognize this as a three-step process.16. To acknowledge the importance of reading the problem statement.17. To realize that problem solving is not “doing some calculations.” Conversely, to correct the misconception that ifyou are not “doing some calculations” you are not solving problems. 18. To acquire skill is explicitly monitoring the process.

********************************Pretest: It is very difficult to realize how much we change in our thinking and research skills from the workshopactivities that you are going to do. To help you develop your confidence and be proud of the progress you make, beforeyou do the workshop activities, please mark your skill now .

Awareness: how aware are you of what you do when you use a strategy? Rate with an “x”

0 1 2 3 4 5 6 7 8 9 10Unaware Aware of Very awareI just Some I can describedo it The details of

how I do it

Skill: how skilled are you in doing this activity? Rate with an “x”

0 1 2 3 4 5 6 7 8 9 10Poor Fair Good Very good Excellent

Comments:

_______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Learning objectives

6-8

MPS 4 Strategy

1.1 given a term listed under "concepts introduced", you should be able to give a word definition, list pertinent characteristics andcite an example.

2.1 given the name "McMaster 6-step strategy," you should be able to describe each and list the cognitive and attitudinal dimensionsassociated with each step.

2.2 you will be able to describe the concept of "nested strategy" and given a problem, identify the number of times the 6-step strategymight be used to solve the problem.

2.3 given a problem in a TAPPS Whimbey-pair context, you should be able to verbalize the process and place a marker to identifythe step (in the McMaster 6-step strategy) upon which you are working. The listener should agree with your assessment 80% of thetime. You should need prompting no more than 3 times in a 7 minute period.

2.4 given a problem in a TAPPS Whimbey-pair context, as a listener you will encourage verbalization, an emphasis on accuracy,active thinking and problem solver to move the marker correctly on the strategy board. Your interventions will be judged by theproblem solver to be helpful, and not judged to be disruptive. 2.5 given the TAPPS Whimbey-pair context, you should exhibit 4 verbal management statements during a 7 minute period of problemsolving.

2.6 you will be able to list the misconceptions people hold about the use of strategies.

4.1 given a problem in a TAPPS Whimbey-pair context, as a listener you will be able to identify monitoring statements said by theproblem solver. Your assessment will agree with the tutor’s assessment 80% of the time. 5.1 given your goal is to improve your application of a strategy, you will be able to identify subgoals, write these in observable terms,create consistent measurable criteria, gather evidence to substantiate claims and write a reflective journal to summarize your claims.

6.1 given evidence gathered from a strategy workshop, you will be able to assess the degree to which goals of the workshop havebeen achieved. Your assessment will agree 90% with the tutor’s assessment.

Concepts introduced

McMaster 6-step strategy, criteria for selecting a strategy, attitudinal and cognitive dimensions of each step in the strategy,Schoenfeld's monitoring/management, nested strategy, role of TAPPS problem solver, role of TAPPS listener, three stages of defininga problem, importance of creating the internal representation, importance of reading the problem statement.

***************************

6-9

Playing the role of the TALKER- MOVER of the marker.

Your role is to talk aloud when you work on the problem and you are to move a marker on the "strategy board" so thatyou show which stage your thinking is in.

This is difficult to do. Be patient with yourself.

You may not completely understand the meanings of the stages yet. You may use different stages than the ones on theStrategy board. Please, do your best.

The listener will not move the marker for you. The listener will nottell you what stage you are in. The listener mightask you " are you still in the "Explore" stage?"

Remember to keep talking, to be active, to use pencil and paper, and to check and check again. Before you start, go overthe meanings of the 6 different stages in the McMaster-6-Step strategy with the LISTENER. Agree on the meaningsof the words. 1. Sit side by side; have paper and pencils available, have the Strategy board and the marker.

2. The talker moves the marker to the READ part of the Strategy Board and starts by reading the problem statementaloud.

3. Then move the marker to whatever stage you are going to work on next and start to solve the problem on you own.Keep talking aloud. You are solving the problem. Your partner is only listening to you. He or she is not solving theproblem with you or for you.

4. Talking and thinking and moving the marker at the same time are not easy. You might forget to move the marker.That is OK. Do the best you can. You are playing the role and trying out something from Canada.

5. Go back and repeat any stage of the strategy you wish. *****************************************************************

Playing the role of the LISTENER-RECORDER

You have an important and difficult role to play. You are to help the Talker see what he or she is doing as they talkabout the travels of their mind as they solve problems and you are to record the amount of time the Talker spends ineach of the stages on the Strategy Board. Do not correct them; do not argue with them about which stage the TALKERis working on. Do not move the marker for them. You may have to ask "Are you still in the >>>>> stage? Before youstart, go over the meanings of the 6 different stages in the McMaster-6-Step strategy with the TALKER. Agree on themeanings of the words. You might want to review the instructions for the LISTENER from MPS Unit 1 onAWARENESS.

*****************************************************************Activity: In pairs, one be a talker, the other be a listener. The talker plays the role for ___________ and “solves”problems during all the allotted time. Do not change roles.

Later you will switch roles. The Strategy Board is given on PageThe listener record sheet is given on Page ++++++++++++++++++Write evidence by reflection on Table 4-1 p. 4-5; Get evidence about the listening on Table 4-2, p. 4-6.

Table 4-1 : A Place for you to Record your ideas about the Strategy Activity: Reflections

6-10

Being the TALKER-MOVER of the marker:

_______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Being the LISTENER/RECORDER_____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

About the process________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

6-11

Table 4-2 : Evidence for listening: Feedback to listener: listener you will .encourage verbalization, an emphasis on accuracy,active thinking and problem solver to move the marker correctly on the strategy board. Your interventions will be judged by theproblem solver to be helpful, and not judged to be disruptive.

Activity 1: talker ________________________ Case ________________ listener ___________________________

encourage verbalization: not needed interruptive OK really helped

encourage emphasis on accuracy: not needed interruptive OK really helped

encourage active thinking not needed interruptive OK really helped

interventions: not needed interruptive OK really helped

Comments:

_____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

signed ________________________________ talker+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Activity 2: talker ________________________ Case ________________ listener ___________________________

encourage verbalization: not needed interruptive OK really helped

encourage emphasis on accuracy: not needed interruptive OK really helped

encourage active thinking not needed interruptive OK really helped

interventions: not needed interruptive OK really helped

Comments:

_____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

signed ________________________________ talker

6-12

Schoenfeld and Voss have shown that we should "monitor" how well the thinking activities are helping us toward thegoal of solving the problem successfully. The monitoring process includes:

M Assessing the benefit to be gained from a thinking activity or a calculation before we do it. For example, "If I calculate ...., what will that tell me?"“If I ask this question ....., what will that tell me?

M Assessing whether a task is completed. This is a natural question that you ask yourself when you are ready tomove the marker on the Strategy board. For example, "Am I finished with this?" "Now where do I go?"

M Assessing what you have learned if a hypothesis is shown to be wrong or if you calculate a "strange answer."For example, "OK, What did I learn from that?"

M Checking for completeness....- do you list the options? then put them in order of importance? and then check them off when you do them?- do you give up on a calculation without checking on what you learned from the work?

M Monitoring.... where you are and where you are going... at least once per minute. Schoenfeld’s evidence is in Table 4-3; Other target evidence is shown in Table 4-4++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Activity:In the second time that you play the role of the TALKER-MOVER you will now try to say aloud the assessing andchecking for completeness and monitoring described above.

In the second time you play the role of the LISTENER-RECORDER you will also put a little black arrow – on the chartwhenever the talker assesses, checks for completeness and monitors.

Ref: Schoenfeld, A.H.(1985) Mathematical Problem Solving" Academic Press, Orlando, Fla.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Table 4-3 : Schoenfeld’s research :

Unsuccessful Successful

Don’t assess potential Assess the potential: what will I learn if I do this calculation? Will thishelp me? How does this move toward a solution?

If fail, abandon If fail ask “What did I learn?”

Make impetuous jumps with noapparent reason

Actions characterized by continual assessment & curtailment

Within the first few minutes, becomefixed on one idea

Qualitatively explores situation to fully understand; Keeps optionsopen

No monitoring Explicitly monitors about 1 / minute; anduses transitions between strategy stages as key monitoring/decisionpoints.“Am I finished with this stage?” “Where now?”

6-13

MPS 4 Strategy © copyright, Donald R. Woods, 1998

Table 4-4 Evidence-based targets for problem solving

Evidence-based targets Progress toward internalizing these targets

20% 40% 60% 80% 100%

M Spend time reading the problem statement. (Up to three timeslonger than unsuccessful problem solvers) (11, 24)

M Define the problem well; do not solve the wrong problem. Bewilling to spend up to half the available time defining theproblem. Most mistakes made by unsuccessful problem solversare made in the define stages (1, 3, 4, 11, 12, 19, 24)

M You solve your mental image of the problem; such a mentalimage is called the internal representation of the problem.

M Differentiate between exercise solving and problem solving.

M Unsuccessful problem solvers tend to search for an equationthat uses up all of the given variables. (1, 4, 11, 12, 14, 24)whereas successful problem solvers focus on an organizedstrategy that focuses on defining the real problem.

M Unsuccessful problem solvers tend to memorize and try torecall equations and solutions that match the situation instead ofdefining the real problem and identifying key funda mentals.(11, 6)

M Defining the problem is a three-stage activity (19, 24).

M Unsuccessful problem solvers tend to take a trial and errorapproach; successful problem solves use a systematic strategy(6). Use a strategy to help you to be systematic and organized

M A strategy consists of a series of about 6 stages. Each stageuses different thinking and feelings. This strategy is not usedserially (following rigidly one step after another). Rather it isused flexibly; applied many times while solving a singleproblem with frequent recycling from one stage to another. (24)

M Problem solving skill interacts with subject knowledge(needed to solve the problem) and with the sample solutions(from past solved problems).

M Successful problem solvers monitor their thought processesabout once per minute while solving problems. (20, 19)

References 1 to 24 are from the Novice versus expert research summarized in PS News 55

6-14

STRATEGY MPS -4: Case 1: Judge Dee and the Caseof the Old Fashioned Pen.

Rodney Stites, professor of French at State University,lay dead across his desk. It looked like suicide.

"I hear the gun shot about an hour ago," said Carl, themanservant. "I came into the room quickly and thencalled you then." Judge Dee, the detective, walked tothe desk, that was located in the middle of theprofessor's library. "Carl, did you touch anything?" "Ionly touched the telephone," replied Carl.

Judge Dee examined the body. Death, which occurredwithin the past hour, was caused by a bullet that wasfired into the right temple at extremely close range.

A thirty-two calibre gun lay on the thick carpet on thefloor to the right of the professor's head. On the deskwas a note. The note was written in ink, with severalblobs of ink smudges. The note read: "I do not want tolive anymore without Elsie."

"Elsie, the professor's wife, ran off with a young artistlast year," remembered Judge Dee.

The judge now looked carefully at the top of the desk.The dead man had the old-fashioned quill pen clutchedin his right hand. An open, antique inkwell sat next tothe desk phone about 2 cm from the point of the pen.Some ink spots were on the desk top near the end ofthe pen. On the base of the inkwell was engraved "ForRodney on our Tenth Wedding Anniversary. LoveElsie."

"Call the police," said Judge Dee. "This is murder andnot suicide." ....HOW DID JUDGE DEE KNOW?

STRATEGY MPS-4: Case 2: The Case of the PhonyPoliceman.

"I am the lawyer for Mr Franklin D. Van Clausand, II,who has been falsely accused for murder and robbery,"said Goodwin as he sat down in the chair in JudgeDee's office. "Franklin has been working for a year inone of his father's banks. Last Tuesday, before thebank opened for business, a man dressed like apoliceman came to the front door and demanded to belet into the bank. He looked like a motorcyclepoliceman: he wore a black leather policeman jacket,boots, dark sun glasses and a white, policeman'smotorcycle helmet. Of course he wore a policeman'sbadge and he showed special identity papers. Franklinwas closing the door on the money vault, when thefake policeman drew his gun and put the muzzle of itagainst the right hand side of Franklin's neck. Heforced Franklin to fill a bag full of money. When thefake policeman was leaving the bank, the bank guarddrew his gun. Two shots were fired. The guard fell;dead.

" Franklin provided the only description of the killer.The fake policeman was about 30 years old, 1.8 m tall,with fair complexion, blue eyes and a scar on his upperlip that was shape like a crescent.

"That afternoon, the police arrested Edgar Burgess fordriving too fast. He matched the description and $300000 was found in his car. Burgess became very angrywhen he found out that Franklin had given so accuratea description of him. Burgess said that Franklin was inon the robbery with him and had told him when andhow to come to the bank. What nonsense! Franklinswears that the first time he ever saw Burgess was inthe bank." Judge Dee said "Is it really nonsense?" WHY DID HESAY THAT?

6-15

STRATEGY MPS-4 Case 3: The Case of the PhonyCrash

Judge Dee was outdoors camping and had just turnedover in his sleeping bad when he saw a big car comedown the short dirt road, which came off the mainhighway, and disappeared over the cliff.

Running after the car was a tall man. He stopped at theedge of the cliff, then he lay down on the ground andmoaned and cried, "Help! Help! My back hurts somuch!."

Four others from the camping group ran over to theman on the ground. Judge Dee went down the cliff tolook at the car. It was overturned and totally wrecked.About the only things that were not broken were thefour worn out tires, which turned slowly in the air.

Two days later, Judge Dee talked to Mr Abbott, theinsurance man who said, "Mr Staines says that whilehe was driving, he fell asleep at the wheel and woke upjust in time to jump out of the car before it went overthe cliff. Now he is asking us to pay the fullautomobile insurance worth the full price of the car,$12,000. You see, he just bought the car new five daysago. But the large amount of money that he is askingfor is for his hurt back. He says he hurt it so badlywhen he jumped from the car, that he cannot work anymore. He expects us to pay the insurance that is worth$5000 per month. We think he faked it all.

"You are the only witness, Judge Dee, but the lawyersfor Staines say that you could not have seen muchbecause it was night and you had been asleep."

Judge Dee said, "I will not have to testify. Stainesplanned to wreck his car and do this "imaginary"story."

HOW DID JUDGE DEE KNOW?***************************************

STRATEGY MPS-4: Case 4: The Case of the RailwayCrash.

On the night of July 15, the engineer of the westboundlocal train missed a signal and the train crashedhead-on with the Rocket, the high speed express trainout of Toronto. The result was one of the worstdisasters in train history.

Judge Dee said,"I think it is strange that all of theserious injuries were in the first seven cars of bothtrains...except for Jess Fromm who was killed in ninthcar of the local train. I have been asked by Fromm'sdaughter to investigate."

The police inspector asked the Judge to be seated andthen the inspector went to his files. "Fromm was goingto a merchant's convention with his business partner,Wendel Smith. According to Smith's testimony, he andFromm shared compartment C in the last car of thelocal train. Just seconds before the crash, Fromm gotup and walked toward the front of the compartmenttoward the toilet. When the trains collided, Fromm wasstanding. He was thrown back and struck his headagainst the top of the table that was between the twoseats. The seats were facing.

"Smith now owns all of the business," said theinspector. "Maybe that was the motive. What about themethod? Maybe Smith hit Fromm in the back of thehead with the table AFTER the crash. But how are wegoing to prove it?"

"That is easy to prove," said Judge Dee.

WHY DID THE JUDGE SAY THAT?

STRATEGY MPS-4: Case 5: The Case of the Shot in

6-16

the Back.

Judge Dee was out taking his daily walk on a coldNovember evening when he heard a gun shot. He sawan old man suddenly lean against the front door of anearby house, fall and lie motionless on the porch. Twoother men and the Judge ran toward the old man. Hewas dead.... shot in the back.

The Judge turned to the two other people and said,"Each of you should have a good alibi. I am sure thatone of you shot this man, and then threw away thegun".

Both men, who were wearing gloves and tightovercoats, said that they did not know the old man.Both said that they were out for a walk.

The first man said, "I am Ted Briggs. I noticed the oldman locking the front door a split second before Iheard the shot. I ran right up to him. But this man," andhe turned to the second man, "was right in front of thehouse and his position seemed to be in a line fromwhere the shot came from."

The second man said "I am Sid Cole. I heard a shot, butdid not connect it to the old man. I did not know whathad happened until I saw the two of you runningtoward the house."

The key was still in the front door. Judge Dee turned it,entered the house, and telephoned for the police.Twenty minutes later, Judge Dee explained toInspector Winters "The dead man's wife is an invalidsick in bed. The old man was going to the drugstore.He always locks the house when he leaves her alone."

"Do you have any ideas?" asked the Inspector.

"Yes," said Judge Dee, "Arrest ......" WHOM?

STRATEGY MPS-4: Case 6: The Case of the SilverPen

"The police have been here all morning, darling! It hasbeen terrible!" said the Hollywood movie star, VivianHobson. Hobson, a star for many years, was in themiddle of the stage production of "Cats" but nowlooked worn out. Judge Dee, an old friend, studied thebedroom, from which the actress's daughter Shari hadbeen kidnapped the previous night. A rope, made frombedsheets and tied to one of the bed legs, hung out thewindow to within about 2 m of the ground.

"The kidnapper must have crept into the house duringthe day, because everything is locked at night," saidVivian. "I was on the upstairs porch around midnightwhen I saw a man climbing down the rope made fromthe bedsheets. He had my daughter Shari over hisshoulder. She was limp. He must have knocked herunconscious. The bad guy!"

"Has anything been moved in the room?" asked theJudge."No, everything is exactly as it was," said Vivian.

The Judge went downstairs, and outside and saw a boydelivering newspapers. "Would you please help me doan experiment?" asked the Judge. "I want you to actout what the kidnapper did."

The boy went up the stairs, pretended to knocksomeone out, put the imaginary person over hisshoulder and then climbed out the window and downthe rope safely to the ground. As he went out thewindow, the bed was dragged several centimetres fromits position against the wall, and uncovered a silver penon the floor. The Judge picked up the pen, looked at itcarefully and said, "Vivian, you faked the kidnaping toget publicity for your new play, Cats". WHY DID THEJUDGE SAY THIS?

6-17

STRATEGY Board: (from Woods, "Problem-based Learning: how to gain the most from PBL," 1994)

6-18

Table 4-5 Record of the Talker’s strategy with – for monitoring statementsTalker ____________________ Case _________ Listener ______________________

Stage

“Define”Assess

Read: “I want to and I can!”

Define-the-stated problem: Sort the givenproblem statement

Explore the problem to discover what theproblem really is

Plan Plan

Implement Do it

Evaluate Look back: elaborate, check

0 2 4 6 8 10 12___________________________________________

Talker ____________________ Case _________ Listener ______________________

Stage

“Define”Assess

Read: “I want to and I can!”

Define-the-stated problem: Sort the givenproblem statement

Explore the problem to discover what theproblem really is

Plan Plan

Implement Do it

Evaluate Look back: elaborate, check

0 2 4 6 8 10 12

6-19

DISCOVERY

Activity Discovered So what? application

6-20

MPS 4: Strategy: Example assessment tasks:

1. As a listener in the TAPPS-Whimbey pair method, the problem solver has not moved the marker. You think she isnow "Exploring" whereas the marker is on "Plan". Your response is:

a. "Please move the marker." b. Reach over and move the marker for her so that you don't disrupt her. c. Forget it; she probably is "Planning." d. "Are you still planning?" e. "Can you check which step you are in?"f. Other (provide your specific response)

2. Your task is to identify Schoenfeld monitoring activities. Which of the following actions/statements would youclassify as "monitoring." Rationalize your choice for each. a. the problem solver puts his hand out to move the marker and pauses. b. "Let's see if I am finished with this step." c. "Which step should I go to now?" d. "What did I learn from that?" e. "If I measure the blood pressure, that should tell me..."

3. Given the following stage-time chart as evidence, to what degree can you claim to have achieved the goals of thisworkshop? Write out your discussion of this evidence.

4. From the in-class activity, you have the following evidence:- your reflections that you wrote three times during the activity.- your worksheets and the statement of the exercises.- the strategy stage-time chart including the monitoring notations. - the DISCOVERY sheet.- your awareness and skill checklist before and after the activity.

Write up a reflective assessment of the degree to which you have achieved the objectives. Refer to the evidence bynumber and relate your evidence and claims to the objectives by number.

6-21

For Teachers: Reflections about this as a learning activity____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

already would might not mydo this work work style

Writing reflections O O O OUsing strategy board O O O OUsing content-independent exercises O O O OHave to develop exercises in subject O O O OUsing the feedback forms O O O OUsing the learning objectives O O O OUsing the DISCOVERY form to summarize O O O OOther_________________________________________ O O O O

6-22

MPS 4: Strategy timing sheets For an abbreviated 1½ h version see PBL: Resources to gain the most from PBL” pagesB-16 or B-21 <http://chemeng.mcmaster.ca/innov1.htm> problem based learning.© copyright, Donald R. Woods, 1998

Topic Trans-par-encyno.

Time,min

Elapsedtime,min

Comment

1. Introduction 1 2 2 Use anteater cartoon to introduce the unit if theparticipants have not previously seen it.

2. Where it fits in 2 0.5 2.5 Always include this type of overview to show thebig picture

3. Pretest 3 0.50 3. Unlike other pretests, here the participants can writein the strategy that they currently use

4. Objectives 3 3 6 Read over, or have one person at a time read theseover. Do not expect comprehension at this time. Tellthem this. Do not give excessive elaborationyourself.

5. Route ahead 4 1 7

6. Example data 5 1.5 8.5 Show 5 and 6 simultaneously on the two overheadprojectors

7. Example process 6

8. Example strategy 7 3 11.5 Show 7 and 8 simultaneously on the two overheadprojectors; you role play the talking and marker; thenrole play the listener recorder9. Example listener

evidence form8

10. Option selectingstrategy: criteria

Opt 9 2 You can use any strategy you want. The key skillswe are developing are being organized, systematic,having a strategy and monitoring. We don’t want toget hung up arguing about the strategy. For example,for workshops with nurses, use the 4-step Nursingprocess of Assess, Plan, im0plement, evaluate.However, for this activity, the talker and the listenermust agree on the strategy. Furthermore, we haveinvested several years of research to develop theMPS 6-step. The purpose of this series of optionaltransparencies is to share that research and to get abuy-in for the strategy board in transparency #7

11. Option research Opt 10 1

12. Option 2 you might display some of the 75+ publishedstrategies

13. Option results Opt 11 2

6-23

Topic Trans-par-encyno.

Time,min

Elapsedtime,min

Comment

14. Option importance ofdefining problems andwhat we do during theprocess

Opt 12 2

15. Option visualreinforcement of threephases to DEFINE THEPROBLEM

Opt13 2 22.5

16. Activity: boredom 3 25.5 If you used the options, then the 20 min attentionspan is exceeded. Include back rubs, handshakes,stretches, or “Turn to a neighbor &... discuss thestrategy you use”

17. Seek agreement andunderstanding ofterminology for thestrategy board

Opt 14 1.5 1327

18. Leader explanation ofactivity and roles

Opt 15 1 1428

An option would be to provide written description ofroles and ask each to read the role. This should goquickly if the participants have experienced the MPS1 Awareness unit. If they have not experienced it,then you need to spend more time.

19. Getting ready. 16 2 1630

This is a paper shuffle. However, this needs to bedone carefully so that each person retains evidenceof his/her work.

20. Activity: pairsTAPPS

17 7-12 2640

Flexible time here. 7 min minimum to 12 min.choose a time to match the breaks. You want tocomplete activity #23 before the break.

21. Reflection, discussion 18 3 2943

22. Activity: pairsTAPPS

19 7-12 39 Flexible time: choose the time to match breaks

23. Reflection, discussion 20 3 42

USUAL end of a 50 min class

24. Research Internalrepresentation

21 1 1 Important concept; basis for the Explore stage

6-24

Topic Trans-par-encyno.

Time,min

Elapsedtime,min

Comment

25 Research importanceof the three stages toDEFINE THEPROBLEM

22 2 3 To some extent, this is a restatement of 14 and 15.However, since this was in the options, include ithere even if you did the option. Here more detail isgiven.

26. Research Larkin’sdata about reading theproblem statement

23 1 4

27. ResearchSchoenfeld’s researchabout monitoring; table

24 2 6

28. ResearchSchoenfeld’s data

25 0.5 6.5

29. Research Schoenfeld;data

26 0.5 7

30. So What? Activitywith monitoring

27 1.5 8.5 Show transparencies together

31. Gathering evidence ofmonitoring

28 1.5 10

32. Activity: TAPPS 29 7-12 20 repeat activity with subject discipline detectivestories; timing given is based on 10 min.

33. Reflection.

30 3 23

34. Activity :TAPPS 31 7-12 33 repeat activity with subject discipline detectivestories; timing given is based on 10 min

35. Reflection 32 3 36

36. Feedback fromSchoenfeld’s data

33 2 38 show simultaneously; the first few minutes inSchoenfeld’s data shows “exercise solving”; thenwith extensive exploration, the problem solver isshowing “problem solving”37. Problem solving

versus exercise solving34 3 41

38. Problem solving 35 3 Option depending on time: these are useful in thatthey show the relationship between exercise solvingand problem solving; the relationship betweentextbook knowledge and knowledge structured forproblem solving; and between experience andmemorized solved problems

39.Exercise solving 36 3 Option depending on time

6-25

Topic Trans-par-encyno.

Time,min

Elapsedtime,min

Comment

40. Reflection & discuss 37 3 Option depending on time

41. Posttest andobjectives

38 4 45 MUST to close out unit

42. DISCOVERY 39 5 50 MUST to close out the unit

total 50

6-26

a108 ©copyright Donald R. Woods, April 2002

MPS 28 Group skills

Donald R Woods, McMaster University, Hamilton

CANADA

Jan., 2001; April 2002

Developing group skills starts with an awareness and application of the Seven Fundamental Rights of all persons:R be RespectedI to Inform others about your own opinionsG to have your own personal Goals and needsH to Have feelings and to express themT to have Trouble, make mistakes and be forgivenS to Select or choose whether you will meet another's expectationsand not to achieve your rights by violating the rights of others

In North America, four behaviours that can destroy an effective group are:contemptcriticismdefensivenesswithdrawal or "stonewalling"

Next, we should be aware of how to give and receive feedback. Suggestions about how to do this effectively are givenin Chapter 5 from "Problem-based Learning: how to gain the most from PBL".

6-27

MPS 28: Group skills © copyright, Donald R. Woods, 1999Group skills: application of problem solving and interpersonal skills to efficiently and effectively complete a task &develop good morale among the group members. Skill development:

1. Learn the characteristics of groups and how the characteristics evolve as the group matures. 2. Realize that each person has a unique contribution to the group process; that contribution should be encouraged,nurtured and used to the advantage of the group. 3. Learn how to give and receive feedback . 4. Recognize that trust is the most valued element in relationships and be able to list those behaviours that build trustand those that destroy it . 5. Acquire some skill at listening6. Acquire some skill in assessment7. Realize that both morale and task are important8. Realize that groups always perform better with a chairperson9. Accept that chairperson is not leadership. Leadership varies from person to person depending on the task.10. Accept that if a group is faltering, process skills are needed. Gradually you will develop skill in facilitating the groupprocess. 11. Through self awareness, begin to improve self confidence12. Begin to develop an environment of trust where risking is OK.13. Group skills are the first step toward developing team skills14. Be able to identify the different elements that make up “group norms” and recognize the importance of discussingthese early in the group process.

********************************Pretest:Group skills how aware are you of what you do in a group? Rate with an “x”

0 1 2 3 4 5 6 7 8 9 10Unaware Aware of Very awareI just Some I can describedo it The details of

how I do it

Skill: how skilled are you in contributing positively to the task and morale of a group? Rate with an “x”

0 1 2 3 4 5 6 7 8 9 10Poor Fair Good Very good Excellent

Comments:

_______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

MPS 28 Group skills

6-28

1.1 given a term listed under "concepts introduced", you should be able to give a word definition, list pertinent characteristics andcite an example.

2.1 given a problem, as a member of a group, you will participate in the task and morale components of the process such that yourparticipation will be judged by an observer to be "active" and to have more positive than negative contributions.

2.2 as above but your performance will be judged to be "most of these skills" for both task and morale by two independent observers.

2.3 given a problem, the group will complete the assigned task by the designated time and over 60% of the members of the groupwill choose to work together again.

2.4 given a group solving a problem, you will be able to observe the group, record plus an minus contributions toward both task andmorale components. You will be able to do this observing an individual client or the group as a whole. Your records will agree towithin 80% of the average records made by four other observers or with those of the tutor.

2.5 as a member of an observer group, you will be able to provide written suggestions on how the group can improve its task andmorale components. Eighty percent of your recommendations will agree with those of the tutor.

2.6 as a member of a group solving a problem, you will be able to self assess the performance of the group. Your assessment shouldagree within 80% of the assessment of outside assessors.

2.7 as in 2.6, you will be able to self-assess your personal contribution to the task and morale components of the group process. Yourassessment should agree within 80% of the assessment of an outside assessor.

Concepts introduced

Task and the positive and negative contributions toward task, Morale and the positive and negative contributions toward morale,FIRO-B, stages of group evolution, feedback form.

6-29

MPS 28: Group skills Where it fits in

Problem solving Knowledge

Team

Uses Prerequisites Group

Principles Interpersonal Assessment Skills

CommunicationAttitude Trust

Listening &self awareness Personal responding

Styleself confidence Differences

self selfactualization control

6-30

Research tells us:MNeed both

-Task oriented activities ( to get job done) & - Morale oriented activities (to keep the group happy, trustful & cohesive)

M Need Chairperson as facilitator- aware of what is & what should be going on in the group- aware of ideas flow- nudge process- remind of time- maintain decorum & sense of organization

M Leadership.. Changes from person to person as the group activity continues; whoever has the most pertinent expertiserelated to the task is the “momentary leader”.. Don’t confuse chairperson with leadership.

Target skills are given in Form 2800.

*****************************************************************************************

Reflections: ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

6-31

Evidence-based targets for group skills (reprinted from references 4,17) Form 2800 .

Evidence-based targets Progress toward internalizing these targets

20% 40% 60% 80% 100%

M Performance improves when we have goals. [18]

M Assessment must be related to the goals [19]

M Both Task (getting the job done) and Morale (feeling good about thegroup work and about how you have interacted with the other groupmembers) are important [19]

M Any group functions better with a chairperson [20]

M Chairperson and leadership are different; different people maybecome leaders at different times.[20]

M Group evolution tends to follow a pattern described as by suchdescriptors as “ forming, storming, norming and performing” [19,20].Schutz’s instrument FIRO-B [12,21] seems to provide reliable insight as tothe personal style of individuals towards other group members duringthree of these phases.

M We can list the roles needed in both Task and Morale to make aneffective group. [19]

M When each person has a clear idea of roles and group norms, thegroup functions better. [19]

M When groups are functioning effectively, about 70% of the time isspent on the task; 15% on morale building activities and 15% of taskprocess activities (how the problem solving process is going;summarizing ideas, guiding the process). [22]

M The products from groups or teams is improved when members havedifferent “styles” (in Jungian terminology some members are dominantS, and some, dominant N). The products from groups tend to be inferiorwhen all the members “think and behave alike”. [23,24, 19,20]

M The quality of decisions, product, task is improved if group membersoffer different perspectives, disagree and seem to introduce conflict intothe process. The trick is to manage the apparent conflict well. [19,23,24]

M The characteristics of “ meetings of individuals,” “effective groups”and “teams” fall on a spectrum with sufficient differences that it isuseful to differentiate based on those characteristics

M In a decision-making mode, after 20 minutes of discussion on anyone topic, few new ideas are presented and repetition of previouslystated ideas occurs. [25]

Feedback about the group work. Form 2802

Observation Form 2801

6-32

Observer _____________________- Client _____________________ Case _____________

Task Group members

Task processobserver;(problem solvingprocess)

Orients group, monitors, summarizes, seeks direction,identifies phases +

Ignores phases, asks whatever wants, blocks, unaware ofcontributions _

Information orOpinion Giver(related to task)

Assertively gives information, makes suggestions +

Withholds information, silent, aggressive, passive _

Information oropinion Seeker(related to task)

Asks questions, checks comprehension +

Silent, refuses to ask questions _

EnergizerRisk taker

Enthusiastic, introduces a spark, novel ideas +

Follower, agrees, silent unsure _

Morale

Observer ofinterpersonalprocess

Sensitive to interpersonal dynamics, comments on what’sgoing on +

Ignores conflict, tension, hopes it’ll go away _

Giver of praiseand support

Warm, responsive, gives help, rewards +

Put downs, aggressive, self-centered, defensive, critical,contemptuous _

Interpersonalproblem solver:Seekerssolutions

Mediates, harmonizes, helps resolve conflicts +

Causes problems, seeks personal goals _

Energizertension relief

Jokes, laughs, shows satisfaction +

Withdraws, causes tension _

Strengths Areas to work on___________________________________ ____________________________________

___________________________________ ____________________________________________________________________________________________________________________________________________

6-33

Observation Form 2801

Observer _____________________- Client _____________________ Case _____________

Task Group members

Task processobserver;(problem solvingprocess)

Orients group, monitors, summarizes, seeks direction,identifies phases +

Ignores phases, asks whatever wants, blocks, unaware ofcontributions _

Information orOpinion Giver(related to task)

Assertively gives information, makes suggestions +

Withholds information, silent, aggressive, passive _

Information oropinion Seeker(related to task)

Asks questions, checks comprehension +

Silent, refuses to ask questions _

EnergizerRisk taker

Enthusiastic, introduces a spark, novel ideas +

Follower, agrees, silent unsure _

Morale

Observer ofinterpersonalprocess

Sensitive to interpersonal dynamics, comments on what’sgoing on +

Ignores conflict, tension, hopes it’ll go away _

Giver of praiseand support

Warm, responsive, gives help, rewards +

Put downs, aggressive, self-centered, defensive, critical,contemptuous _

Interpersonalproblem solver:Seekerssolutions

Mediates, harmonizes, helps resolve conflicts +

Causes problems, seeks personal goals _

Energizertension relief

Jokes, laughs, shows satisfaction +

Withdraws, causes tension _

Strengths Areas to work on___________________________________ ____________________________________

___________________________________ ____________________________________________________________________________________________________________________________________________

6-34

Case 1: The Fishing Trip (from C.E. Wales)

It was the first week in August when six friends set out on an overnight fishing trip in the Gulf of Mexico.Everything went well the first day - the sea was calm, they caught fish and later camped out on a lovely littleisland. However, during the night a very strong wind pulled the anchor free, drove the boat ashore and thepounding waves broke the propeller. Although there are oars it the boat, the motor was useless.

A quick review of the previous day’s journey showed that the group was about 80 km from the nearest inhabitedland. The small deserted island they were on had a few scrub trees and bushes but no fresh water. They knew fromtheir portable AM-FM radio that the weather would be hot and dry, with daytime temperatures expected to beover 35oC the rest of the week. They are all dressed in light clothing but each had a windbreaker for the coolevenings. They agreed that whatever happened they would stick together.The families back on shore expected the group to return from their trip that evening and would surely report themmissing when they didn’t show up. However, they realized that it might take time for someone to find thembecause they went out further than anyone might have expected.

Items available Rating

Each person has

a. one windbreaker

b. one poncho

c. one sleeping bag

d. one pair of sunglasses

The boat contains

e. a cooler with 2 bottles of pop/person plus some ice

f. one large flashlight

g. one first-aid kit

h. fishing equipment

i. matches, rope and a few tools

j. one compass mounted on the boat

k. two rear-view mirrors which can be removed from the boat

l. one official navigational map of the Gulf area where you are

m. one salt shaker (full)

n. one bottle of liquor

6-35

Case 2: Crash land in the Canadian subarctic

You and five other passengers are in a plane flying toward Schefferville in the subarctic when you crash-land 100km away from Schefferville. You are on dry land yet there are at least nine water crossings on a direct linebetween you and Schefferville. The only means of navigation you have is a small, difficult-to-read map althoughyou have a compass. The immediate area is covered with small evergreen trees and the terrain is hilly and rockywith moss covering the perma frost. The temperature today is 2 0C and the weather conditions are for colder airin the next week. The next airplane expected is in 3 days. There is a 800 m high hill about 2 km away, and thismight put you in the line of sight with the Schefferville airport. The prevailing winds are 24 to 28 km/h. Thereis an abandoned mine 5 km away but off the line of sight of an airplane going to or from the airport. The planeis wrecked but before it burned up you are able to rescue the following 15 items. You all decide to stay togetherwhatever you do. Rate the items of relative importance to you with 1 meaning the most important.

Rating

a. 13 wood matches in a metal screw top waterproof container.

b. a fifth of Bacardi rum (151 proof) in a glass bottle

c. three pairs of snowshoes

d. one sleeping bag per person (arctic type, down-filled with liner)

e. a gallon can of maple syrup

f. a magnetic compass

g. a 7m x 7m piece of heavy duty canvass

h. an operating, 4 battery flashlight

i. safety razor kit with a mirror

j. a hand axe

k. a large bottler of water purification tablets

l. 80 m of 1 cm diameter braided nylon rope, 25 kg test.

m. a book entitled “Northern Star Navigation”

n. a wind-up alarm clock

o. one aircraft inner tube for a 35 cm diameter wheel; punctured.

6-36

Case 3: NASA Problem

You are a member of a space crew originally scheduled to rendezvous with a mother ship on the lighted surface of the moon.Due to mechanical difficulties, however, your ship was forced to land at a spot 350 km from the rendezvous point. Duringlanding, much of the equipment aboard was damaged, and since survival depends on reaching the mother ship, the mostcritical items available must be chosen for the 350 km trip. Below are listed the 15 items left intact and undamaged afterlanding. Your task is to rank order them in terms of their importance to your crew in allowing them to reach the rendezvouspoint. The most important is rated #1.

Items available Rating

a. box of matches

b. food concentrate

c. 15 m of nylon rope

d. parachute silk

e. portable heating unit

f. two 0.45 calibre pistols

g. one case of dehydrated milk

h. two 50 km tanks of oxygen

i. stellar map of the moon’s constellation

j. life raft

k. magnetic compass

l. 10 L of water

m. signal flares

n. First aid kit containing injection needles

o. solar powered FM receiver transmitter

6-37

Case 4: The Case of the Reluctant Fluidized bed (J. Sweetman, McMaster)

The fluidized bed is used to dry damp solids powder. The feed contain 5% water. The feed is a potash muriated from fourlarge centrifuges. The potash enters the bed from a feed screw. A knife gate is used to maintain the bed height in the fluidizedbed.

Fresh air is brought in from outside, heated in a natural gas burner and distributed through tuyeres at the bottom to fluidizethe bed. After about one week of operation, the operator claims “even though the air flow is at the desired level, the flowstarts to decrease, the bed collapses and the whole unit shuts down. We then have to dig it out and start all over again. “ Thisis intolerable. Fix it.

**************************************

Case 5: Terry sleuth and the case of the recirculating blower.

Harold whipped through the last calculations of the power required for the gas recirculation blower on the catalytic reactor.“Let’s see, that’s finished - calculate the density of the gas via the ideal gas law based on the temperature, pressure andcomposition of the recirculating gas, multiply the actual volumetric flow for the recirculation conditions to get the ideal powerrequired. A reasonable value....Oh, Hi, Terry.. I’m just finishing up the design calculations for the new reactor.”Terry Sleuth looked over Harold’s shoulder. “That’s the recirculating system operating predominantly hydrogen beingrecirculated at 150 oC and 12 kPa pressure gauge” “Right! replied Harold. “I was just saying that I adjusted my ideal powercalculation by dividing it by a reasonable value for the motor-blower efficiency. I used 60%.” Terry Sleuth perused thecalculations done so far. To Harold’s surprise, the Sleuth cautioned “I don’t think we’d better install the motor based on yourcalculations.” ************************************************

Case 6: The reluctant Pump

A centrifugal pump has been installed to pump water from a holding basin through a filter and on into the process. We oftenbypass the filter and just pump out of the basin to an atmospheric drain. The system head curve (and thus the designconditions) is 12.5 L/s against a total head of 10.5 metres. A pump has been selected to operate most efficiently at thiscondition. Everything works fine when the liquid is directed through the filter. However, when we bypass the filter and tryto pump out the basin we can only pump down to a water level of 3.3 metres below the centreline of the pump. This leaves1.2 metres of water in the bottom of the basin that can’t be pumped out.

***************************************

6-38

Form 2802Task: Problem defined, many issues and hypotheses explored, criteria listed and the issues prioritized.Refrained from early closure. Task carried out and looked back at the result to assess it. Groupagreement as to goals. Process was active with monitoring. Completed task on time. Group appliedsuccessive approximation and optimum sloppiness. Group avoided contributing excessive information. None of Few of these Most features All of thesethese behaviours but demonstrated behavioursbehaviours major omissions

G G G G G G G1 2 3 4 5 6 7

Morale: Group relaxed; enjoyed working together. They gave emotional support to each other and wereable to express disagreement or disappointment directly. Seven fundamental rights preserved. Membersare enthusiastic and involved. None of Few of these Most features All of thesethese behaviours but demonstrated behavioursbehaviours major omissions

G G G G G G G1 2 3 4 5 6 7

Individual Contribution to Task and Morale

____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Group Strengths Group Areas to work on_____________________________________ ________________________________ ____________________________________ _____________________________________________________________________________________________________________________________________________ from D.R. Woods (1995)

6-39

Reflection:__________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

6-40

DISCOVERY

Activity Discovered So what? application

6-41

MPS 28: Group skills: Example assessment tasks:

1. Watch the videotape of a group meeting and use the feedback form given in Table 1 to assess the performance of each group memberand the group as a whole. Note the five strengths of the group and the two areas to work on.

2. The following are the results of FIRO-B for Andre.

4 7 4

8 3 7

a. Assume that this is a reasonable description of Andre; how is Andre likely to behave in a group?

b. Based on your own FIRO-B, compare your responses with those of Andre and suggest how you and Andre might interact from at leastfour different points-of-view: inclusion, control, affection, expressed toward others and wanted from others.

3. You have just joined a PBL group. They have found out that you have had some training in group process. They ask you to "tell themhow to improve their group skills." Summarize the key ideas about group process.

4. You have just joined a PBL group. They have found out that you have had some training in group process. They ask you to "give ussome training in group skills." Prepare a workshop. Estimate the time required, outline the activities, highlight the main points you wantthem to focus on.

6-42

Teacher action: Reflections about this as a learning activity to be used with your students____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

already would might not mydo this work work style

Writing reflections O O O OUsing the group observation form O O O OAssessing group success form O O O OUsing the learning objectives O O O OUsing the DISCOVERY form to summarize O O O O

Other___________________________________________________ O O O O

7-1

7. How to manage the transition to PBL? copyright ©2002 Donald R. WoodsDonald R. WoodsMcMaster University, Hamilton, Canada

Case 5: The letter to the Dean.

You have been five years in a tenure-track position. You have 15 refereed publications with 4 others in the mill. Youhave 2 Master’s students and 2 PhD students plus a large series of grants to keep your research program viable.

Your teaching has received “fair” ratings from the students in the past. last year the Chair suggested that you mightdevote a little time to improving your teaching. Indeed, the Department sent you on two workshops: one oncooperative learning and the other on PBL. These were very motivating. You saw that you could introduce PBL topart of your course 393. You added cooperative learning to each tutorial and you took a four week section of thecourse and converted it to PBL. It is now six weeks into the term. The Dean invites you to see him and gives you acopy of the following letter: ___________________________________________________________Oct 15

Dear Dean Habizz

We the undersigned represent 82% of the students in Course 393. The professor is incompetent! The professor is notdoing the job. The professor should be fired. We are paying big bucks to come to this Polytechnic. We demand thatyou replace this teacher with one who knows how to teach.

H. AndreA. SabinaR. GottzF. GoammR. JonesK. Sigvaler K. ArmstrongA. SiggsP. Scripps

B. CarruthersZ. GafterH. DominikP. SellersS. HavamanG. DykknoD. Chang43 more Signatures

******************************************Activity: In small group of 5 or 6, with chair _______________________________; reporter ________________________, brainstorm the issues this case raises. Identify what you know already.Identify what you need to learn.

Prioritize the issues: criterion: what do you want to gain from this workshop in the context of the issues raised._________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

7-2

Feedback about the group work. Form 2802

Task: Problem defined, many issues and hypotheses explored, criteria listed and the issues prioritized. Refrainedfrom early closure. Task carried out and looked back at the result to assess it. Group agreement as to goals. Processwas active with monitoring. Completed task on time. The accuracy in the group’s answer matched the time available.Group avoided contributing excessive information. None of Few of these Most features All of thesethese behaviours but demonstrated behavioursbehaviours major omissions

G G G G G G G1 2 3 4 5 6 7

Morale: Group relaxed; enjoyed working together. They gave emotional support to each other and were able toexpress disagreement or disappointment directly. Seven fundamental rights preserved. Members are enthusiastic andinvolved. None of Few of these Most features All of thesethese behaviours but demonstrated behavioursbehaviours major omissions

G G G G G G G1 2 3 4 5 6 7

Individual Contribution to Task and Morale

____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Group Strengths Group Areas to work on_____________________________________ ________________________________ ____________________________________ _____________________________________________________________________________________________________________________________________________

from D.R. Woods (1995)

7-3

Feedback from the Goals Meeting

Case 5:IssuesNumber identified: 1 2 3 4 5 6 7 >7

Agreement with tutor <50% 50% 60% 70% 80% 90% 100%

Knowledge/skills to be learned Consensus among group little some a lot complete

Agreement with tutor's list little some a lot complete

Learning objectivesQuality poor fair OK good excellent

LearningQuality of questions asked during the teach session none some astute excellent

Willingness to continue to contribute <50% 50% 60% 70% 80% 90% 100%

Your AttitudePerry shift 2 3 3.5 4 4.5 5******************************************************************************

Reflections:______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________-

7-4

Managing the transition to PBL

PBL can be an extremely effective learning environment. Nine issues related to the fundamentalunderpinnings of PBL make it effective. If these nine issues are not handled well, then thetransition to PBL will be challenging!

1. Teachers ready? 2. Students ready? 3. Appropriate version of PBL 4. PBL well-designed? Know the reason why you select PBL and include keys to improve student learning: active, coop, prompt feedback, expect success, ownership5. Students accountable? 6. Lifelong learning developed? 7. Problems mirror professional practise?8. Process skills applied, extended?9. Assessment done astutely?

1. Teachers ready?

The potential issues are:

Role in conventional Role in self directed students learning

source of “knowledge”“Sage on the stage”

coach of learning process“ Guide on the side”cheerleader, mentor

Perceived control in conventional programs Perceived control when empower students

over student behaviour: make individual behaviour visible; provide “norms”

over standards of student performance: I set and markthe exams

monitor and assess the self-assessment process used bystudents

over content: “cover material” “covering material” is not student learning;

Conventional Self-directed learning

Teacher has intrinsic motivation of synthesizing newperspectives of the new knowledge

Teacher’s joy comes from student accomplishment

Power of all eyes on you; students writing down yourevery word; you are centre stage

You are coach, in the wings; encouraging andapplauding

Student feedback about superstar teachers; awards more challenging for students to give you credit.

7-5

So what might you do?

1. Select one issue in self-directedness with which you are comfortable.2. Check for consistency with your attitude: use the inventory MRIQ.3. Set yourself up for success. Whatever approach to PBL you take, give your interventions thebest chance to succeed. Whenever you try something new, most students resist.

- Students prefer the familiar to something new. They know the “lecture system” and how tomake it work for them to get the highest marks. If we change the familiar “system”, mostresist the change (Benvenuto, 1999). When they experience change, students often followthe eight-step grieving process of shock, denial, strong emotion, resistance, acceptance,struggle, better understanding and integration (Woods et al.,2000a; Woods, 1994). To helpthem through the process, run a workshop. - Start small. Call it a “pilot project”. Start where you are. Don’t wait for others.- Explain why you are making the change. - Help students see their personal benefits of the new approach in the short term and in thelong term. - Explain your role. Students may believe that they are paying for teachers to “lecture.”(Benvenuto, 1999). Use Perry’s inventory (Perry, 1970, described in Woods, 1994; Woodset al., 2000a) to help students understand their perceptions; explain your role in terms of thePerry model.- Monitor the program frequently. Use ombudspersons or one minute papers (described inFelder et al., 2000) and adjust in response to their feedback.- Be flexible. If this particular class is vehemently opposed to the new approach, exploreoptions to achieve your goals and theirs. Perhaps scale down the intervention. Gather data onhow to make the intervention more effective the next time you try it. - Bring in success stories. Invite recruiters or students, from other programs where theintervention has been effective, to give testimonials at the start of your course. - Help students cope with the upheaval when old habits are identified and changed. This isparticularly true for problem solving. For example, part way through a program to developproblem solving, students complained that “focussing on the process of problem solving hasmeant that they could no longer solve problems.” We used the following analogy to helpthem through this frustration. Consider that you are a reasonably good tennis player. You go to an expert to improve yourgame. The expert takes your game apart. As you relearn how to do each part, your game isnot what it used to be. Be patient. Gradually you improve and surpass your past skills.

4. Realize you will probably follow the grieving process as you make the shift. (The “Grievingprocess” is given in Chapter 1 of Woods, “How to gain the most from PBL” and in Workshop#4).

2. Are the students ready? Some issues: a) Are students ready? do they know what it is? What is PBL? Show videotape of PBL: NewMexico, MPS Run an Example Workshop:

7-6

b). Are students ready? know what is expected? Manage the change? awareness of expectedroles of student teacher: traditional versus self-directed. Use of Perry inventory. grieving processif there is a mis-match.Example Workshop on Managing change: This is a double winner because it 1. helps them (andyou) with PBL or any other change; and 2. empowers students with skills to manage change inthe future professional career and life.

c). Are students ready? maturity? Do the students have the self confidence and maturity tohandle the new empowerment accountability? Bandura (1982) studied self confidence, what it isand how you develop it. He found the following differences between self confident persons andthose who had little self confidence. Self confident Low self confidence

Prepare AvoidFocus on the “tough stuff” Hope it doesn’t happenPersevere Give upManage stress Stress debilitatesFocus first on the familiar Focus first on the “strange”Positive self talk Negative self talk

He found that an environment to develop self confidence provides rewards for performance;highlights the positives achieved because those with low self-confidence will tend to becomesdespondent and self-devalue. The presence of highly competent individuals undermines theeffective use of routine skills by others.

To develop self confidence Bandura (1982) recommended enactive mastery via workshops >observe models perform the task > emotive strategy to tell yourself you can do it > cognitivestrategy. Stress is related to self confidence. We have found that helping individuals identifytheir “uniqueness” is very helpful. Training students to self assess is another activity thatincrease self confidence. In general, the stages in developing self confidence might follow thefollowing pattern: ! Self awareness.! Aware of what others do.! Acceptance of self and others.! Cognitive and attitudinal shifts related to stress management: positive self talk, learn to sayNO; worry about what have control over; sense of being in control; willingness to risk! Attitude about assessment. Receiving ± and responding to feedback. (Perry level 5)! Aware of models of behavior: observe, and become aware of the target skills.! Successfully complete achievable goals posed by others with +ve feedback from others (as isdone in workshops).! Set achievable goals and self assess.Some ideas on helping students move toward self confidence:Awareness: know what you do1. Awareness of what I do? Talk Aloud Pairs Problem Solving (TAPPS), reflect. (Workshop 2)

7-7

2. Aware of others? TAPPS, reflect, peer feedback3. Receive feedback from trained observers. Know goals in performance 4. Be aware of target behaviours: communication, listening, giving and receiving feedback,problem solving, group work, change management, self assessment. 5. Use diagnostics to know unique style where there is no right / wrong.

Example diagnostics

Indicators of PS Heppner PSIRotter locus of controlKolb-Basadur strategy applicationKAI-Basadur for creativityBillings-MoosDecision-making styleRiskKellner-SheffieldHolmes-Rhae

Indicators of learning style & writing exams Visualizer-verbalizer-symbolLASQ deep versus surfaceJungian typology, MBTIKolb learning cyclePerryAlpert Haber Anxiety Achievement

self directednessPerry

Group and interpersonalFIRO-BJohnson’s conflict resolutionJungianTrust

Example: Jungian typology, MBTI: your preferred style in four dimensions. This set ofdimensions suggests approaches to problem solving, potential conflict with others who havedifferent style, your style in responding to conflict, how you study for exams, taking lecturenotes, your approach to trouble shooting & decisions, the challenge to teaching others in cooplearning or PBL.

7-8

Example: use for teamwork: For the members of your team list values of Jungian typology andKirton Inventory. Identify: blind spots for your team areas where there might be apparentconflict

M S (N) T (F) I (E) P(J) KAI

you

Mike

Sue

Marie

Jose

Example: teaching in PBL: For the members of your team list values for Perry, LASQ, JungianS, Jungian T . For each person, identify - their learning expectations, - blind stops for identifyinglearning issues, - test question preparation or concept map. Critique a “teach handout”

Perry Strat. Rote Meaning S ST,NT,SF,NF

you

6. Use self-assessment, reflections, personal enrichment projects.

7-9

d). Are students ready? skills Problem solving? team work? self-assessment? Run a series of workshops: (Workshops 2,4)

Define skillRationalize

Put in contextPretest

Goals, criteria, evidenceRoute ahead

Activity <-> FeedbackTarget skills/ behavioursActivity <-> Feedback

PostestSummary DISCOVERY

3. Appropriate version of PBL Select PBL form for YOU and your purpose

1. Chickering & Gamson +Improve learning:1. active2. cooperative3. quality of teach-learn interaction4. prompt feedback5. time on task6. environment expects success7. personal learn style+8. assessment by students9. published goals/criteria10. attention span = 20 min11. wait time before answering 20 s12. knowledge/skills/attitude

2. Ramsden & EntwistleDeep learning:1. good teaching2. openness to students3. freedom to choose

4. clear goals & standards5. vocational relevance6. social climate

7. (-workload)8. (-use of formal lectures)

Select a version of PBL to improve learning & develop deep learning.

PBL: pose problem 1st. Motivates, knowledge structure: Many options. challenge: problemsolving skill, cues.

Self-directed: students own. LLL skills. Many options. challenge: accountable.

Self assessed: students create & do assessment. Many options. challenge: you monitor process.

Small group: active, prompt, coop, learning styles. Many options.challenge: make it do this.

7-10

SG,SD,SA interdependent PBL:

4. Is the version of PBL used well-designed? Then.. it’s going to create knowledge structures that are cued for future professional recall;improve learning; promote deep learning; develop lifelong learning skills, and extend anddevelop “process skills”.

5. Are the students accountable? M Make student performance visible;M Expect completion of forms relating to contributionsM Assessment journal writingM Use self assessment

Work together. Teach <-> learnMonitor

6. Are lifelong learning skills developed?

This is how I view it: “Lifelong learning is assuming responsibility for identifying what one needs to know, creating learning goalsand criteria, identifying pertinent and accessible resources, engaging in learning the knowledge especiallythe difficult parts, integrating the new knowledge with past knowledge and experience, using the knowledgeto solve the need and assessing the degree of comprehension. One will use people and peers asresources.”

dependent -> independent ->interdependent learner Treat the development of this skill the same as any other:explicit goalshttp://www.chemeng.mcmaster.ca/innov1.htm click on PBL and download Chapter D ofResources. Criteria; gather evidence, present an assessment journal.

7. Do the problems mirror professional practise?How to create a problem1.contains “cues”; correct learning

objectives. 2. triggers pertinent previous knowledge.3. complexity OK.4. requires integration across disciplines

& knowledge, skills, attitudes.5. allows openness.6. motivational; similar to future professional practice.7. promotes student activity.

7-11

8. identifies context;- concrete scenario;- clearly identifies expected task.

Great advantage of PBL is that the students learn knowledge in the context of future need. Thisway the new knowledge is “cued” to aid recall. Pick representative, realistic professional problems.

8. Are the process skills applied, extended?

Ideally students have “good” skills in problem solving,group work,self assessment.

We want to explicitly extend and further develop the skills as they use them in PBL.

Ideas: - ask them to write reflections,- elaboration at the end of the case,- assess team work after each meeting,- build on self and peer assessment skills.

9. Is the assessment done astutely?

A definition of Assessment is..“a judgement based on the degree to which goals have been achieved based on measurablecriteria and pertinent evidence.”

Five principles:1. Based on performance... not personalities.2. Based on evidence... not feelings.3. Done for a purpose with clear performance

conditions.4. Done in the context of published goals,

measurable criteriaagreed-upon forms of evidence.

5. Based on multidimensional evidence.

Enrichment:1. Please help me understand how best to help you2. MRIQ

7-12

3. Jungian typology and scoring4. Example data for 2nd year students in engineering program

References:Woods, D.R. “Participation is more than attendance,” J. of Engineering Education, 85, 3, 177-181, 1996.

Dolmans, D. et al., “Why aren’t they working?” Chapter 18 in “Problem-based Learning: casestudies, experience and practice,” P. Schwartz, S. Mennin and G. Webb, Kogan Page, 2001

Kaufman, D.M., and K. V. Mann, “I don’t want to be a groupie,” Chapter 19 in “Problem-basedLearning: case studies, experience and practice,” P. Schwartz, S. Mennin and G. Webb, KoganPage, 2001

Woods, D.R. et al., “Tutored versus tutorless groups in PBL,” Am. J. Pharm. Ed., 60, 231-238,1996

Woods, D.R. “They just don’t pull their weight,” Chapter 22 in “Problem-based Learning: casestudies, experience and practice,” P. Schwartz, S. Mennin and G. Webb, Kogan Page, 2001 highlights the effective use of peer and self assessment to improve the accountability within thegroup.

7-13

Helping students gain the most from their PBL experience

Donald R. Woods, Chemical Engineering Department, McMaster University, Hamilton, Canada

Chapter submitted to the book “The PBL book: the transformation process to PBL” Erik deGraaf and Anette Kolmos, eds. Nov 9, 2005

We have been using small group, self directed, selfassessed, interdependent problem-based learning in oneof our courses since 1982, and as part of two of ourcourses since 1990. We chose to use PBL because itimproves student’s learning, it is one of the best waysthat we know of to develop lifelong learning skills, andit gives us an opportunity to develop student’s skill ingroup process, being a chairperson, communication,self assessment, learning/teaching and to develop theirself confidence.

In this chapter we 1. elaborate on the form of PBL we use2. we describe the context3. we describe the skill development activities that wethink are required before students experience PBL4. we describe the workshops and activities we useimmediately before PBL5. we show evidence of student performance related tothese activities.

1. Variations on PBL and the form we use PBL seems to be the flavour of the month. Many claimto be using this popular educational innovation. Yet,closer inspection shows that their approach iscompletely different from the one reported in thisChapter. Not that there is anything wrong with that.Merely, we should have clear communication.

Most engineering faculty use problems extensively inthe program. Students solve homework problems, solveproblems on tests and exams and usually solve anumber of projects (that may or may not involvedesign). So it is not surprising that many claim that theyare already using PBL because they are using andassigning problems. As I see it there are different waysin which problems can be used that range from the twoextremes: problem based synthesis to problem-basedlearning. In problem-based synthesis, students learnthe required knowledge before they are given theproblem, and the students are asked to synthesizepreviously-learned knowledge to solve the problem.Typically, students receive lectures on the subject andthen synthesize that knowledge in the project/problem.A frequent response from an instructor who uses thisapproach is “How can the students solve problems if Ihaven’t given them lectures first?” In problem-basedlearning, the students are given a problem and thestudents identify what they need to learn and then thestudents learn and teach each other the new knowledgeneeded to solve the problem. They use the problem todrive the learning; new knowledge is learned after theproblem is posed. This poses a problem in that, for

example, four of the programs described in a recentbook on PBL are what I would call problem-basedsynthesis.

In this Chapter the focus is on small group, self-directed, self assessed, interdependent problem-basedlearning.

2. The McMaster engineering context

Here we describe the PBL experience in the senior yearwhere PBL is part of a required course in an otherwiseconventional program. The class size varies from 45 -59 students with three, 50 min classes per week. PBL isused to replace the four weeks of lectures (12 hours) weused to use for the topic “Engineering Economics”.Instead, in the 12 hours of class time four shortproblems are presented (one each week) in the contextof following topics: 1: interest/depreciation/financialstatements/risk/ taxation; 2: financial attractiveness; 3:capital cost estimation; and 4: operating cost estimation.(For the timing to work out, we needed five and a halfweeks). For each cycle there are three, in-class studentmeetings: a goals meeting, a teach meeting and anassess meeting. Students are assigned to groups of fiveor six. They remain with the same group for all the PBLactivities. A different student is assigned to chair eachmeeting. The chair is expected to circulate an agendaahead of time; to run the meeting; and, based on thefeedback about performance, submit a self-assessmentof his/her performance as chair.

In the Goals meeting the students interact with theproblem and list what they know and what they need toknow to solve the problem. They create about 6 to 8learning objectives and prioritize these. To ensure thatthe objectives meet the needs of the program, theinstructor provides written validation or, if theobjectives are inadequate, he/she provides hints andnew perspectives that the students can use to reconsiderthe problem. Once each group has approved learningobjectives the group members identify resources andcontract with each other as to the topics that each willresearch and teach. In the Teach meeting, the students, in turn, hand outteach notes, interact with their peers and teach eachother the new knowledge. Each receives immediatefeedback as to the quality of the teach. In the Assess meeting, each prepares a good, 10 minutetest question (and answer) in a subject area differentfrom their area of teaching. The group has eight minutes

7-14

to select “the best” test question. That question is given(via the instructor) to another group to solve so that allgroups have a question to answer. Thus, for example, aquestion from group A might be given to group J. Eachgroup has 30 minutes to write an answer. Then, aperson to mark the answer (from the group that posedthe problem) goes to the group identified as working ontheir problem; the marker marks and responds toquestions from the respondee group answering thequestion. Thus, group A would send a marker to groupJ. At the end of the session, the instructor collects allthe posed questions and answers from the posers andthe group answers from the respondees and marks thequality of each. In some instances, the instructor adjuststhe marks given to the respondees.

One week after every meeting, the chairperson submitsa self assessment report on his/her acquisition of skillsas chairperson. Each person also submits a selfassessment report on his/her acquisition of skills inlifelong learning. Their assessments are done in thecontext of published learning objectives, measurablecriteria and many different forms of evidence.

For nominative marks (term marks) each studentreceives marks, based on their self assessment report,on their skill as chairperson and as a lifelong learnerincluding the quality of the teach notes; groups receivemarks on the quality of the assess components. At theend of the course, there is a 3 hour written finalexamination created by both the instructor and thestudents. The final mark in the course is based on astudent contract. The student may contract to have theterm mark count as any percentage between 10 and90%. A default option is that the student receives thebest of a 40-60 split between the term and final exammark. Details are given elsewhere [Woods, 1997b;1996]

3. Prerequisite skill development activities beforestudents experience PBL

In our program, prerequisite skills that the students needfor effective PBL include skill in problem solving, selfassessment, group process and learning. This issupplied by about 80 hours of required workshops, viathe MPS program, in years 2 and 3 before they enter thePBL portions. Details given in Woods et al.1997;Woods et al., 1988, Woods and Sheardown, 2004.

4. Helping your students make the most from PBL

In addition to developing the required prerequisiteskills, we introduce the PBL activity by providingpublished learning objectives for both the lifelonglearning skills and the chairperson skills - skills that areacquired through the PBL activity while the subjectknowledge is being learned. Tables 1 and 2 list theseobjectives. We also provide progress charts so thatstudents can monitor the development of the targetskills. Table 3 is an example. We give some practicecompleting feedback forms about performance related

to these skills. Table 4 is the feedback form for teachingwithin the PBL context and Table 5 is the form forchairperson. We show a 23 minute videotape of studentsparticipating in the three meetings: goals, teach andassess. Introducing the learning objectives, discussingthe videotape, getting familiar with the feedback formsand clarifying expectations usually takes two hours ofclass time.

In addition, in Option #1, we encouraged them to createa learning contract, to have a “norms” meeting and touse inventories to explore learning preferences withineach group. However, we provided no class time nordid we have explicit activities. This was the option weused when we had between 6 and 8 autonomous groups.

We used option #2 when our class size increased to 10to 12 autonomous groups. In Option #2, we added fourhours of activities so that issues of learning contracts,group norms, learning preferences and how to gain themost from PBL were addressed explicitly. The detailsof these four are as follows:

1. Students completed the following inventories:Myers-Briggs/Jungian/Keirsey (Hogan and Champagne, 1979; Keirsey and Bates, 1984); KirtonKAI (Kirton, 1976), Lancaster approaches to studying(Ramsden, 1983; Woods et al., 2000) and Perry (Perry,1970; Woods, 1997a). A workshop was used to helpstudents see the implications and effects these resultsmight have on their performance. Within their groupsall entered the results of the inventories and discussedthe implications. Example data are given in Table 6.

2. The second activity was a one-hour lecture-interaction in which we rationalized the choice of PBLas a learning environment and described the underlyingprinciples of how case problems were created. Webasically addressed the question “What’s in it for you,the students?” We related how, traditionally, instructors publishlearning objectives for every course and how in PBL,instead of handing students the course learningobjectives, we pose professionally- significantproblems that are cued so that the students will generatelearning objectives equivalent to the instructors.Teachers monitor this process to ensure that each groupis learning the “right material”. The case problemshould generate about 6 to 7 learning objectives eachrequiring about one to two hours of “learning time” andone to two hours to create effective teach notes for atotal of about 2 to 4 hours of homework per groupmember. This provoked a lot of discussion and veryprobing questions. What seemed to be helpful to thestudents was a sense of direction behind the activity, theteacher’s validation of the learning objectives and theclarification about the time commitment.

3. The third explicit activity was a one-hour workshopon how to search for cues, generate learning objectives,how to research and teach and how to prepare effective

7-15

teach notes.

4. The fourth activity was one hour class time for thegroup norms meeting and the possible creation of groupcontract. Each group considered 17 issues that rangedfrom the method of making decisions to the level ofintervention when apparent conflict occurs to asking aperson to leave the group (Woods, 1994a). Threegroups created learning contracts during this time. Anexample is given in Table 7. Other groups decided tohave verbal agreements; one group initially decided ona verbal agreement and later created a formal writtencontract.

In summary, we feel that introducing the students toPBL is very important. This includes the publication oflearning objectives, practice with theassessment/feedback forms and some introduction toPBL as a learning environment. We used two options:Option #1 that required a total of two hours of in-classactivities, and Option #2, requiring six hours of in-classtime. Next we compare some dimensions about theimpact of these two options.

5. Impact on student performance.

The students entering the PBL activity showed thefollowing characteristics: - both groups had about the same overall grade pointaverage based on the previous courses that they hadcompleted- the students who received Option #1 had a slightlybetter score on the Perry inventory. This suggested thatthey, as a group, entered PBL with an attitude morecompatible with the PBL learning environment. Thosein Option # 1 entered the PBL activity with an averagePerry score of 3.6 with 14 % of the students having aPerry score < 3; those entering Option #2 had a slightlylower average Perry score of 3.3 with 22% of thestudents with a score below 3.

How did the students fare who received the differentintroductory experiences?

For the student performance, our focus was on 1) thequality of the subject knowledge learned, 2) thestudent’s attitudes about PBL as a learning environmentand 3) the quality of the interdependent teaching.

5.1 Quality of the knowledge learned

Some evidence related to the quality of the knowledgelearned is the marks on the test questions used in theassess meetings. For option#1, the average mark for thequestions posed in the assess sessions (and monitoredby the teacher) = 93%; for Option #2, 97%. Weconclude that there was a negligible difference betweenthe groups as far as the quality of knowledge learned.

5.2 The attitude about the learning environment

Some evidence that we used included the Perry attitudeafter the PBL sessions; the observations of the tutor ashe circulated among the groups, and the commentsmade in the individual’s self assessment journals. Thisinformation is summarized in Table 8. For both Option#1 and Option #2, 25% of the students left the PBLactivity with Perry scores < 4. These students hadtrouble shifting their attitude about their role (relative tothe role of the instructor), about assessment, and aboutknowledge. Considering the incoming values of thePerry inventory, more students shifted their attitudewhen they received Option #2. The tutor’s comments were that all of the groups thatreceived Option #2, except 5 and 10, were very stronggroups. Furthermore those groups that received Option#1 tended to struggle and were not as successful insupporting each other or in trying to make PBL workeffectively (as those students who received Option #2).The tutor comments were based mainly on the analysisof the self assessment journals.

The students who received Option #2, by and large,embraced the PBL experience and made it work forthem. Those who received Option #1 tended to strugglewith PBL - some tried to make it work; some didn’teven though they entered PBL with slightly bettervalues of Perry. The Perry inventory is helpful but thissuggests that it should not be the only factor considered.

5.3 Quality of the interdependent learning

To assess the quality of the interdependent learning,evidence includes the assessments of the quality of theteaching from the form 3601 (shown in Table 4) and comments made in the individual’s self assessmentjournals. Each receives feedback on form 3601 fromevery person in his/her group every time he/she teaches.Therefore, in a group of five and with three cycles ofPBL, each person has15 ratings of the quality ofknowledge brought to the group. This is averaged foreach person and averaged over all the people in thegroup to give a group average. This average is reportedin Table 8.

For the students who received Option #1, the ratings ofthe quality of the teaching were relatively low on alldimensions for all groups. Only one group had a ratingabove 7.2/8 on the quality of knowledge brought to thegroup and no group had ratings over 7.2 for the qualityof the teaching or the small amount of learning on one’sown that might be needed. Even in group 6 where allstudents seemed committed to PBL and all had a highPerry rating, the quality of the teaching was higher thanin other groups in that cohort but it was not outstanding.

For Option #2, all eleven groups had ratings above7.2/8 for all dimensions.

7-16

The comments in the individual self assessment journalsare too extensive to report here. Rather, somerepresentative comments from the journals andcomments by the tutor are given. For Option #1:Here are representative comments from student’sjournals:Learning objectives were the hardest to create. I wasfrustrated because others didn't prepare. Enjoyed this type of learning environment"PBL is not as time saving as I thought." "I'm glad that Idon't have to listen to lectures on this topic. I wasstimulated by working with others to solve a problemunder time constraints." "PBL showed me my strengths and weaknesses incommunication and expressing myself.""Members who previously seemed disinterested didgradually become involved." "Probably the mostenjoyable part of PBL was designing the test questionsduring the feedback session. This gave us anopportunity to explore the interrelationships amongdifferent topics."The tutor’s comments were:For group 1, the third PBL cycle was the tough one withnot all doing their fair share. The group did not confrontthis directly. Wide range of Perry attitude. Althoughlearning style information was given to everyone aboutall the group members, these data were not formallyreported or used by group. The group was too generousin ratings. One tried but preferred lecture; two didn’t try.

For group 2: the group got off to a tough start withabout 3 members uncertain if this would work;somewhere between case 3 and case 4 they seemed toget their act together. However, two were ineffective inteaching others. They really couldn't get their teachskills in place.

For option #2,Our group has continued to work at a professional levelfor both task and morale. Chair skills are improvingwith each meeting. All group members continue to put astrong effort into all PBL activities. PBL is really suitable for me and our group. Our groupis attending, thinking, friendly, willing to contribute andenthusiastic.

Overall the PBL meetings are working out very well formy group and for me. They are helping me assess theway I operate within a group and my learning style andability. They also are a good introduction to what it islike to run a meeting with success and confidence. ThisPBL was very successful overall. I personally was ableto make strides in my attitude and in the bettercompletion of the objectives.

Tutor’s comments: All groups worked well althoughgroup 5 struggled and one person in group 10 felt thattwo were not doing his/her share. In all groups studentscommented on their development of confidence, respectfor others, and skill in teaching and in the group

process.

In summary, the quality of the interdependent learningdemonstrated by those who received Option #2 wassuperior to the performance of the cohort of studentswho received Option #1.

6. Summary

We clarified our definition of PBL, described thecontext in which we use autonomous groups of 5 to 6students in a senior class of 45 to 59 students. Webriefly described the workshops in the second and thirdyear to develop the background skills needed forautonomous group PBL.

Details were given of various activities teachers can useto introduce students to PBL. The short two-hourversion produced inferior results compared with thelonger six-hour version as measured by the attitudetoward PBL and the quality of interdependent learning.

Issues other than the Perry inventory are important toaddress. These include identifying and exploitingpersonal preferences within the groups, training in howto teach, rationalizing PBL as a learning environment ofchoice and elaborating on the checks and balances tomaintain standards and make PBL effective. Providingin-class time for a group norms meeting and the use ofself assessment journals are two other importantfeatures we have found useful to help students gain themost from PBL.

Learning objectives and feedback/assessment formswere given.

7. Acknowledgements

I am pleased to acknowledge the students in ourprogram who willingly shared their experience with us.

8. References:

Hogan, C., and D.W. Champagne (1979) “PersonalStyle Inventory” personal communication Craig Hogan.

Keirsey, D and Marilyn Bates (1984) “PleaseUnderstand Me,” Prometheus Nemesis Books, Del Mar,Ca,

Kirton, M. (1976) “Adaptors and Innovators: adescription and measure,” J. Applied Psychology, 61,no. 5, 622 - 629

Perry, W.G. jr (1970) “Forms of Intellectual and EthicalDevelopment in the college years: a scheme,” HoltRinehart and Winston, New York.

Ramsden, P. (1983) “The Lancaster Approaches toStudying and Course Perception’s Questionnaire:Lecturer’s Handbook”, Educational Methods Unit,Oxford Polytechnic

7-17

Woods, D.R. et al. (1988) ASelf assessment in thecontext of the McMaster Problem Solving Program,Assessment and Evaluation in Higher Education, 13,no. 2, p 107 - 127.

Woods, D.R. (1995) PBL: resources to gain the mostfrom PBL”, Woods, Waterdown, ON Canada

Woods, D.R. (1996) AProblem-based Learning forLarge Classes in Chemical Engineering@, Chapter 10 inABringing Problem-based Learning to HigherEducation: theory and practice,@ L.A. Wilkerson andW.H. Gijselaers, ed., Jossey Bass, San Francisco

Woods, D.R. (1997a) “PBL: how to gain the most fromPBL” 2nd ed, Woods, Waterdown, ON, Canada

Woods, D.R. (1997b) AIssues in implementation in anotherwise conventional program@, Chapter 17 in ATheChallenge of Problem-based Learning@ D. Boud andG.H Feletti, eds., Kogan Page

Woods, D.R. et al. (1997) ADeveloping problem solvingskills: The McMaster Problem Solving program@, J.Engng. Ed, 86, no 2, p 75 to 93.

Woods, D.R., A.N. Hrymak and H.M Wright (2000)“Approaches to Studying and Learning Environments inProblem-based versus lecture-based learning,” ASEEConference Proceedings, Session 2213, St Louis, MO.

Woods, D.R. and H.D. Sheardown (2004) AAnApproach to developing student=s skill in selfassessment@ paper 832, Proceedings of the 2004 ASEEAnnual Conference and Exposition, Salt Lake City.

7-18

Table 1. MPS 36: Lifelong learning in the context of small group, self assessed, self directed, Problem-basedLearning PBL © copyright, Donald R. Woods, 1998Lifelong learning in the context of small group, self assessed, self directed, Problem-based Learning PBL is asfollows: given a professionally-significant problem you will identify what you know already to solve the problem,what new information you need to know, contract with other members of the group that individuals will learndifferent aspects of the new information and return to the group and teach all members of the group the newinformation. As a group you will use all the old and newly-learned knowledge to solve the problem. The group willelaborate on other ways to use the new knowledge. Skill development:1. Ability to work effectively in a group.2. Can value the unique contributions each person brings to the group. 3. Able to deal with unexpected frustrations.4. Able to adjust to empowerment and accountability. 5. Can identify the knowledge needed to solve a problem.6. Able to teach one another. 7. Able to manage stress and time. 8. More skill in giving and receiving feedback. 9. Through self awareness, to improve self confidence.10. Skill in self and peer assessment. 11. Develop trust where risking is OK.12. Building on skill in information literacy and individual “research projects” and moving toward reactive lifelonglearning.

********************************Pretest:Awareness: how aware are you of what you do when you do this ability? Rate with an “x”

0 1 2 3 4 5 6 7 8 9 10Unaware Aware of Very awareI just Some I can describedo it The details of

how I do it

Skill: how skilled are you in doing this activity? Rate with an “x”

0 1 2 3 4 5 6 7 8 9 10Poor Fair Good Very good Excellent

Comments:

_______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________******************************************MPS 36 Self-directed learning or lifelong learning in the context of small group, self-directed, self assessed PBL

1.1 given a term listed under "concepts introduced", you should be able to give a word definition, list pertinent characteristics andcite an example.

2.1 Given a problem, you will ask questions, explore issues so that within 30 min, as a group of five, you will be able to identify allof the five to six major issues, and these shall agree within 95% of those identified by the tutor.

2.2 Given a problem, you will be able to list the possible knowledge you would need to know to solve the problem; your list shouldagree within 80% of the list of others in your group and within 85% with the list of the facilitator/tutor.

3.1 Given a problem and a list of the possible knowledge and resource available, you will create learning objectives and methods ofassessment. These will be 90% acceptable according to the criteria for self-performance assessment (in MPS 3)

3.2 Given the learning objectives and methods of assessment, you will be able to identify reasonable and pertinent sources ofinformation and be able to allocate the resources to achieve the objectives in the time available.

7-19

3.3 Given the learning objectives, you will create for yourself examination questions that are consistent with the objectives. Thesewill be judged by peers and or tutor to be 90% acceptable.

3.4 Given that other members of the group have acquired key knowledge pertinent to the situation, you will ask questions so thatyou can learn from them the knowledge you need. You will interact such that they will rate a willingness above 70% to continue toshare information with you.

3.5 Given that you and your group have completed a teach meeting, you will pause and write reflections once every ten minutesabout the process used.

5.1 Given that you have contracted to teach topics to meet selected learning objectives of the group, you will learn the knowledge,and with due consideration of the learning styles of other members in your group, prepare “teach notes” and use these to teach theother group members such that you receive feedback from all members of the group with ratings > 6/7 for knowledge and > 6/7 onquality of instruction.

5.2 Given the first PBL teach session described in Section 5.1 and given the feedback of your five strengths and two areas to workon, within three PBL cycles you will maintain all your strengths and shift one of your weaknesses to be a strength. The evidencewill be the feedback from all the members of your PBL group.

5.3 Given the initial problem and the solution to the problem (from 3.6), you and the group will elaborate in writing to identify atleast thirteen different perspectives. The elaboration will be assessed as being correct and enriching by the tutor or a group of peers.

6.1 Given the initial problem and the knowledge that you and the group have learned and your solution to the problem, you and thegroup will use the knowledge to solve the problem. You and the group will self assess that the problem solving process, the answerto the problem and lifelong learning was the best you could do with the time available.

6.2 Given the feedback from a Perry inventory, you will be able to judge the degree to which this assessment is valid; you can listthe five strengths and two areas to work on such that in a cycle of six PBL sessions you will shift such that a retest using the Perryinventory will show a rating of > 4 for all dimensions of the inventory.

6.3 Given skill in self assessment (MPS 3) you will create a learning journal in which you record and self assess your progress as aself directed learner. This will be assessed once per month by a peer, mentor, tutor to be “satisfactory”.

Concepts introduced

Peers as resources, Perry's inventory, reflection, elaboration, learning objectives, assessment. Elaboration: is the creation of a new perspectives of the ideas learning, the problem solved and the relationship of thenew ideas to previous experience. Elaboration includes activating previous knowledge and searching for connectionsbetween previous and new knowledge. Elaboration includes determining the details of the new knowledge, creatingexamples, analogies, restatements, conditions, deductions and simplifying and generalizing. Elaboration includessearching for similar concepts and noting confusing elements, similarities and differences.

7-20

Table 2 MPS 29: Chairperson skills © copyright, Donald R. Woods, 2005Chairperson skills: facilitation skills used so that a group applies problem solving and interpersonal skills toefficiently and effectively complete a task & develop good morale among the group members. Skill development:

1. Learn the characteristics of groups and the role of the chairperson to facilitate group evolution to a team.2. Realize that each person has a unique contribution to the group process; that contribution should be encouraged,nurtured and used to the advantage of the group. The chairperson is to facilitate this encouragement, and nurturing.3. Learn how to give and receive feedback . 4. Recognize that trust is the most valued element in relationships and be able to list those behaviours that build trustand those that destroy it . 5. Acquire some skill at listening6. Acquire some skill in assessment7. Realize that both morale and task are important8. Accept that chairperson is not leadership. Leadership varies from person to person depending on the task.9. Accept that if a group is faltering, process skills are needed. Gradually you will develop skill in facilitating thegroup process. 10. Through self awareness, begin to improve self confidence11. Begin to develop an environment of trust where risking is OK.12. Group skills are the first step toward developing team skills13. Be able to identify the different elements that make up “group norms” and recognize the importance ofdiscussing these early in the group process.

********************************Pretest:Chairperson skills how aware are you of what you do as chairperson of a group? Rate with an “x”

0 1 2 3 4 5 6 7 8 9 10Unaware Aware of Very awareI just Some I can describedo it The details of

how I do it

Skill: how skilled are you as chairperson? Rate with an “x”

0 1 2 3 4 5 6 7 8 9 10Poor Fair Good Very good Excellent

Comments:

_______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

MPS 29 Chairperson skills

1.1 Given a term listed under "concepts introduced", you should be able to give a word definition, list pertinent characteristicsand cite an example.

5.1 Given that a chairperson is assigned to the group and that, for growth, the chairperson will receive feedback from everygroup member at the end of each meeting, all will have practice completing form 2901 (Feedback to chairperson) such that therewill be no more than 1.8 scale variations on the rating of Task and of Morale among all the raters and the Total ratings ( about

7-21

the role of the chairperson) will not deviate by more than 10 among all raters. The ratings will also agree with those of a trainedoutside assessor.

6.1 Given a group, you will facilitate a norms meeting so that the group unanimously decides on the typical 17 issues related togroup process. These range from the role of the chairperson to procedure for asking a person to leave the group.

6.2 Given a group and a decision that has to be made (and perhaps the process to be used to make the decision, eg, vote,consensus, unanimity, nominal group), and given the feedback from form 2901 from all the participants, you will be able toobjectively summarize the findings, extract a description of the role you played, contrast the descriptors the members give youwith the ones you wrote in the pretest and write out an action plan to help you improve. Your analysis of the evidence shouldagree within 80% of that of the tutor.

6.3 Concerning the assessment of the group and of the chairperson using form 2901, given the same group functioning underdifferent chairpersons at different times and with different tasks, as a group you will consistently rate the group’s performance onTask and Morale and consistently discriminate between the group’s inherent ability and the chairperson’s contribution to thegroup. The consistency will be within 1.8 units on the 7 point scale used on form 2901.

6.4 Concerning improvement, given the assessment in #6.2, and given other opportunities to be chairperson, you will convert atleast one of your “Areas to work on” into a “strength”.

6.5 Given that you are to chair a meeting and given the purpose, you will prepare an agenda and circulate it to all attendees atleast 48 hours ahead of the meeting. The agenda shall include names, time and place of the meeting, purpose, prework that isexpected of all participants, a list of materials and information to bring to the meeting, a list of the topics to be addressed at themeeting and the time allocated for each item. An outside observer or tutor should assess the agenda as being 100% complete.

6.6 Given that you are to chair a meeting and given the purpose, you will prepare an agenda , devise ahead of time whatevermaterials you feel need to be brought to the meeting, circulate (at least 48 hours in advance) the agenda to all interested parties,run the meeting and complete 60% of the agenda in the time available and receive better than average ratings on both task andmorale for the conduct of the meeting and receive a total rating on task (the sum of the totals from all group members onchairperson feedback form) > 30 and > 30 on morale.

6.7 As in #6.6, but complete 80% of the agenda and receive total ratings for each Task and Morale of >100

6.8 As in #6.6, but complete 100% of the agenda and receive total ratings for each Task and Morale of > 300.

Concepts introduced

Agenda, norms meeting, feedback form 2901, chairperson, leader, Task, Morale, Sanderson’s 20 minute rule, vote, unanimity,nominal group, consensus.

7-22

Table 3 MPS 36 Lifelong learning skills

Evidence-based targets for lifelong learning. Although some general principles have been reported that relate to lifelong learning,most of the target skills are based on best practices as recommended by researchers.

Lifelong learning is learning how to learn so that whatevercomes our way we empower ourselves to master new knowledge.Evidence-based targets

Progress toward internalizing these targets

20% 40% 60% 80% 100%

1. Consider peers and classmates as resources to help me see mylearning needs, plan my learning and provide new ideas so that Ican learn from them.

2. Able to assess learning needs realistically.

3. Able to create observable, unambiguous and achievablelearning objectives to match or satisfy my needs. I use these toassess my progress.

4. Can relate to teachers as resources, facilitators and helpersrather than as the sole source of knowledge. Have acquired anattitude toward learning comparable to Perry level 5.

5. Able to identify people and material resources needed toachieve my learning objectives.

6. Able to shift from being a dependent learner through being anindependent learner to being an interdependent learner.

7. Able to devise a time plan and stick to it reasonably well.

8. Willing to assume responsibility and ownership for the tasksin learning, including goal setting, resource identification,learning and assessment.

9. Able to meet contract commitments to teach others.

10. When teaching others, able to use the principles of learningand address differences in learning styles (instead of reportinginformation and expecting the learner to sort it out.

11. When learning a difficult topic, am willing to accept thechallenge to unravel the complexity - instead of skipping over itand hoping it won’t be on the exam.

7-23

Table 4 Feedback form 3601

Feedback to ___________________________ for Unit ____ Date ___________ Present & on time: G Present but late by ___ min. Absent G

Quality of Knowledge: good intellectual understanding of the topic, the material supplied was complete andappropriate.

None of | A few but | Most of these. | All ofthese. major omissions. these. O__________O_________O_________O_____________O________O________O_________O1 2 3 4 5 6 7 8

Quality of Instruction: he/she was here on time, the presentation was focussed on the new knowledge; good choiceof material and medium with effective communication and resource material supplied.

None of | A few but | Most of these. | All ofthese. major omissions. these. O__________O_________O_________O_____________O________O________O_________O1 2 3 4 5 6 7 8

Followup: from this teaching I will have:

Must study subject Major self-study Some No self-on my own; needed. I have some self- study of I learned starting references study of the basics. I willnothing from from your teach. the basics. practice on your teach. problems/reflect on

ideas.. | | |O__________O_________O_________O_____________O________O________O_________O1 2 3 4 5 6 7 8

Strengths Areas to Improve on_____________________________ _____________________________________________________ ____________________________________________________________________________________________________________________________________________ from D.R. Woods(2005)

7-24

Table 5: Form 2901 Chairperson Feedback Date of meeting: _____________ Group: _____ Chairperson: __________________________Agenda circulated ahead of time, yesG no G; start on time, yes G, no G;. end on time, yes G, no G!Task: Problem defined, many issues and hypotheses explored, criteria listed and the issues prioritized. Refrained from early closure. Taskcarried out and looked back at the result to assess it. Group agreement as to goals. Process was active with monitoring. Completed task on time.The accuracy in the group’s answer matched the time available. Group avoided contributing excessive information. None Few but Most features All of these

major omissions demonstrated behaviours G G G G G G G1 2 3 4 5 6 7

Total

_______

Agenda: Names, time, place,purpose, prework, bring, topics,timing

Use of agenda: Keep ontopic, effectively & notdisruptively keep on time,20 min rule

Facilitation as needed when team functioning below norms on Task

15 to 20 Excellent & ahead of time Used very well Mega positive: clarified, brought back on task, monitored Task

10 to 14 Poor & ahead of time OK Some positive as needed: clarified, back on task, monitored Task

5 to 9 Excellent but handed out at start Fair Neutral because working well

0 to 4 Poor and handed out at start

0 to -4 Excellent created at start Poorly Neutral when help was needed or unskilled and unable to facilitate

-5 to -9 Poor created at start

-10 to -15 No agenda Not used Kept dominating as leader, when facilitator need, Solved problem forgroup, when they didn’t want that solution

-16 to -20 Mega negative, imposed own ideas and answers. DictatorMorale: Group relaxed; enjoyed working together. They gave emotional support to each other and were able to express disagreement ordisappointment directly. Seven fundamental rights preserved. Members are enthusiastic and involved. None Few but Most features All of these

major omissions demonstrated behaviours G G G G G G G1 2 3 4 5 6 7

Total Agenda: Use of agenda: Facilitation as needed when team functioning below norms onMorale

15 to 20 Excellent & ahead of time Used very well. Comfort highbecause know goals

Skilled observer of interpersonal process, praiser, facilitates,conflict resolution and provides tension relief as needed.

10 to 14 circulated ahead but missingprework & bring

Comfort moderate, can inferwhat to do ahead of time

Some skill and intervention

0 to 9 Fair No intervention because all working well

-1 to -9 Some frustration; feel ill-prepared because didn’t knowwhat to prework or bring

Poor No intervention when help was needed or unskilled and unable tofacilitate

-10 to -15

No agenda, High distress Not used, high distressbecause unclear as to wheregoing

Some, causes lower-than-norm morale because of criticism,defensiveness, contemptuousness, stubborn, conflict

-16 to -20

Mega causes tension, seeks personal goals, critical, abusive, selfcentered and defensive, contemptuous, ignores others

Personal Satisfaction with decisions made: G 0 extremely dissatisfied...5 OK..... 10 very satisfied !Strengths Areas to work on_____________________________________ _______________________________ ____________________________________ ____________________________________________________________________________________________________________________________________________ Instructions: as an individual rate the Task elements of the meeting.Then consider the role the chairperson played by circling the appropriate descriptor in each column for the three elements:agenda, use of the agenda and facilitation. Total the corresponding numerals from the LH column. Example: excellent agenda buthanded out at start (7); fair use of agenda (5), neutral because working well (2) for a total of 14. Repeat for the Morale. Scoreyour personal satisfaction with the decision made: numeral out of 10; List five strengths and two areas for the chairperson to workon.

7-25

Table 6 Example of Individual learning styles and preferences

Jungian typology Lancaster approaches to studying Perry KAI

S T E P strategic rote deep

Self 25 24 28 10 14 16 15 3.5 78

John 22 21 15 15 14 18 16 2.8 100

Karen 27 24 23 16 19 15 19 3.9 100

Rob 18 23 29 23 11 17 16 3 102

Sarah 21 26 24 14 14 19 17 3.3 86

7-26

Table 7, Example learning contract (reprinted with permission from Karen Rogers, Jeff Bunn, Deirdre Schroder, Ben Yue, CraigLee and Deepti Asthana, copyright 2005)

7-27

Table 8 Summary of Tables comparing Option # 1 with Option #2

Option # 2no. studentswith Perryno. Rating out of 8 from Form 3601

Comments

Group < 3 < 4 knowledge communicationfeedbackrequired

1 0 1 7.4 7.4 7.4 Strong group2 0 0 7.7 7.8 7.7 Strong group3 1 2 7.5 7.6 7.2 Strong group4 2 1 7.8 7.8 7.6 Strong group5 2 4 7.7 7.5 7.6 Struggling group6 3 0 7.9 7.9 7.7 Strong group7 1 1 7.9 7.9 7.8 Strong group8 0 1 7.3 7.2 7.2 Strong group9 1 1 7.6 7.4 7.4 Strong group

10 1 2 7.5 7.5 7.5 Not all doing their share; one in particular11 2 2 7.5 7.5 7.4 Diverse but productive

>7.2 11 11 11

Option #1no. studentswith Perryno. Rating out of 8 from Form 3601

Comments

Group < 3 < 4 knowledge communicationfeedbackrequired

1 2 0 7.1 7.1 7.1 Not all doing their share; two in particular2 0 0 6.3 6.8 6.8 Two ineffective in teaching3 0 0 6.9 6.7 6.6 Tried hard to make this effective

4 0 3 7.1 7.1 7Major problems; lateness, rote memory;inadequate teaching

5 1 1 7.1 7 6.86 0 0 7.4 7.2 7 All committed; all Perry shifted

7 0 2 6.8 6.9 7Two are not committed; no learningcontract;

8 2 3 6.3 6.8 5.3Morale issue; Perry problems; onedelinquent

>7.2 1 0 0

8-1

8. How to assess student performance: copyright © 2002 Donald R. WoodsDonald R. WoodsMcMaster University, Hamilton, Canada C/PBL/wksp 6-singapore02

Case 6: In this PBL approach, the students are teaching each other. How do we know if they arelearning it correctly? Many of my students don’t even understand it from my lectures, so howcan they “teach” someone else?Besides how do I assign a mark for each student. Some are scholarship students. I better givethem a 3 hour exam.”

*******************************Activity: In small group of 5 or 6, with chair _______________________________; reporter ________________________, brainstorm the issues this case raises. Identify what you know already.Identify what you need to learn.

Prioritize the issues: criterion: what do you want to gain from this workshop in the context of the issues raised._________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

8-2

Feedback about the group work. Form 2802

Task: Problem defined, many issues and hypotheses explored, criteria listed and the issues prioritized. Refrainedfrom early closure. Task carried out and looked back at the result to assess it. Group agreement as to goals. Processwas active with monitoring. Completed task on time. The accuracy in the group’s answer matched the time available.Group avoided contributing excessive information. None of Few of these Most features All of thesethese behaviours but demonstrated behavioursbehaviours major omissions

G G G G G G G1 2 3 4 5 6 7

Morale: Group relaxed; enjoyed working together. They gave emotional support to each other and were able toexpress disagreement or disappointment directly. Seven fundamental rights preserved. Members are enthusiastic andinvolved. None of Few of these Most features All of thesethese behaviours but demonstrated behavioursbehaviours major omissions

G G G G G G G1 2 3 4 5 6 7

Individual Contribution to Task and Morale

____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Group Strengths Group Areas to work on_____________________________________ ________________________________ ____________________________________ _____________________________________________________________________________________________________________________________________________

from D.R. Woods (1995)

8-3

Feedback from the Goals Meeting

Case 6:IssuesNumber identified: 1 2 3 4 5 6 7 >7

Agreement with tutor <50% 50% 60% 70% 80% 90% 100%

Knowledge/skills to be learned Consensus among group little some a lot complete

Agreement with tutor's list little some a lot complete

Learning objectivesQuality poor fair OK good excellent

LearningQuality of questions asked during the teach session none some astute excellent

Willingness to continue to contribute <50% 50% 60% 70% 80% 90% 100%

Your AttitudePerry shift 2 3 3.5 4 4.5 5******************************************************************************

Reflections:______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________-

8-4

Activity:a perfect circle Discuss with neighbour what’s going wrong in this scenario_________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

1. What is assessment? Use a framework of assessment for the accountability. Five principles and Six issues inpractice.Five Principles:Principle 1. Assessment is a judgement based on performance - not personalities. We need tohelp a student realize that a poor mark does not mean he/she is a bad person. The judgement ismade about performance in completing a task. It has nothing to do with his/her value as anindividual. This is an issue, especially for students with attitudes characterized by Perry’s level2. Principle 2. Assessment is a judgement based on evidence - not feelings. We might intuitivelyfeel that a student is a good problem solver. However, we need to replace that intuitive feelingwith physical evidence such as the written script on an exam or in a project report. Help thestudents (and staff) gather evidence. Provide time for students to write reflections. The staffresponsibility is to create well-designed standardized feedback/assessment forms. Providepractice understanding the terms and working with the forms before the program.

Principle 3. Assessment should be done for a purpose with clearly-defined performanceconditions. The student should know when he/she is being assessed.

Principle 4. Assessment is a judgement done in the context of published goals, measurablecriteria and pertinent, agreed-upon forms of evidence. Use research, not personal intuition, toidentify the target skills. Publish goals with measurable criteria for process skill. Such goalsshould provide clear needs for documented evidence. These should be published at the start ofthe program so that there are no surprises for the students and no unwanted student backlash tothe teacher.

Principle 5. Assessment should be based on multidimensional evidence: static and dynamicsituations; small assignments and lengthy projects; academic, social and personal contexts; undera variety of performance conditions (exams and homework, written and oral, performance as anindividual and as a member of a group,) formative and summative data and with differentpersons being the assessors (self, peer, teacher and trained external observers).

To remove ambiguity from the assessment the following six issues in practice should beaddressed (Alverno, 1985, Woods, 1994).1. Goals: What is being assessed? Knowledge in engineering or a discipline subject? Skills? Attitudes? Have thegoals been expressed unambiguously in observable terms? Who creates the goals? Are the goals explicit and

8-5

published?2. Criteria: Are there criteria that relate to the goals? Can each criterion be measured? Who creates the criteria? Arethe criteria explicit and published?3. Form of evidence: What evidence is consistent with the criteria? Are the checklists used for the assessment askingquestions related to the criteria? Do both the assessor and the student know that this form of evidence is acceptable? 4. Resources: Are the goals and the collection of the evidence possible to achieve in the time and with the resourcesavailable? 5. Assessment process: What is the purpose of the assessment? Under what conditions is the student’s performanceassessed? Who assesses? What type of feedback is given by the assessor? (For example, Pass fail? a grade? fivestrengths and two areas to work on?) What is the form of feedback? Verbal? Written? What is the timing offeedback? Who delivers the feedback? 6. Training in the assessment process: Have both the student and the assessor received training in assessment?

Additional Ideas:Provide ± feedback; not just -ve. Guideline: Use 5:2 ; 5 positives for every 2 areas to improve.Gibbs “Whoever owns the assessment owns the learning.”Student misconceptions; Agent 100% “Licence to pass” & Perry level 2.

2. What to assess?Subject knowledge “Process skills”

identify issuesgoal settinglocate informationcritically assesswritten/oral communicationlifelong learning skillsproblem solvinggroup skillsself/ peer assessmentanalyze/ create deal with conflict

___________________________________Activity:For your PBL course or assuming you are going to use PBL next termas an individual: what % of the emphasis is on the development of “process skills”.

% Subject knowledge _________ % Process skills _______

3. Set goals and criteria Goals:List Target skills (based on research not intuition)Make them Observable and unambiguousLevels of development: start with simple and achievable. Bloom’s taxonomy is a good guide.Numerous: guideline for a course: 30

8-6

“Process” or “product”

Criteria:Must be related to the goals: don’t download “metrics”from www.Need to measure.Challenge: “ answer should agree 100% with that of the instructor”. Rarely is this stated inundergraduate courses, so... when we need to create them in “soft skills” or “thinking skills” itusually is a challenging task. (Took us 13 years to create a set for 52 process/thinking skills)*****************************************Example: Team Skills: GoalsLevel 1: To be able to characterize and diagnose effective interpersonal skills and effectivegroups and to become aware of the personal role one typically assumes in group work. To setgoals for group growth.

Level 2: To build up individual skills and apply them effectively to group work. The individualskills include self awareness, self acceptance, acceptance of others, and self confidence, conflictresolution, listening and responding , decision making and assertiveness.

Level 3: To build up individual skills and apply them effectively to group work. The individualskills include chairperson skill, reaching consensus, goal setting and coping with difficultbehaviours.

Level 4: To integrate and apply the skills to team development through benchmarking,diagnosis, and implementation of plans for growth.

Example Team skills: MPS 28: Group skills © copyright, Donald R. Woods, 1999Group skills: application of problem solving and interpersonal skills to efficiently andeffectively complete a task & develop good morale among the group members.

1.1 given a term listed under "concepts introduced", you should be able to give a word definition,list pertinent characteristics and cite an example.

3.1 given a problem, as a member of a group, you will participate in the task and moralecomponents of the process such that your participation will be judged by an observer to be"active" and to have more positive than negative contributions.

3.2 as above but your performance will be judged to be "most of these skills" for both task andmorale by two independent observers.

3.3 given a problem, the group will complete the assigned task by the designated time and over60% of the members of the group will choose to work together again.

3.4 given a group solving a problem, you will be able to observe the group, record plus an minus

8-7

contributions toward both task and morale components. You will be able to do this observing anindividual client or the group as a whole. Your records will agree to within 80% of the averagerecords made by four other observers or with those of the tutor.

6.1 as a member of an observer group, you will be able to provide written suggestions on howthe group can improve its task and morale components. Eighty percent of your recommendationswill agree with those of the tutor.

6.2 as a member of a group solving a problem, you will be able to self assess the performance ofthe group. Your assessment should agree within 80% of the assessment of outside assessors.

6.3 as in 6.2, you will be able to self-assess your personal contribution to the task and moralecomponents of the group process. Your assessment should agree within 80% of the assessmentof an outside assessor.

Concepts introducedTask and the positive and negative contributions toward task, Morale and the positive andnegative contributions toward morale, FIRO-B, stages of group evolution, feedback form.

*****************************************Activity:As group of ____________ create a learning goal at Bloom’s level ________ to identify skillexpected in “__________________________”

Examle: Level 1: KnowledgeLevel 2: ComprehensionLevel 3: ApplicationLevel 4: AnalysisLevel 5: SynthesisLevel 6: Evaluation

4. Forms of evidence:For some skills, activities, performances we need“Observation forms” that are well-designed:! valid (measures the goal), ! reliable (different people using the same form will give the same results),! easy to use.Should provide training in its use.Valid: published goals, specify levels of performance along appropriate continuum scales,compare ratings with other dataReliable: clear goals, clear criteria for levels of performance, train ratersEase in use: consider Likert type scale.

Some example forms of evidence:

8-8

ReflectionsReflective learning journalPortfolioWorkshop activitiesPeer and self assessmentOral presentationWritten presentationTests & Exams in Process Skills, TEPS

5. Assessment: who does the assessment?Self

PeersTutor

Student MisconceptionsNot licence to give yourself 100%, Not Agent 100Based on the FundamentalsProvide workshop training in how to do an assessment

Student expectation: and attitudes about their role and teacher roleAttitude: Perry Scale

Activity Example: “Self assessment”Goals: observable, unambiguous, published (Where going)

Criteria: measurable, related to goals, published (How know when arrived)

Performance evidence: published, well designed

Enrichment:1. Please help me understand how best to help you

8-9

2. MRIQ3. See Workshop #2 for a set of Assessment-Feedback forms.

References:Lovie-Kitchen, J., “Reflecting on Assessment” Chapter 20 in “Problem-based Learning: casestudies, experience and practice,” P. Schwartz, S. Mennin and G. Webb, Kogan Page, 2001 highlights the successful use of Reflective Journals

Forrest, A., and L. Walsh, “Assessable Damage” Chapter 21 in “Problem-based Learning: casestudies, experience and practice,” P. Schwartz, S. Mennin and G. Webb, Kogan Page, 2001 highlights difficulties that occur when students are not well-prepared for “self and peer”assessment. The issues include mistakes made in allowing students to self select their groups;collusion where all students decide to give each other high (and the same) marks suggesting lackof clear goals for self assessment, training via workshops; and the sense of “loss of face”.

Woods, D.R. “They just don’t pull their weight,” Chapter 22 in “Problem-based Learning: casestudies, experience and practice,” P. Schwartz, S. Mennin and G. Webb, Kogan Page, 2001 highlights the effective use of peer and self assessment to improve the accountability within thegroup.

Woods, D.R. et al., “Assessing Problem Solving Skills: part 1: the context for assessment,” ChemEng Education, 35, no 4,

Woods, D.R. et al., “Assessing Problem Solving Skills: part 2: the context for assessment,” ChemEng Education, 36, no 1, 60 - 67, 2002

Woods, D.R. et al., “Assessing Team skills,” Chemical Engineering Dept., McMaster University,Hamilton ON

Woods, D.R., “Assessing Liflelong learning skills,” Chemical Engineering Dept., McMasterUniversity, Hamilton, ON

8-10

1 Current address, Department of Chemical and Materials Engineering, University of Alberta, Edmonton,AB, T6G 2G6, Canada <sdtaylor@ualberta.ca>

2The Procter & Gamble Co., 8256 Union Center Blvd., West Chester, OH, 45069, USA <jaffer.sa2@pg.com>

Assessing team skills. Part 1: successful teams

Donald R. Woods, Shawn Taylor1, and Shaffiq Jaffer2

Chemical Engineering DepartmentMcMaster University, Hamilton ON, L8S 4L7 [128]

Abstract

The five principles of assessment are summarized. The evolution from a cluster of people to an effectivegroup to a team is described. Goals and criteria are given for effective teams. Nine options for gatheringevidence are described. Actions to implement a program to improve group skills are listed.

Keywords

Assessment, group work, teams, research on effective groups, rating forms, ABET 2000, programoutcomes.

8-11

Laboratory work, design projects, cooperative learning - many learning situations are used wherestudents are expected to work in teams. Group work is used 1) to prepare students for the future (wherethey will be expected to work in teams), 2) to save faculty marking time, 3) to minimize laboratoryequipment and the number of experiments that are needed for laboratory work, 4) to improve learning byhaving students work cooperatively instead of competitively, 5) to allow faculty to assign moreambitious projects and 6) to develop team skills. An analysis of the future of engineering educationsuggests that skill in working in teams is needed by our graduates. [1] For example, ABET EngineeringCriteria 2000 criterion [2] 3.d requires that we show that our graduates have Aan ability to function onmultidisciplinary teams.@ An effective team has been defined as Aa group of people who work togethertoward a common vision and accomplish extraordinary results.@ Francis and Young [3] suggest that teamsAsucceed despite difficulties and with members feeling responsible for the output and acting to cleardifficulties standing in the way.@

Unfortunately, most students do not develop team skills from being put on a team and asked to Abe ateam.@ Just placing students in groups or teams does not automatically develop group skills. Studentdissatisfaction is usually high because everyone gets the same mark even though some team membersbelieve that they contributed more to the project than did others. Some students hide in the group or justfail to be responsible to the group. They don=t do any work and/or they fail to come to meetings.

In this paper, we review the principles of assessment, differentiate between effective groups and teams,list goals and criteria for effective teams, and suggest options for gathering evidence. Actions toimplement a program to develop overall effective teams are given. In the next paper in this series, Part 2, we consider how empowerment and accountability are linked, list goals for the personal development ofteam skills, list some example options for gathering evidence and give some example data.

For ABET purposes and to improve our educational efforts we also want to evaluate the effectiveness ofour programs to develop team skills. More details are given in Part 2.

1: Principles of Assessment

We find it useful to remind students of the elements of assessment. Students often view assessment as theprocess of Atrying to figure out what the instructor wants.@ We define assessment as Aa judgement basedon the degree to which the goals have been achieved using measurable criteria and pertinent evidence.@We have found that breaking this definition into five principles assists in applying this definition. Thefive principles are: [4-7]

Principle 1: Assessment is a judgement based on performance - not personalities. We need to help astudent realize that a poor mark does not mean he/she is a bad person. The judgement is made aboutperformance in completing a task. It has nothing to do with his/her value as an individual. This is anissue, especially for students with attitudes characterized by Perry=s level 2. More details about Perry=slevels and their implications to teaching and learning are given elsewhere. [4,8,9]

Principle 2: Assessment is a judgement based on evidence - not feelings. We might intuitively feel that astudent is skilled at team work. However, we need to replace that intuitive feeling with physical writtenevidence.

Principle 3: Assessment should be done for a purpose with clearly-defined performance conditions.

8-12

Principle 4: Assessment is a judgement done in the context of published goals, measurable criteria andpertinent, agreed-upon forms of evidence.

Principle 5: Assessment should be based on multidimensional evidence: static and dynamic situations;small assignments and lengthy projects; academic, social and personal contexts; under a variety ofperformance conditions (exams and homework, written and oral, performance as an individual and as amember of a group), formative and summative data and with different persons being the assessors (self,peer, teacher and trained external observers).

To remove ambiguity from the assessment, the following six issues in practice should be addressed.[4-7]

1. Goals: What is being assessed? Knowledge in chemical engineering? Skills? Attitudes? Have the goalsbeen expressed unambiguously in observable terms? Who creates the goals? Are the goals explicit andpublished?2. Criteria: Do the criteria relate to the goals? Can each criterion be measured? Who creates the criteria?Are the criteria explicit and published?3. Form of evidence: Is evidence consistent with the criteria? Do both the assessor and the student knowthat this form of evidence is acceptable? 4. Resources: Are the goals and the collection of the evidence possible to achieve in the time providedand with the resources available? 5. Assessment process: What is the purpose of the assessment? Under what conditions is the student=s performance assessed? Who assesses? What type of feedback is given by the assessor? (For example,Pass/ fail? A grade? A list of the five strengths and two areas to work on?) What is the form of feedback?Verbal? Written? What is the timing of feedback? Who delivers the feedback? 6. Training in the assessment process: Have both the student and the assessor received training inassessment?

Failures of assessments to accomplish their purpose can usually be traced to violations of any one of thesefive principles or to the incorrect application of the six issues in practice. The major challenges studentshave seem to be:M Students can understand goals and objectives for subject knowledge. AIt=s Chapter 3 in the text.@ orAIt=s like the assignments we=ve had, only changed a little (at least we hope it is changed only a little).@ But how do you assess teamwork? An answer is that we need to create easy-to-understand anddemonstratable goals.M Students underestimate the importance of evidence. Furthermore, they have trouble seeing the types ofevidence that might be useful.

2. Groups versus Teams.A team is not just a collection of individuals. It is more, much more. It is useful to differentiate betweencollections of individual efforts, group work and team work. Table 1 summarizes some of the keyfeatures. Often when we ask students to work together as Ateams@ they function as collections ofindividuals. Getting collections of individuals to function as effective groups is a challenge. Ourexperience is that, after at least two courses in which students receive explicit training on how to worktogether effectively [10] , the best we seem to be able to achieve is for 10 groups working effectively asgroups with occasionally two of these working as effective teams. In our experience, the transition ischallenging because:1. groups evolve through the stages of forming, storming, norming and performing. [4] Some groups, afterfour weeks, are still in the transition between forming and storming. They never reach Aperforming@. 2. providing training in self assessment, goal setting, conflict resolution and sharing information aboutpersonal styles (such as Kirton=s KAI, [11] Schutz=s FIRO-B, [12] Jungian Typology, [13] men and women in

8-13conversation [14]) is time consuming and often viewed initially by teachers and students as being a Awasteof time@. Later they realize how important this information and skill are for group work. For groups tolearn cooperatively, we have found it useful to use the Perry inventory [4,8,9] and the Lancaster Approachesto Studying [15] to give the needed underpinnings for effective interpersonal interaction. Without suchtraining, the transition from Acollections of individuals@ to Agroups@ or Ateams@ is very difficult.3. few people are skilled chairpersons. Research has shown that every group performs better with achairperson. , and people have not developed strong skills as chairs/facilitators of the group or team. Achairperson=s role is to facilitate the completion of the designated task while maintaining highinterpersonal morale among all group members. The chairperson usually thinks through the tasks to bedone, decides on the need for a team meeting, identifies the time and place for the meeting, sets andcirculates an agenda, and facilitates the meeting.4. the distinction between chairperson skills and leadership skills needs to be made. Whereas one personis assigned to be chair each meeting, leadership changes from time to time as the team work progresses.The leader is the person who has the most to contribute to the meeting at one particular time. Forexample, an immediate task that occurs in a meeting might be to develop a plan. The person withextensive experience in planning usually Aleads@ the team at this time. Another example might be that thegroup is experiencing conflict. Usually the chairperson will Alead@ the group at this time. However, ifanother member of the team has extensive experience in coping with conflict, that person might Alead@ atthat time. Any team member can be a Aleader@ depending on the group needs. 5. the only motivation most students have for trying to work effectively in groups is marks. If the marksare assigned for just the product produced by the group and no individual accountability is required, thenthe individual effort needed to make the group work effectively is often not made by all members.Contrast this with most business and corporate environments; those that do not shape up are shipped out.

Since a major challenge seems to be to get groups working effectively, the prime focus in this paper isthe assessment of effective groups. The ideas can be extended to teamwork by changing the targets, theforms of evidence gathered and the criteria for assessment.

Ultimately we want to improve, assess and evaluate the effectiveness of developing an individual=s skillin team work. However, we need to know the overall team=s effectiveness first and help each individualunderstand his/her role in the team. We consider assessing the overall team in this paper and assessing theindividual skill in Part 2.

3: Publish Goals and Criteria for Team work

Consistent with Principle 4, goals should be published to describe the target skills and attitudes ofsuccessful teams. (In Part 2 of this paper, we consider skills and attitudes for successful team players.) Although initially we might be satisfied with a general objective [16] like Aable to function well onmultidisciplinary teams@ we have found it extremely helpful, especially for the purposes of assessment, toelaborate on the skill. Such elaboration should be based on research findings [10] about the performance ofeffective teams and effective individual contributions to teams. Here are some options for creating goalsand criteria for effective teams.

! Create a list of descriptors of effective groups or teamsAn example list for a Team consists of two elements: the Aprocess@ Task (or problem solving processused) and Morale. Process Task: Athe problem defined, many issues and hypotheses explored, criterialisted and the issues prioritized. Group refrained from early closure. The Task was carried out and thegroup Alooked back@ and checked that the result matched the goal and seemed feasible. The group agreedon the goals. The Task process was Aactive@ with frequent monitoring. The task was completed on time.The group applied successive-approximation and optimum sloppiness. The group avoided contributing

8-14excessive information.@ Morale: AThe group was relaxed and enjoyed working together. They gaveemotional support to each other and were able to express disagreement or disappointment openly anddirectly. The seven fundamental RIGHTS of individuals were preserved. Members were enthusiastic.Trust is growing.@

A disadvantage of this approach is that few criteria are included. For example, without published criteriaone observer might judge that Atrust is growing@ whereas another assessor might decide that Atrust isdecreasing.@

! Create a list of target skills and attitudes for the team:An example list for team skills is given in Table 1. [17-25] As with the previous option no criteria are given.

! Provide a structured list of goals moving from beginning to advanced.The following set of levels can be used as a general guideline for growth.

Level 1: All members meet together, spend time getting to know each other and gaining respect for eachother=s perspectives. Some are apathetic. The group sets realistic goals for improvement and makes someprogress toward achieving the goals; the distinction between chairperson and leadership is still confusingbut the members are willing to work toward better understanding. Members tend to hope that conflictdoes not occur and hesitate to apply conflict resolution techniques and assertiveness. Decisions tend to bemade by consensus. Decision making is characterized as being Aacceptable but inadequate.@ Theprogression through Aform, storm, norm, perform@ is slow and painful.

Level 2: All members meet together. Few are apathetic. Generally all accept the goals, some may beunsure of their role in the group, the chairperson is developing his/her skill; good understanding of thedifferences between chairperson and leadership. Leadership does not shift smoothly as the group processevolves; often discussion and conflict occurs over who is leader Anow.@ Decision making is by formalvotes. The group has some methods for coping with conflict and attempts are made. The group remainsreticent to welcome conflict. Trust is developing. Not all are involved. The group still prefers to havemembers with similar styles. The progression through Aform, storm, norm, perform@ is reasonablycompetently done.

Level 3: All members look forward to meeting together. They unanimously agree on the goals, each isclear about roles, the chairperson leads the Aprocess@ very well (circulates clear and detailed agendas,focuses on and is skilled in applying the process), there are no fights for leadership; leadership shifts. Thegroup makes decisions by unanimity. The team welcomes differences because Ausing differentperspectives improves the product.@ If differences do not occur naturally then someone willingly plays theDevil=s advocate or skeptic to challenge Agroup think@ and bring in other perspectives. Trust is high, allare involved and accountable. Unique skills of individuals are used effectively; empathy is high. Theywork rapidly through the process of Aform, storm, norm and perform.@

! List component skills and create sets of behavioural, unambiguous goals with measurable criteria foreach. For example in the McMaster Problem Solving program, [10] MPS, the four levels of developmentare: Level 1: To be able to characterize and diagnose effective interpersonal skills and effective groups and tobecome aware of the personal role one typically assumes in group work (MPS 3, 52, and 28). To setgoals for group growth, some goals might be:

After the first meeting of the group, the group will write down five strengths and two areas to workon. 80% of these match those written by an external observer.

8-15As a group, over a sequence of five meetings the strengths will be maintained and one area to workon has been shifted to be a strength. The assessment will agree with that of an external assessor.

Level 2: To build up individual skills and apply them effectively to group work. The individual skillsinclude self awareness, self acceptance, acceptance of others, and self confidence (MPS 1, 4, 11), conflictresolution (MPS 49), listening and responding (MPS 26), decision making (MPS 24) and assertiveness(MPS 44).

Level 3: To build up individual skills and apply them effectively to group work. The individual skillsinclude chairperson skill (MPS 29), reaching consensus (MPS 57 ), goal setting (MPS 54) and copingwith difficult behaviours (MPS 46).

Level 4: To integrate and apply the skills to team development through benchmarking, diagnosis, andimplementation of plans for growth (MPS 53 on teams and MPS 58 on leadership).

Each of the 15 plus component skills has about a dozen objectives, published measurable criteria andexample assessment tasks. [10,17,26]

The use of the last approach - identifying goals for team and team development - and the explicitdevelopment of the component skills has the advantage of individual accountability and ease inassessment of individual skill development. More is discussed in Part 2 of this paper.

4. Forms of Evidence: performance options to assess team work

Valid assessment is based on evidence: Principles 2 and 4. First we consider how to design a rating form.Then, we describe nine options of evidence.

M Create a well-designed assessment or rating form for team work.

Before any options are considered, a team rating form needs to be created. The rating form should bevalid and reliable.. One of the challenges in assessment for teachers is the design of the rating form. Theitems included in the form must be consistent with the criteria. The rating form should not contain moreitems than are convenient for a rater to consider at any one time, and the form must be related to the goalsand criteria which are described in some detail on the form. An observer assessing a group should not be expected to rate more than two or three people at one time[27.28] and the form should be organized in convenient chunks around the main elements of the processbeing observed. Compare, for example, the two forms in Figure 1. Form A shows only the first entry of26 items; Form B shows the first of 8 items. First of all, expecting an observer to keep track of and rate 26different items is asking for super-observers. We think, in form B, that an observer has a chance toremember and rate only 8 items. Secondly, Form A places these in the category of "helpful behaviors" .Research, summarized in Table 2, reminds us that Task and Morale are the two major issues to considerin group work. These descriptors in Form A sound like task issues but Aarriving on time and prepared@can be construed as a Morale issue. Such ambiguity makes it difficult for the rater. Form B, clearlyidentifies Task as being the issue and identifies one of four Task items for assessment. Each item canhave either positive or negative contributions. In summary, form B is preferred over form A.

It is important to provide external descriptions of the criteria or standards. Otherwise, raters tend to createinternal standards- standards selected based on one=s own performance or a performance judged to besimilar to one=s own. [29,30] Consider Figure 2 that shows four different rating forms. Form 1 gives no

8-16details of the goals; it only lists " effective clinician" and provides no measure of the criteria. The rater isexpected to pick some number between one and ten. What is acceptable or outstanding to one rater, willbe inferior to another. No standards are given, and so each rater will create personal and different internalstandards. Form 2 offers at least some description of the skill being considered: the Aknowledge@ that the clinicianbrings to the patient-doctor encounter. However, the term Aknowledge@ is ambiguous; different raters willuse different internal criteria in choosing the box "meets objectives".

Form 3, in Figure 2, gives more description of the goals by listing three attributes under Aknowledge@ thatan effective clinician is expected to display. Some criteria are given to help the rater understand what"meets objectives" means. However, the term "solid knowledge" and "common problems" are ambiguous.What is solid knowledge to one rater is not solid knowledge to another.

AI rated you low because you knew very little about diphtheria.@ "Diphtheria is not a common problem in this area. Why should I be expected to know about

diphtheria?" responds the resident being evaluated.

Although form 3 is a marked improvement over forms 1 and 2, the form still has ambiguous words.

Form 4, in Figure 2, gives a more detailed and less ambiguous description of what is expected. True, alist is not given of the 38 illnesses but, at least we know that we are expected to know 38 and not just 5.This elaboration on the goal allows the rater to check or circle the particular deficiency from the list thatforms the basis for the rating.

Criteria-referenced, rather than norm-referenced, assessments[31] should be used. The use of norm-referenced assessments means that one=s performance is compared to another=s instead of being comparedto external standards defined by the criteria. Norm-referenced leads to competition that is contradictoryto the cooperative atmosphere expected of groups and teams.

Well-designed rating forms or questionnaires can be used to provide evidence about team effectivenessand be used to assess group growth. Here are nine options for forms of evidence for decision makingabout the overall team effectiveness. M Option 1: Quality of the answer supplied. Teams are usually assigned a task to do. The product could be a written or oral report, a recommendation,a set of plans, goals scored in a sports game, money raised in a campaign, successful trade show or partyor event. In academia, we tend to assess primarily the quality of the technical answer, design report, andlab report. If these are good, we assume the team work was good and vice versa. In the sports analogy, ifmany goals were scored we must have a great team. Unfortunately, this relatively simple way to assessteamwork is usually an incorrect measure of teamwork. We are using the wrong evidence. The product isnot necessarily a good measure of the process. This approach is not recommended.

M Option 2: Individual assessment of group=s accomplishment of the target skills. Table 2, form 2800, provides the evidence-based targets for effective groups and teams. Ask students tomonitor the group=s attitudes and skills. They can indicate the dates when the group achieved differentdegrees of exhibiting the target skills. Although some students may fake it and fill the whole thing in thenight before it is due to be handed in, nevertheless, we have found this to be a very effective focal pointand form of evidence. The assessors can be outsiders (non-group members) or insiders (group membersself assessing their team=s efforts). Dyer [32] suggests that outsiders are usually in a better position to assessthe failure to communicate, the formation of subgroups and conflict. Insiders are usually in a betterposition to assess levels of trust, frustration, personal cautiousness in presenting ideas and lack of

8-17enthusiasm in attending meetings. Saavedra and Kwun [29] have found that outside, passive observersinterpret the intergroup dynamics differently from assessors who are active members of the group. Thisdoesn=t negate the value of having external observers. However, the research suggests that groups shouldvalue feedback from both external observers and self and peers.

M Option 3: External assessors use a rating form for overall team work.Figure 3, form 2801, can be completed by external observers to suggest the positive and negative rolesdifferent members played and to give overall strengths and areas to work on for the working group. Thefocus is on the Task and Morale dimensions and on effective groups (as opposed to teams). Because ofthe difficulty in simultaneously observing more than three people [27,28] and in participating in the groupprocess, we recommend that this type of form be used by a team of external observers. For a team of fivein the working group or Aclient@ we recommend a team of five observer/assessors in the observer group orAobservers@. Each assessor considers the whole working group but focuses especially on two or threepeople. When the working meeting is over, the observer group convenes to share data. Each person in theworking group has been looked at in detail by two persons from the observer group. Within a short time,the observer group can compare and summarize their findings and recommendations. We have usedstudent groups as the assessors. The report from the observer group is usually a one-page summaryform for the group using form 2801 plus one report on each student=s contribution from each of the groupobservers original analysis form 2801. No names identifying the observers shall be given. However, thenames of each individual in the working group being assessed will be given. The instructor initials thereport and passes it on to the client group. There is no direct discussion of the assessment between theclient and the assessing groups. No two groups can be both the client and the observers. That is, if clientgroup a is Aobserved@ by group b, then group b cannot ask to be assessed by group a. We have found it isbetter if group b is assessed by a third group, group c.

Another option is to use external observers who have been trained in assessment. For example, theMcMaster Medical School identifies interested people from the community, from the medical profession,from senior students and from faculty. These people are given an evening workshop on assessment andthen, as teams of three, observe groups through a one-way mirror. The observation is for one hour with anadditional one hour used for the observers to summarize their findings. The cycle is then repeated.Alverno College uses volunteers from business and from the community. [33]

M Option 4: Group self-assessment plus individual reflection of overall team work.Figure 4 gives a variation on form 2801 for use by the group for self assessment. This form (form 2802)focuses on Task and Morale and on effective groups. It can been used for a 10 minute reflection andassessment at the end of each meeting. For three minutes, the chairperson facilitates the assessment of thegroup=s ability with the task and morale components. All must agree and use the same rating for the self-assessment of the Task and Morale. Each member then has 30 seconds to state his/her personalcontributions to the group process. This is not for discussion; this is just each person stating an opinion. For example, a person could say, AI think that I aided the task when I... I was a little frustrated because Ifelt I had good ideas but was not given an opportunity to express them@. During the last four minutes the group lists its strengths and areas to work on. Such evidence can be usedby the group to show growth over the semester as a working group. We have found this to be anextremely powerful and useful form of assessment and evidence. We have used this in the freshmandesign course and the junior level project course.Angelo and Cross [34] suggest Classroom Assessment Technique 47, CAT 47 on Group-work evaluations,to collect feedback about group work. They use this CAT to monitor classroom learning. We propose thatgroups can use such forms to monitor and provide evidence about group growth.

M Option 5: Individual assessment of the team.

8-18For those groups that have shifted from being an effective group to being a team, individual membersshould use form 2803, given in Figure 5. Individuals assess the task and morale characteristics of theteam, add personal comments and indicate areas for growth. We have used this form as part of theevidence in our senior design course. On the form, the individual is reminded of the forms of evidenceupon which the rating is to be made.

M Option 6: Individuals complete diagnostic tests about the team and then team members reachconsensus. Some diagnostic tests that might be used are:

Francis and Young=s 108 item inventory [3]

Dyers= 11 item inventory (p. 69 or p. 75) [32]

Alexander=s 10 item inventory, given in Phillips and Elledge [35]

Jamieson=s 37 item inventory, given in Phillips and Elledge [35]

Quick=s 30 item inventory (p. 6 and another on p. 81) [36]

Johnson=s tests on specific attributes (trust, conflict resolution) [37]

Scholtes's 21 item (p. 7-28). [38]

M Option 7: Individuals complete diagnostic test about the team and then team members reachunanimity.This is a variation on how to handle the individual results of the diagnostic tests listed in Option 6. M Option 8: Individuals complete diagnostic tests about the team and results are averaged.This is a variation on how to handle the individual results of the diagnostic tests listed in Option 6.

M Option 9: Individuals write portfolios about team performance.

Individuals could use the group goals for growth, the criteria and the amassed evidence (from formssimilar to 2802) to write a critique about team performance. Another approach is to ask groups to submitfive summary reports (form 2802) for each of five successive meetings and note the change in ratings andthe shifts in strengths and areas to work on. Additional evidence can be included to support the claims.

5. Team work and project answersTeams of engineers are formed for a purpose. The team is to solve a technical problem using theirknowledge of Chemical Engineering. Hence, at least two elements should be assessed: the overall qualityof the answer and the quality of the team. Evidence related to the technical answer is described in Option1. This evidence should be obtained. The evidence suggested in Options 2 to 9 provide data important for:M Team growth: setting standards, bench marking performance, setting goals and improving.Improvement follows the principles of assessment: set goals and criteria, gather evidence, monitorprogress and measure achievement. M Individual awareness: the individual skills needed to make the whole team work. Teams are acollection of individuals. The individual performance is assessed, as described in Part 2 in this series, butin the context of the overall group.

In summary, although the educational goal might be to improve an individual=s performance in a team, weneed to measure the overall team performance.

6. Options for Action

The general guidelines to implement a program to develop team skills has been given elsewhere.[9,39] Hereare some additional practical suggestions.

8-19M See the development of an individual=s skill in team work as a two-part plan: 1) develop group=s skilland confidence as a team (the emphasis in this paper) and 2) develop the individual=s skill and confidenceas a member of a team (the emphasis in part 2 in this series). Both are needed.

M Take a single three-hour tutorial somewhere in your program and run a training workshop oninterpersonal and group skills. This gives students practice using forms 2800 and 2801. Example goals,criteria, transparencies and timing suggestions are available. [26,40]

M Take several three-hour tutorials somewhere in your program and run a training workshop on self-assessment. This gives students practice in applying the Principles of assessment outlined in section 2.Example goals, criteria, transparencies and timing suggestions are available.[26,40,41] Such training helpsimprove the reliability and the student=s understanding and acceptance of self and peer evaluation. Wefeel that such training is essential if self and peer evaluation is to be used in the program.

M Expect all groups - lab groups (even if it is a team of two), project and design groups, formalcooperative learning groups and problem-based learning groups - to set goals for growth and assessperformance. We have used all four forms, 2800 to 2803, for different aspects of group work in the MPSprogram.[10] Table 3 provides guidance for the selection of the form of evidence and purpose ofassessment most appropriate for your situation.

M Support your students in their efforts to improve group performance. Suggestions are available for thedozen plus issues to address in a Anorms@ meeting. [40] First-aid for different symptoms of ineffectivegroups have been synthesized [42] from Francis and Young,[3] Tiberius [43] Fisher et al., [44] Langford andPostans,[45] and Valiquet[46]

M To evaluate the effectiveness, to improve your program, and to provide evidence for ABET 2000,gather benchmark data and collect sample portfolios about group growth (described in Option 9). Datafrom Form 2800, Table 2 and Option 2, can also provide evidence to document growth.

M Assessing the overall team work is important. Our goal is to have effective teams and not justcollections of individuals. The individuals must pool their skills to create great teams. Assessing theoverall team work is important because this provides the context for our ultimate goal - the developmentof individuals who function well in teams.

7. Summary

1. The five Principles of assessment provide a framework for developing and using instruments forassessment of student performance and of the evaluation of program effectiveness.

2. Crucial to any assessment is the creation of published goals and measurable criteria that form thecontext for the performance of the student. Evidence should be gathered and assessed in the context ofthese goals and criteria. Four example sets goals and criteria were presented.

3. Some example objectives and levels of development for team work are given to provide a context forlearning and assessment.

4. Assessment is based on evidence of performance. Nine options for gathering evidence were described.

5. Practical suggestions for the development of effective teams include developing both overall team skilland skills in individuals, provide training in group skills and self assessment, expect all groups to gather

8-20evidence each time they meet, provide support for their efforts to overcome difficulties and gather benchmarking data.

6. Our ultimate goal is to develop and assess an individual=s skill in working effectively in teams.

8. References

1. Rugarcia, A., R.M. Felder, D.R. Woods and J.E. Stice, AThe Future of Engineering Education: Part 1. avision for a new century,@ Chem. Eng. Ed., 34 (1), 17-25 (2000)

2. ABET 2000, http://www.abet.org May (1999)

3. Francis, D., and D. Young, AImproving Work Groups: a practical manual for team building,@University Associates, San Diego, CA. (1979)

4. Woods, D.R., AProblem-based Leaning: how to gain the most from PBL,@ Woods Publisher,Waterdown ON Canada distributed by McMaster University Bookstore, Hamilton, ON (1994)

5. Alverno College, AAssessment at Alverno College,@ Alverno College Publications, 3401 39th St., Box343922, Milwaukee, WI. 53234-3922 (1985)

6. Alverno College, AStudent Assessment-as-Learning at Alverno College,@ Alverno College Publications,3401 39th St., Box 343922, Milwaukee, WI. 53234-3922 (1994)

7. Boud, D., AEnhancing Learning through Self Assessment,@ Kogan Page, London (1993)

8.. Perry, W.H, Jr., AForms of Intellectual and Ethical Development in the College Years,@ Holt Rinehartand Winston, New York, NY (1968)

9. Woods, D.R., R.M. Felder, A. Rugarcia, and J.E. Stice, AThe Future of Engineering Education: Part 3.developing critical skills,@ Chem. Eng. Ed., 34 (2), 108-117 (2000)

10. Woods, D.R., et al., ADeveloping Problem Solving Skill: the McMaster Problem Solving Program,@ J.Engineering Education, April, 75-91 (1997) http://www.chemeng.mcmaster.ca/innov1.htm and clickon MPS for a summary of the research findings and some details for many of the units.

11. Kirton, M.J., AAdaptors and Innovators: a description and measure,@ J. Applied Psychology, 61, 622-629 (1976)

12. Schutz, W.C., AFIRO-B; a three dimensional theory of interpersonal behaviour,@ Holt, Rinehart andWinston, New York., NY (1958)

13. Jungian typology, see Keirsey, D., and M. Bates, APlease understand me,@ Gnosology Books, Del Mar,CA (1984) http://keirsey.com May (1999)

14. Tannen, D., AYou just don=t understand: men and women in conversation,@ Ballantine Books, NewYork (1990).

8-2115. Ramsden, P., AThe Lancaster Approaches to Studying and Course perceptions Questionnaire:Lecturer=s Handbook,@ Educational Methods Unit, Oxford Polytechnic (now Oxford Brookes University),Oxford, UK (1983)

16. McKeachie, W.J., ATeaching Tips,@ 9th edition, D.C. Heath, Lexington, MA (1994)

17. Chemical Engineering Dept homepage: http://chemeng.mcmaster.ca/innov1.htm MPS program, May(1999)

18. Locke, E.N., et al., A@Goal setting and task performance, 1969 - 1980,@ Psychological Bulletin, 90, a,125-152 (1981)

19. Fisher, B.A., ASmall Group Decision Making,@ Second edition , McGraw Hill (1980)

20. Dimock, H.G., AFactors in Working in Groups,@ AHow to Observe your Group,@ AHow to analyze andevaluate group gorwth,@ APlanning Group Development,@ Concordia University Bookstore, Montreal(1970)

21. Whetten, D.A., and K.S. Cameron, ADeveloping Management Skills,@ Scott, Foresman and Co.,Glenview IL (1984)

22. Reddy, W.B., AIntervention Skills: process consultation for small groups and teams,@ Pfeiffer and Co,San Diego, CA (1994).

23. Hoffman, R.L., E. Harburg and N.R.F. Maier, ADifferences and Disagreement as Factors in CreativeGroup problem-solving,@ J. of Abnormal and Social Psychology, 64 , 206-214 (1962)

24. Boulding, E., AFurther reflections on Conflict Management,@ in APower and conflict in organizations,@R.L. Kahn and E. Boulding, eds., Basic Books, New York (1964)

25. Sandler, B., personal communication (1988)

26. Woods, D.R., AHow to set up courses and course objectives,@ Chapter D in AProblem based learning:resources to gain the most from PBL,@ Woods, Waterdown (1999) distributed by The Bookstore,McMaster University and downloadable from http://www.chemeng.mcmaster.ca/innov1.htm and clickon PBL and download from the resources book.

27. Swanson, D.B., S.M. Case and C.P.M. van der Vleuten, "Strategies for Student Assessment," Chapter27 in "The Challenge of Problem-based Learning," D. Boud and G. Feletti., eds., Kogan Page, London(1991)

28. De Stephen, R. personal communication, Hilton Head Conference on Teaching Thinking, (1985) 29. Saavedra, R., and S.K. Kwun, APeer Evaluation in Self-managing work groups,@ J. of AppliedPsychology, 78, 3, 450-462 (1993)

30. Brown, S., C. Rust, G. Gibbs, ASection 2: Setting Standards,@ in AStrategies for DiversifyingAssessment in Higher Education,@ Oxford Brookes University, (1994)http://www.lgu.ac.uk/deliberations/oscd_pubs/div-ass2.html April (1999)

8-2231. Woods, D.R., AProblem-based Learning: helping your students gain the most from PBL@, 2nd ed.,Woods, Waterdown, ON (1997), distributed by the Bookstore, McMaster University and anddownloadable from http://www.chemeng.mcmaster.ca/innov1.htm and click on PBL and download

32. Dyer, W.G., ATeam Building: issues and alternatives,@ 2nd ed., Addison Wesley, Reading, MA (1987)

33. Alverno College, AThe Volunteer Assessor at Alverno College,@ Alverno College Publication,Milwaukee, WI (undated)

34. Angelo, T.A., and K.P. Cross, AClassroom Assessment Techniques,@ 2nd ed., Jossey Bass, SanFrancisco (1993)

35. Phillips, S.L., and R.L. Elledge, AThe Team-building Source Book,@ University Associates, SanDiego, CA. 9now distributed by Jossey-Bass, San Francisco, CA (1989)

36. Quick, T.L., ASuccessful Team Building,@ AMACOM, American Management Association, NewYork,(1992)

37. Johnson, D.W., and Frank P. Johnson, AJoining Together: group theory and group skills,@ 2nd ed.,Prentice Hall (1982)

38. Shoultes, P.R., AThe Team Handbook,@ Joiner Associates, Madison WI. (1988)39. Woods, D.R. ATeam Building: how to develop and evaluate individual effectiveness in teams,@Reprint, AIChE Annual Meeting, Los Angeles, Nov. (2000) 40. Woods, D.R., AWorkshops@ Chapters B and C in AProblem based learning: resources to gain the mostfrom PBL,@ Woods, Waterdown (1999) distributed by The Bookstore, McMaster University.

41. Woods, D.R. et al., ASelf-assessment in the Context of the McMaster Problem Solving Program,@Evaluation and Assessment in Higher Education, 12, 2, 107-127 (1988)

42. Woods, D.R., F.L. Hall, C.H. Eyles, A.N. Hrymak and W.C. Duncan-Hewitt, ATutored versustutorless groups in PBL,@ American Journal of Pharmaceutical Education, 60, 231-238 (1996)

43. Tiberius,R.G., ASmall Group Teaching: a trouble-shooting guide,@ OISE Press, Toronto, ON (1990)

44. Fisher, K., et al., ATips for Teams: a ready reference for solving common team problems,@ McGrawHill, New York, (1995)

45. Langford H., and T. Postans, Meetings Booklet and Problem Solving Workshop semester course,@Griffith University, Brisbane, Australia, (1985)

46. Valiquet, T., Workshops on Team Skills, course Engineering and Management 5G1, McMasterUniversity, Hamilton (1993)

8-23

Table 1 Features of collections of people, groups and teams

Collections of people;Ameetings@

Effective Groups Teams

Goals Each has own goals;the collection may

have meetings withoutany clear goals

Individuals may haveown goals but there isgeneral acceptance of

group goals

Clear stated goals thatall accept

unanimously.

Roles Each does his/her thing May be unsure of role;may have been

assigned role and doeshis/her best in the role.

Each is very clearabout role

Decision-making Must we make adecision?

By vote By unanimity

Interpersonal conflict Hope it never happens;if it does, ignore.

Have some methodsfor coping effectively

with conflict butconflict still tends to bean unwelcome visitor.

Team has method forhandling; conflict

welcome because itaids in seeing things

from differentperspectives

Trust In myself Developing High

Level of involvement Low, prefer to beelsewhere

Moderate High

Skills supplied to team Tend to put on a happyface and accept

whatever. Rarely letyour true skills show.

Personal assets arerarely known by other

group members

Some unique skills areknown and are usedmoderately by thegroup. The group

prefers to have friendswith similar stylesworking together.

Unique skills are usedeffectively by the

team. Interpersonaldifferences are

recognized as assets tothe team.

Attitude AI@ AI-we@ AWe@

8-24Table 2 Evidence-based targets for group skills (reprinted from references 4,17) Form 2800 .

Evidence-based targets Progress toward internalizing these targets

20% 40% 60% 80% 100%

M Performance improves when we have goals. [18]

M Assessment must be related to the goals [19]

M Both Task (getting the job done) and Morale (feeling good about thegroup work and about how you have interacted with the other groupmembers) are important [19]

M Any group functions better with a chairperson [20]

M Chairperson and leadership are different; different people maybecome leaders at different times.[20]

M Group evolution tends to follow a pattern described as by suchdescriptors as A forming, storming, norming and performing@ [19,20].Schutz=s instrument FIRO-B [12,21] seems to provide reliable insight asto the personal style of individuals towards other group membersduring three of these phases.

M We can list the roles needed in both Task and Morale to make aneffective group. [19]

M When each person has a clear idea of roles and group norms, thegroup functions better. [19]

M When groups are functioning effectively, about 70% of the time isspent on the task; 15% on morale building activities and 15% of taskprocess activities (how the problem solving process is going;summarizing ideas, guiding the process). [22]

M The products from groups or teams is improved when membershave different Astyles@ (in Jungian terminology some members aredominant S, and some, dominant N). The products from groups tend tobe inferior when all the members Athink and behave alike@. [23,24, 19,20]

M The quality of decisions, product, task is improved if group membersoffer different perspectives, disagree and seem to introduce conflict intothe process. The trick is to manage the apparent conflict well. [19,23,24]

M The characteristics of A meetings of individuals,@ Aeffective groups@and Ateams@ fall on a spectrum with sufficient differences that it isuseful to differentiate based on those characteristics

M In a decision-making mode, after 20 minutes of discussion on anyone topic, few new ideas are presented and repetition of previouslystated ideas occurs. [25]

8-25

Table 3 Some options for the use of the assessment forms and for individual and group skill acquisition

General Goal Details Sourcesof

evidence

Comment

1. To improve studentaccountability and thereby

improve student moraleabout what is going on in

the group;To explicitly require groupsto continually monitor theirprogress and to improve.

Group Self assessment: As a group, over asequence of meetings, the task and morale

ratings for the group shall increase by 1 unit(between the first and last meeting). Individuals

must submit their own collection of formsincluding their personal input.

2800,2802

Requires 10minutes aftereach groupmeeting.

Marking loadcan be reduced

by having groupreports at theexpense ofindividual

accountabilityand skill

development.

2. To explicitly requiregroups to continually

monitor their progress andto improve. Because a

single report is expectedfrom the group, individualaccountability is missing.

Group Self assessment: As a group, over asequence of meetings, the task and morale

ratings for the group shall increase by 1 unit(between the first and last meeting). One

summary report is handed in per group witheach person getting the group mark.

2800,2802

3 To explicitly requiregroups to continually

monitor their progress andto improve.

Group Self assessment. As a team, over asequence of meetings, the five strengths will be

maintained and one area to work on will beshifted to a strength (between the first and last

meeting). This can be done requiring singleteam reports or with individual reporting.

2800,2802

4.To explicitly requiregroups to continually

monitor their progress andto improve.

Group peer assessment. An assessment observergroup will complete form 2800 for the team

with each member listed and provide feedbackabout the team=s and individual process. Theresult of this feedback will be compared with

the evidence given by the group in #1 to 3. Peerassessment in addition to self assessment.

2800,2801

Cuts the studentworking time on

the project inhalf if use peer

assessors.

5. To help studentsAunderstand@ what shouldbe happening in a group.The focus is on becomingskilled at observing group

dynamics and gaininginsight into selfperformance.

Use group on group individual observation in aworkshop devoted to becoming familiar with

form 2800 and the group process. Theworkshop takes between 1 and 4 hours. No

direct follow up is expected. Students hand inindividual reports based on the evidence each

received from a personal observer.

2801,2800

Requiresworkshop.

8-26Figure 1: Example options for assessing group and individuals within groups

Form A: First of 26 entries

Names of group members

Helpful Behaviors

Initiates and contributes ideasArrives on time and is prepared

Completes assigned tasksArrives with questions or comments

Shares knowledge

Form B: First of 8 entries

Names of group members

Task

Observer of thetask process

Orients group, monitors, summarizes, seeksdirection, identifies phases in problem solving +

process

Ignores phases, asks whatever he/she wants, -blocks, unaware of his/her contributions

8-27

Figure 2: Optional ways in offering rating scales

Forms:1 unsatisfactory outstanding

1 10Effective clinician: G G G G G G G G G G

2Effective clinician: Unsatisfactory area meets area NAknowledge base requires objectives strength

attentionG G G G

3Effective clinician:

Domain Unsatisfactory Needsimprovement

Meets objectives Exceeds objectives

Knowledge base:growth and

development;family systems;diagnosis and

management of commonpediatric and adolescent

problems

Occasional major orunacceptably frequent

minor gaps inknowledge base

G

Frequent gaps inknowledge

G

Solid knowledge basefor common clinical

problems

G

Thorough knowledge ofall basic areas and goodknowledge in additional

areas

G

4.Effective clinician: Knowledge base: has knowledge base in fundamentals, can recall correctly values from diagnostic tests for healthy infants, children teens, adults,elderly and can give casual relationship between potential diseases and numerical values from diagnostic tests, can list positive and negative testsfor 15 diseases, can list 2 dominant and 3 inferential symptoms for 38 illnesses. can list 15 history-taking questions and prioritize them, cancorrectly hold 13 different instruments and use them with patient satisfaction greater than 95%. ..

None of Few of these Most features All of thesethese behaviours but demonstrated behavioursbehaviours major omissions G G G G G G G1 2 3 4 5 6 7

8-28Figure 3: Rating form 2801 for external observers of individual contributions to the group (reproduced with permission from Woods [4])

Observer _____________________- Client _____________________ Case _____________

Task Group members

Task processobserver;(problem solvingprocess)

Orients group, monitors, summarizes, seeks direction,identifies phases +

Ignores phases, asks whatever wants, blocks, unaware ofcontributions _

Information orOpinion Giver(related to task)

Assertively gives information, makes suggestions +

Withholds information, silent, aggressive, passive _

Information oropinion Seeker(related to task)

Asks questions, checks comprehension +

Silent, refuses to ask questions _

EnergizerRisk taker

Enthusiastic, introduces a spark, novel ideas +

Follower, agrees, silent unsure _

Morale

Observer ofinterpersonalprocess

Sensitive to interpersonal dynamics, comments on what’sgoing on +

Ignores conflict, tension, hopes it’ll go away _

Giver of praiseand support

Warm, responsive, gives help, rewards +

Put downs, aggressive, self-centered, defensive, critical,contemptuous _

Interpersonalproblem solver:Seekerssolutions

Mediates, harmonizes, helps resolve conflicts +

Causes problems, seeks personal goals _

Energizertension relief

Jokes, laughs, shows satisfaction +

Withdraws, causes tension _

Strengths Areas to work on___________________________________ ____________________________________

___________________________________ ____________________________________________________________________________________________________________________________________________

8-29Figure 4. Rating form 2802 for group self assessment including an opportunity for individual reflection (reproduced with permission from Woods[26] )

Form 2802Task: Problem defined, many issues and hypotheses explored, criteria listed and the issuesprioritized. Refrained from early closure. Task carried out and looked back at the result to assessit. Group agreement as to goals. Process was active with monitoring. Completed task on time.The accuracy in the group’s answer matched the time available. Group avoided contributingexcessive information. None of Few of these Most features All of thesethese behaviours but demonstrated behavioursbehaviours major omissions

G G G G G G G1 2 3 4 5 6 7

Morale: Group relaxed; enjoyed working together. They gave emotional support to each otherand were able to express disagreement or disappointment directly. Seven fundamental rightspreserved. Members are enthusiastic and involved. None of Few of these Most features All of thesethese behaviours but demonstrated behavioursbehaviours major omissions

G G G G G G G1 2 3 4 5 6 7

Individual Contribution to Task and Morale

____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Group Strengths Group Areas to work on_____________________________________ ________________________________ ____________________________________ _____________________________________________________________________________________________________________________________________________

from D.R. Woods (1995

8-30Figure 5 Rating form 2803 for individual assessment of teams

Assessment of your team Name: _________________

M Teamwork: Task all members clear about and committed to goals; all assume roles willingly; all influence the decisions; knowwhen to disband for individual activity; all provide their unique skills; share information openly; the team is open in seekinginput; frank; reflection and building on each other's information; team believe they can do the impossible; all are seen aspulling their fair share of the load.

The degree to which these descriptors describe your team's performance (as substantiated by evidence: meetings,engineering journal, interim report, presentations). None of Few of these Most features All of thesethese behaviors but demonstrated behaviorsbehaviors major omissions

G G G G G G G1 2 3 4 5 6 7

M Teamwork: Morale: Trust high, written communication about any individual difficulties in meeting commitments; cohesivegroup; pride in membership; high esprit de corps; team welcomes conflict and uses methodology to resolve conflicts anddisagreements; able to flexibly relieve tension; sense of pride; we attitude; mutual respect for the seven fundamental rights ofall team members; Absence of contempt, criticism, defensiveness and withdrawal.

The degree to which these descriptors describe your team's performance (as substantiated by evidence: meetings,engineering journal, interim report, presentations). None of Few of these Most features All of thesethese behaviors but demonstrated behaviorsbehaviors major omissions

G G G G G G G1 2 3 4 5 6 7

Additional comments:____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Strengths of your team Areas to work on________________________________________ ________________________________________________________________________________ ________________________________________________________________________________________________________________________________________________________________ from D.R. Woods (1997) Attach any pertinent additional evidence that

we should consider. Thanks.

8-31

1 Current address, Department of Chemical and Materials Engineering, University ofAlberta, Edmonton, AB T6G 2G6, Canada <sdtaylor@ualberta.ca>

2The Procter and Gamble Co., 8256 Union Center Blvd., West Chester, OH, 45069, USA <jaffer.sa2@pg.com>

Assessing team work. part 2: individual contributions to the team

Donald R. Woods, Shawn Taylor1, and Shaffiq Jaffer2

Chemical Engineering Department McMaster University [a128]

Hamilton ON June, 2001

Abstract:

Individual contributions to team work should be assessed to keep individuals accountable and todevelop individual skill. Four options are given for the goals and criteria for individualcontributions. Thirteen forms of evidence are listed. Six relate to one’s contribution to thetechnical result. Five forms give evidence about an individual’s contributions to the teamwork.Two options provide evidence about skill in self-assessment.

Rationale is given for the need to assess both the technical and team aspects of team work.Examples are given of how teamwork was assessed in the McMaster Problem Solving programin both a junior level project course and in a capstone project course. Data are given to illustratethe types of evidence collected and the relationship among the evidence.

Suggestions are given about program evaluation and gathering evidence for ABET 2000accreditation.

Keywords:Assessment, group work, teams, goals and criteria, forms of evidence, rating forms, ABET 2000, capstone course,exit surveys, program outcomes.

8-32

He shoots; He scores! The crowd goes wild as the home team wins. Yes, it is vital that the team performs well. Inaddition to the wins and losses of the team, individual records are kept for each player: the number of penaltyminutes, the goals scored, the assists, the plus-minus and the goals against. Based on lousy individual performancethe player will be benched, traded or sent to the minors. Individual contribution to the team can and should bemeasured.

In Part 1 of this series, [1] we differentiated between effective groups and teams, suggested methods of developingskill in group work and reviewed the elements of assessment. Then we considered options to assess the overall groupor team performance. In this part 2, we consider how empowerment and accountability are linked, list goals andobjectives for individual skill development in team skills, give options for forms of evidence, the merits of self andpeer assessment, our results from using some of these approaches and offer ideas about how to select assessmentconsistent with your context. These suggestions are directly applicable to satisfying the ABET 2000 [2] criterion 3.dwhich requires that our graduates have “an ability to function on multidisciplinary teams.”

1. Why it makes sense to assess individual contributions to the team: Empowerment and individualaccountability

When students are empowered to operate as autonomous groups to complete a task, then each member of the groupshould be accountable to the group and to the person who created the groups in the first place, namely the teacher.Being assigned to work in an autonomous group or team does not give any student the right to shirk his/her duties, toopt out of being responsible to the group, or to hide in the group.

To continue with the sport analogy, the contribution of each player to the team effort is clearly visible to the fans,owners and coach. So it is with team work; we need to make the individual contribution to the team visible.

Hence, the four major reasons why we need to measure and mark individual performance are 1) to make individualsaccountable, 2) to allay the resentment and frustration that working members have when they think that others arenot pulling their weight, 3) to acknowledge those who make outstanding individual efforts that may not berecognized by other group members because the others do not see that effort, and 4) to make team skills anindividually-achievable outcome from engineering programs. Making individuals accountable creates a climate forgood team work and development.

2. Goals and Criteria for Team players

Consistent with Principle 4 for assessment described in Part 1, [1] goals should be published to describe the targetskills and attitudes of individuals who participate fully and effectively on teams. In Part 1 we listed options forsetting goals for teams, for gathering evidence about team behavior and for assessing team effectiveness. Here weconsider the goals for individuals in their quest for skill as a team player. Although initially we might be satisfiedwith a general objective [3] like “the ability to function on multidisciplinary teams” we have found it extremelybeneficial, especially for the purposes of assessment, to elaborate on the skill. Such elaboration should be based onresearch findings [1] about the performance of effective team players. Here are some options for creating goals andcriteria for individual skill in participating in teams.

! Create a list of descriptors of the process:An example list could be the individual is skilled at both the process Task and Morale components of teamwork.Elaboration on these components was given in the companion section 3 of Part 1 in this series. The disadvantage ofthis approach is that the measurable criteria are not published.

! Create a list of target skills and attitudes:An example list for team skills was given in Table 1 of Part 1[1] of this series As with the previous approach nocriteria are given.

8-33

! Provide a structured list of goals moving from beginning to advanced: Alverno College [4] identifies six levels of goals as applied to the development of an individual’s skill in team work.

Level 1: Identify own interaction behaviors utilized in a group problem solving situation. (Eleven behaviors for thetask activities used as criteria; observation by peers where strengths and weakness are noted; agreementbetween observer and client).

Level 2: Analyze behavior of others within two theoretical frameworks: task and morale components.Level 3: Evaluate behavior of self within two theoretical frameworks for at least three different, videotaped

situations. Self, peer and teacher assessment. Growth contract created.Level 4: Demonstrate effective social interaction behavior in a variety of situations and circumstances.Level 5: Demonstrate effective interpersonal and intergroup behaviors in cross-cultural interactions.Level 6: Facilitate effective interpersonal and intergroup relationships in one’s professional situation.

Forms of evidence and criteria are available.[4]

M List component skills and create sets of behavioural, unambiguous goals with measurable cruiteria for each: For example, in the McMaster Problem Solving program[5], MPS, the five levels of development for individual skill(and the corresponding MPS units) are:

Level 1: Self awareness, self acceptance and acceptance of personal style and preferences of others (MPS 11).Level 2: Awareness of attributes of successful groups/teams for the two theoretical frameworks of task and morale:

(MPS 28); Groups can self assess, set goals for growth.Level 3: Self awareness of own contributions to the group/team (MPS 28; 29). Individuals can state his/her

contribution to the group process. One can gather evidence about his/her effectiveness in role aschairperson and set goals for growth. One understands the implications of FIRO-B for personal contributionto group evolution.

Level 4: Being an effective member of a “good group”: demonstrate trust and skill in conflict resolution, problemsolving, giving and receiving feedback and assertiveness (MPS 49, 26, 24, 43 and 44).

Level 5: Being an effective member of a team (MPS 53, 54, 47, 46 and 58).

Objectives, criteria and forms of evidence are published for the individual MPS units.

3. Forms of evidence: options for documenting and measuring individual contributions to the team

The Principles for the design of well-designed assessment or rating forms for individual contributions to teams wereoutlined in Part 1[1] of this series.

Fourteen options and variations have been suggested. The first six options relate primarily to the technicalcontribution to the final result, report or product. Options seven to twelve relate primarily to the process skills ofhow the individual functioned in and contributed to the team. Options thirteen and fourteen address self assessment.

M Option 1: Quality of the team’s written technical report or presentation. The product or project report is marked and then:a. The same mark is given to everyone on the team “because this is a team.”b. The mark on the technical report is distributed among the team members on some basis. Some options include:

- All get the same mark but students who do not attend the team meetings are asked by the instructor toleave the group. [6] Those without a group produce their own reports. This provides one mechanism for theremoval of a delinquent member. The conditions and consequences must be spelled out clearly at thebeginning. The teacher issues a letter to the delinquent student after consultation with the group. Theoverall mark in the course must allow a student to pass based on the individual effort. For example, theteamwork counts 20%, the project report 80%.

8-34

- The total technical mark is allocated among the individuals. The group, the tutor, or the student leaderdecides the marks of individuals at the end of the project. If group mark is 62/100, and for a group of 5members, then the total number of marks to distribute are 5 x 62 = 310 marks. [7] This approach violatesassessment Principle #4: [1] there are no published goals and criteria for group work. This also violatesPrinciple #2:[1] no quantitative evidence is used. Indeed, the very idea of having to divide up the marksintroduces competition rather than cooperation.

- The total technical mark is allocated among individuals based on group criteria. The group decides on the criteria at beginning of project and uses the criteria at the end of the project to distribute marks.[7] Thisapproach is wrong because the group mark is a part of the technical mark. Furthermore, the approach doesnot include the identification of the target skills (contrary to Principle #4) nor is the form of evidence to beused identified (violating Principle #2 [1]). Furthermore, all of the assessment is done at the end of theproject. Data and evidence should be gathered throughout the project.

- The total technical mark is allocated equally among individuals unless they meet the contract conditions.At the beginning, the group contracts with each other that all will contribute the same to each task.[7] Rarelydo individuals contribute the same to the team. Some show leadership in computing; others are better atwriting; others prefer to research the literature. Each brings different skills to the team. Each should contractto contribute their specialty, and not to do the same.

Regardless of how ingenious one becomes at allocating the marks, all these approaches are incorrect. If team work isa valued skill then there must be two separate assessments and two separate marks (not some division of thetechnical marks).

M Option 2: Individual oral presentation of and defense of the technical results.

Individual oral presentations of the technical findings could be assessed by the client, outside assessors, teachers,peers or self. The criteria could be defined to include communication skill, technical competence and ability to thinkon feet as one responds to questions. These refer to the individual contribution to the technical task. This is animportant issue. However, these are not related to team skills.

M Option 3: Combination of peer and self assessment of individual’s technical contributions.

A form similar to Form 2804, given in Table 1, lists pertinent contributions to the technical results and givesmeasurable criteria for at least three levels of performance for each. Privately, individuals rate themselves and theirpeers. The results are given directly to the instructor who collates or summarizes the data. These data can begathered periodically throughout the semester and at the end. Form 2805, given in Table 2, can be used to self assessthe professional, problem solving and communication skills brought to the project.

M Option 4: Individual Test and Exams on Technical Knowledge, TETK.

Exams and tests can be created to give individuals a chance to demonstrate that they understand the technical detailsof the project. [7] These tests can be written or oral and given as spot-checks or formally built into the program. Allstudents could be expected to take the test.

An option is to randomly call on one individual from each group. The individual’s performance becomes the group’sperformance. This approach is used in cooperative learning groups to ensure that all of the individuals have learnedthe subject knowledge.

M Option 5: Individual log/journal of the project: engineering log book.

The log book is a combination of a reflective journal, daily goal setting and assessment, project planning andmanagement, detailed documentation and record of contacts, and calculations. We have used this for several

8-35

projects. Some example criteria that could be used in assessing the document were:Professionalism: (40%) documentation of progress, goals, degree achieved, new goals; standardsmaintained throughout, initiative in locating unique sources of data, trying different approaches and keepingan open mind; completeness: equations included, units given, nomenclature defined; quality ofdocumentation: names, e-mail addresses, phone numbers, date contacted, questions asked, answers suppliedand follow up planned; comments on reasonableness of approach and answers; comments about thelimitations and errors; dating and initialing every page.

Problem solving: (20%) active with charts, diagrams, comments; monitoring statements about rationale,progress and process; systematic and organized; focus on accuracy and checking and double checking;flexible and variety of problem solving skills used; use of criteria for decisions.

Communication: (40%) readable and understandable, index and table of content; references welldocumented, organized.

The log books, or engineering daily log books, can be marked part way through the project and at the end. Thecriteria used to mark the log books are the same as the ones used in the self assessment form 2805. M Option 6: Individual written critique of the project (a viva).

Individuals write a personal critique of the project: what went well, how would you improve, satisfaction withresults. Brown and Pendlebury [7] call this a “viva”. This is an interesting form of evidence. However, the goal andcriteria that provide the context for creating this evidence is unclear. Is the evidence to show that:1. The students understood the technical project?2. The students could assess the team activity?3. The students could develop skill in self-assessment?

******************************M Option 7: Peer and/or self assessment of “role” played in team meeting.

Individual roles (such as chairperson, reporter, researcher, checker, skeptic and computer specialist) can be assignedby the teacher or by the group. The roles can rotate or remain fixed throughout the project. Goals and criteria foreach role should be defined at the beginning. Evidence/feedback about how effectively each played the role shouldbe gathered from each meeting. For each role, there is reporting and inter-role reporting and responsibility thatshould be included at the end of each meeting. In other words, “I was unable to play my role very well becauseothers refused to get information to me on time.”

Example data for “chairperson,” objectives, criteria, forms of evidence and self assessment of the evidence areavailable.[5,6]

M Option 8: Test and Exams in Process Skills, TEPS.

Based on the goals given in Part 2, create examinations that test the achievement of the published objectives for teamskills. Some example questions are given in Resources Chapter D [6] and in Table 3.

M Option 9: Self assessment of achievement of target skills for team players.

Form 2800 given in Table 2 [1] in Part 1 lists the target for group skills. As described in Part 1 of this series[1] theform was completed for the team as a whole. For individual assessment the form can be completed by individualsperiodically throughout the semester.

M Option 10: Combination of peer and self assessment of individual contributions to the team process.

8-36

A form similar to Form 2804, given in Table 2, lists pertinent contributions to the team work and gives measurablecriteria for at least three levels of performance for each.

M Option 11: Reflective assessment journals about individual contributions to team work .

The goals and criteria for individual contributions to the team are created, as suggested in section 2. Individualsgather evidence about his/her own contributions to the team: reflections, peer feedback, agendas, documents broughtto the group for a task. Individuals gather evidence from the reflections after each group meeting, add reflectionsabout what they personally did to help the team grow and assess the degree to which he/she has achieved theobjectives. Examples are available.[8]

M Option 12: Portfolio about individual contributions to team work.

A portfolio is a collection of evidence of “best” work. Mourtos,[9] for example, asks students to include assignments,tests, enrichment activities and an overall reflection about the learning. Whereas Option 11, reflective assessmentjournals, is structured around faculty-generated goals for team work and assessment, the portfolio tends to allowstudents to select materials that represent their best work. Ideally the Portfolio should have a clear summary by thestudent of the skills they have achieved as demonstrated by the materials included in the portfolio. Mourtosrecommends the development of extensive marking criteria and provides a general format for the portfolio. He foundthat markers spent 30 to 45 minutes marking each portfolio. We would add that the marking criteria should bepublished and given to the students at the beginning of the semester.

M Option 13: Feedback or self reflection about the assessment process.

As individuals maturing and move toward being an effe4ctive team player, each gains skill in self awareness, selfacceptance and self assessment. The form in Table 4 can be used to provide evidence about skills in self assessmentand in assessment.[9] This reminds us of the Principles of Assessment outlined in Section 3 of Part 1.[1]

M Option 14: Feedback about the assessment process via a personal interview.

Individuals submit a summary of the evidence and their assessment of their performance. Teachers then use this as abasis for privately reviewing the performance with each student.[5] This provides excellent feedback about the selfassessment process and prepares the student for future performance reviews in industry.

4. Team skills and Technical Answers

In Section 2 of both Parts 1 and 2 of this series, we described the goals for generic skill in working in teams. Theseapply to any professional in any profession and to any one. The generic skills need to be applied successfully intackling tasks in subject domains. Engineers should use generic skills in teamwork when working on engineeringproblems so as to produce a superb technical answer. An outstanding team can produce a lousy answer if the teammembers do not have an excellent grasp of the fundamentals and practical considerations of engineering. Similarly,an excellent engineering solution could come from a lousy group.

We should gather evidence about the team’s overall performance and the team’s answer to the problem (as outlinedin Part 1 [1]) as well as the evidence about the individual’s technical contribution and his/her contribution to theteamwork. For the individual contribution, Options 2 to 6 can be used as evidence about the former; options 7 to 11,for the latter. Since self assessment provides evidence, we recommend that the assessment process be assessed aswell. Options 12 and 13 describe methods to determine the student’s skill in self assessment.

5. Example Assessment of Teams and Team work.

We have applied these principles of assessment and selected options in two project courses. Before both projects, the

8-37

students had workshops on interpersonal skills, group process, conflict resolution, problem solving andcommunication.[5] In these workshops the students used forms 2800 to 2802 extensively. These were described asoptions 2 to 4 in Part 1.[1]

A. Junior level project with a local company.

The task was given by the instructor who coordinated the activity with the client company. The students self selectedtheir groups of two to four. The project lasted for four weeks. We encouraged students to use form 2802 after eachgroup meeting. We required them to complete form 2804, described in Options 3 and 10, part way through theproject and at the end of the project. Data from Options 3 and 10 were gathered to improve individual accountabilityand were not used in determining the final mark. This worked well. Groups that initially were underperformingimproved dramatically. Details of this approach, given in a slightly different context, are described elsewhere.[10] Weused Option 8(TEPS) to assess individual skill in team work. This contributed between 10 and 20% of the finalmark.

B. Senior level capstone project with a local company.

In prerequisite courses that are part of the MPS program,[5] TEPS (Option 8) were used to assess individual skill inteam work; chairperson skills (Option 7) were assessed using reflective journals (Option 11) and TEPS. Options 12and 13 were used.

In this course for 15 to 20 students, a client company posed ten tasks. Students selected one of the tasks and whothey wanted as team members. The company supplied a contact person for each project. Extensive interaction, plantvisits and onsite data collection occurred. The classroom contact hours allotted to the project was 40 hours availablefor student-run team meetings, communicating with the company contact and visiting the site.

The technical aspects of the team project were assessed based on the executive summary and written project reportfrom each team. Individual contributions to the team were assessed based on:- form 2804 that was used part way through the project and at the end of the project (Option 3); - the individual oral presentations to the client company that were assessed by the client (Option 2);- the engineering log book (Option 5) that was independently marked by two assessors; - form 2805, the self assessment of the skills reflected in the log book (0ption 3).

The team and individual contributions to the team work were assessed based on:- form 2804 that was used part way through the project and at the end of the project (Option 10); - form 2803 that was used at the end of the project (Option 5, described in Part 1 of this series).

In our context, the team skills were assessed primarily in prerequisite courses, and we used the evidence primarilyfor individual accountability and monitoring. Hence, for this capstone course, the mark allocation focused only onthe technical contributions. The mark breakdown was:

- 10% team mark for the interim progress report; - 30 % for the individual daily engineering log book marked independently by two markers based on the

criteria outlined in form 2805: professionalism, problem solving and communication;- 10% team mark for technical content of report as marked by the company based on a team presentation atthe company;- 10% individual mark on presentation skill as marked by the company based on the presentation at thecompany;- 40 % team mark for technical content as independently marked by two markers.

The final grades ranged from 51 to 97% with the arithmetic average of 86.7 and a standard deviation of 10.1.

If the team skills and individual contribution to the team are not assessed in other courses, then an illustrativeallocation of marks might be 50% marks for teamwork and 50% for technical:

8-38

- 50% of the above marks for “technical contribution;”- 50% for team work with a possible breakdown as follows:

- 5% individual assessment based on TEPS (Option 8);- 5% individual assessment of rotating chairperson activities with individual reports (Option 7);- 5% self assessment of acquisition of target skills, form 2800 (Option 9); - 5% team mark of a team report on the quality of and growth of team based on a series of forms2802, (Option 4, part 1) together with the average of individual self-assessments of the team, form2803; - 10% self assessment of individual contributions, form 2804, Option 10;- 10% average of the peer assessment of individual contributions, form 2804, Option 10;- 10% individual reflective journal about individual contributions to team work, Option 11.

Since self and peer assessment are significant forms of evidence, we suggest that Option 13, a personal interviewabout the performance on the team, be included.

This suggested allocation of marks follows assessment Principle 5: many different forms of evidence are used.

6. Elaboration on the use of Forms of Evidence for Teams and Team skills

Example data are given for self and peer assessment of individual contributions using form 2804. We illustrate howself and peer assessments relate to the overall final grade and to each other.

A. Example data for Form 2804.

Form 2804 lists a set of desired behaviours and criteria. Each student rates him/herself and all of the other members.The rating forms were given directly to the teacher and not seen by anyone else.

Table 5 is an example of the results for Maria’s group. First, the students rated themselves as a group 8.3 out of 10.The arithmetic average of the scores for all elements on the form are given in the body of the table with the self-rating given in bold. Thus, Maria rated herself highly at 9.5 yet her overall average rating for all members of thegroup was 8.22. Maria was rated highly by the other members with all ranking her as first. I have added the rankingto each entry. That is, Brad rated Marie as 9.2/10, Jason 6.9/10, Suzelle 7.6/10, Margarite 6.2/10 and assigned thesame average score for himself. Based on these ratings by Brad, Marie is ranked first; Suzelle, second and so on. These are typical results from a “fair” group.

These results were obtained early in the semester. Once everyone realized that this form was some measure ofaccountability, this group started to communicate with each other and their performance improved. This table isrepresentative of the results from the seven other autonomous groups. This particular self and peer rating formworked well in that:

- The scores are relatively consistent. In this example, except for Margarite, the self rankings wereconsistent with the peer rankings.- Just the fact that such rating forms were completed and the results seen by the teacher meant animprovement in individual accountability. Individuals could not get away with poor performance withoutreceiving low ratings from their peers. The diligent workers feel satisfied that their dissatisfaction with theirpeer’s performance is reflected through their ratings that, in turn, are seen by the teacher.

B. Relating Self assessment and the Final Grade. Data from self assessment from form 2804 correlates well with the final grade R 2 = 0.922. Data from the peerassessment from form 2804 correlates well with the individual final grade R 2 = 0.924.

C. Relating self and peer assessment from Form 2804.

The self and peer assessments correlate with a correlation coefficient of R 2 = 0.972. A paired t-test indicates that the

8-39

self and peer assessments show no significant difference (p < 0.009).

D. Teacher’s marks for log books and the student’s self assessment, Form 2805.

Students complete the engineering log book and independently self assess their skills based on the same criteria usedto assess the log book. The correlation coefficient between the marks by two instructors of the log books and the selfassessment of their skills is R 2 = 0.47.

E. Student assessment of overall team effectiveness.

The students assess the quality of the teamwork in two independent inventories: one question on Form 2804 andform 2803. The inventories were used on two different occasions. The correlation coefficient between the tworatings is R 2 = 0.923.

F. Individual growth from junior to senior year team activities.

A sample of students (N = 12) worked in teams in the junior year and then functioned in different teams in adifferent course in the senior year. A paired t-test showed statistically significant improvement in the individualaverage of peer and self ratings from the junior to the senior year (p < 0.0042).

G. High performing teams.

For those teams that rate themselves highly (on either of the instruments discussed in Section E), the averagestandard deviation of the ratings from self and peers for individual performance is smaller than it is for self and peerratings of students in poorly functioning teams. For example, the average standard deviation of the scores forindividual performance was 2.0 for a team rated as 2 out of 10; the standard deviation for individual performancewas 0.24 for a team rated as 9.3 out of 10. The correlation coefficient between the team rating and the averagestandard deviation of individual performance ratings was R2 = -0.87.

7. ABET 2000: assessment and program evaluation

ABET 2000 has given praiseworthy leadership in requiring us to do what we should be doing all along to provideongoing evaluation of our programs. Individual Departments can set goals, establish criteria, gather evidence andmake claims. This ongoing Total Quality Management approach is a great step forward in our approach to education.Following the guidelines from this paper can make it easier to satisfy criterion 3d, the creation of graduates with “anability to function on multidisciplinary teams” and to continually improve our efforts.

The evidence can include the evidence used to assess the students for their skill (or to give students individual “marks” on teamwork):

- the assessment evidence used by students and by teachers to demonstrate achievement of theclearly stated goals; - portfolios or reflective journals/reports produced by students to support claims of achievement;- results of TEPS.

the evidence of a long range, scholarly plan to develop the skills: - documentation of the goals, criteria and Departmental plans (described in Section 2);- example TEPS and the marking criteria;- analyses of data gathered from forms 2804 and 2803.

the evidence of the impact and comparison with benchmarks measured internally and externally:- an exit survey: Queen’s University [11] has developed an excellent exit survey questionnaire. Thekey question is

8-40

“Please indicate the degree to which your education at Queen’s contributed to yourlearning and development in each of the following areas: The rating is 1 = very little; 5 =a great deal.l. ability to work well with others"

Data for engineering from 1994 are available as benchmark data.[11] For the MPS program, the results of the exitsurvey were 4.72 with a standard deviation of 0.46. Some benchmark data from other programs are 4.14 [0.69], 4.53[0.96] [12] and 4.14. [11]

8. Summary

1. Individual performance in a team should be assessed. This holds individuals accountable, minimizes resentment,allows individual’s efforts to be identified and helps build skill in teamwork.

2. Any assessment requires published goals and measurable criteria. Four options are given that vary in their degreeof explicitness.

3. A commonly misused form of evidence is an allocation among team members of the team mark for the technicalproject. At least two assessments and marks should be given: the quality of technical project and the quality of theteam work.

4. Five forms of evidence are listed of the individual’s performance of the technical subject aspects (chemicalengineering).

5. Five forms of evidence are given of the individual’s contribution to the team work.

6. Because written self assessment is such a valuable form of evidence, two options are given to assess skill in selfassessment.

7. A description is given of how team skills were assessed in a junior level project: forms of evidence selected andthe contribution toward the final mark.

8. Details are presented of how team skills were and could be assessed in a senior capstone project course.

9. Exit surveys provide useful evidence of the effectiveness of programs to develop team skills. Our graduands ratedour program as contributing 4.72 out of 5 in the development of the ability to work with others. Other benchmarkdata show values of 4.14, 4.14 and 4.53.

10. Suggestions are given about various data that can be used for overall program evaluation, such as the ABET2000 evaluation.

9. Acknowledgments

We thank three colleagues from McMaster University Professors C.M. Crowe and A.N. Hrymak, both fromChemical Engineering, and Professor R.C. Hudspith, Engineering and Society Program, for their assistance inwriting this paper.

10. References

1. Woods, D.R., S. Taylor, and S. Jaffer, “Assessing Team work, Part I: the groups’ performance,” ChemicalEngineering Department, McMaster University, Hamilton ON Sept (2000)

8-41

2. ABET 2000, http://www.abet.org May (1999)

3. McKeachie, W.J., “Teaching Tips,” 9th edition, D.C. Heath, Lexington, MA (1994)

4. Alverno College, “Faculty Handbook on Learning and Assessment,” Alverno College Publications, 3401 39th St.,Box 343922, Milwaukee, WI. 53215-3922 (1977)

5 Woods, D.R., et al., “Developing Problem Solving Skill: the McMaster Problem Solving Program,” J EngineeringEducation, April, 75-91 (1997) http://www.chemeng.mcmaster.ca/innov1.htm and click on MPS for a summaryof the research findings and some details for many of the units.

6. Woods, D.R., “How to set up courses and course objectives,” Chapter D in “Problem based learning: resources togain the most from PBL,” Woods, Waterdown (1999) and downloadable from http://www.chemeng.mcmaster.ca/innov1.htm and click on PBL and download from the resources book.

7. Brown, G., and M. Pendlebury, “Assessing Active Learning, Parts 1 and 2,” CVCP Universities’StaffDevelopment and Training Unit, Resouce Administrator, USDTU, Level Six, University House, Sheffield, SI0 2TN,England (1992)

8. Woods, D.R., “How to assess,” Chapter F in “Problem-based Learning: Resources to gain the most from PBL,”Woods, Waterdown, 2nd edition (1999) distributed by McMaster University Bookstore, Hamilton ON

9. Woods, D.R. et al., “Self-assessment in the Context of the McMaster Problem Solving Program,” Evaluation andAssessment in Higher Education, 12, 2, 107-127 (1988)

10. Woods, D.R. "They just don't pull their weight" Chapter for the book “Case Studies on PBL,” P. Schwartz andS. Mennin, eds., Kogan Page (2000)

11. Office of the Registrar, “Undergraduate Learning Experiences at Queen’s: results from the exit poll,” Queen’sUniversity, Kingston, ON (each year since 1994)

12. Hudspith, R.C., “Improving the Bridge: Making Engineering Education Broader and Longer,” Engineering andSociety Programme, McMaster University, Hamilton and “Engineering and Society Programme Review,” Report,July 2000

8-42

Table 1 Form 2804, Self and peer assessment of contributions to team work (developed by C.M. Crowe and usedwith his permission)

______________________________________Name

Assessment of the group

(10) Group worked well together, making the project more interesting. (5) Group usually worked well together, with occasional difficulties (0) Group could not work together at all, making the project moredifficult

Please enter a mark from 0 (worst) to 10 (best) in each row, for yourself and the other members of your team or group. If you cannot make anassessment, enter U for Unable to Observe. Your assessment should be about performance and should be based on evidence you have seen orheard.

Enter your initials in the first cell to the right and those of the other members ofyour group in the remaining cells

self

Leadership

(10) Provided direction, inspired the others ( 5 ) Willing follower, took direction easily ( 0 ) Frustrated the group, blocked progress, criticized the others

Cooperation

(10) Worked readily with the others, contributed to the group work ( 5 ) Cooperated with occasional prompting ( 0 ) Rarely contributed, worked mostly alone, had to be coerced into cooperating or would not cooperate.

Initiative

(10) Produced good ideas which helped others, solved problems ( 5 ) Accepted other’s ideas and improved on them ( 0 ) Criticized other’s ideas, never contributed his/her own ideas’

Attitude (10) Positive, encouraging the others to work better ( 5 ) Neutral, worked well with the group without enthusiasm or grumbling ( 0 ) Negative, complained about the project, worked unwillingly with the group

Computation

(10) Helped others to understand and use computer tools better ( 5 ) Good but not innovative in using the computer tools ( 0 ) Uninterested or unable to use computer tools effectively; neededconstant help to become useful at computing

Effort

(10) Worked very hard on tasks assigned by the group or on his/her own ( 5 ) Worked reasonably hard, given other courses and commitments ( 0 ) Did not work much at all, tasks were either not done or were unsatisfactory

8-43

Enter your initials in the first cell to the right and those of the other members ofyour group in the remaining cells

self

Research

(10) Found many additional published papers, contacts or internet sites which greatly helped the project ( 5 ) Read only the assigned material and used that knowledge effectively ( 0 ) Had not read any material about the project, relied on others for information

Communication and written reports

(10) Organized and wrote major parts of the report ( 5 ) Contributed his/her share of the writing of the report ( 0 ) Made little or no contribution to writing of reports.

Individual assignments (10) Did all of them on his/her own, helped others ( 5 ) Did most of them, perhaps with some help from others ( 0 ) Either did few of them or most of them handed in were copied.

Pertinent technical/subject Knowledge

(10) Excellent grasp of the technical fundamentals for this project and shared and used this effectively ( 5 ) Reasonable understanding of the basics and used this effectively ( 0 ) Little or superficial understanding, relied on others

8-44

Table 2. Form 2805, Self assessment of engineering day books for professionalism, problem solving and communication.

Self-assessment Name: _________________

M Professionalism: set goals, monitor weekly progress and document, maintain consistent standards of performance each week throughoutthe project; show initiative; keep an open mind; calculations and work are complete and correct (equations given, units specified, nomenclaturedefined), comments about reasonableness of answers, about the assumptions, and about the limitations, errors included; date and initial each page.

The degree to which these descriptors describe your performance (as substantiated by evidence: meetings,engineering journal, interim report, presentations). None of Few of these Most features All of thesethese behaviors but demonstrated behaviorsbehaviors major omissions

G G G G G G G1 2 3 4 5 6 7

M Problem solving: Active: make checklists, charts, lists, diagrams; Monitor: ask why? identify purpose and interpretation of the PSprocess you are using; Systematic and organized: Focus on accuracy: check and double check; use many different methods; Flexible andkeep options open: Use a variety of PS tactics and skills: brainstorm, use explicit criteria Define: focus on defining the problem instead ofactivity with unknown purpose; willing to spend at least ½ the time to ensure that you are working on the right problems.

The degree to which these descriptors describe your performance (as substantiated by evidence: meetings,engineering journal, interim report, presentations). None of Few of these Most features All of thesethese behaviors but demonstrated behaviorsbehaviors major omissions

G G G G G G G1 2 3 4 5 6 7

M Communication: Audience: address and answer client's questions at client's level of understanding; Content: include factual evidenceto substantiate claims; include units, assumptions, clearly explain limitations of your work; complete references for the source of data; Organization: systematic, well-organized with appropriate advanced organizers, transitions and summaries; no faulty subordination orcoordination, index or table of contents included in engineering day-book; Style: understandable, coherent, readable, interesting, succinct;Format: grammar, spelling, punctuation correct; appropriate format used for all communications.

The degree to which these descriptors describe your performance (as substantiated by evidence: meetings,engineering journal, interim report, presentations). None of Few of these Most features All of thesethese behaviors but demonstrated behaviorsbehaviors major omissions

G G G G G G G1 2 3 4 5 6 7

Strengths Areas to work on________________________________________ ________________________________________________________________________________ ________________________________________________________________________________________________________________________

____________________________________ from D.R. Woods (1997) A87 Attach any pertinent additional evidence that we should consider. Thanks.

8-45

Table 3 Sample questions based on objectives for component skills for team work, TEPS

1. (20 min) Meetings The following is the first part of a design meeting with Charles (age 35, head processengineer), Jack (age 55, engineer), Jane (age 23, engineer in training), Josephine (age 30, process control engineer)and You, a summer student.a) Comment on the group skills that each person is using; use the Group feedback form in Figure 2 as a guide.b) List five strengths and two areas to work on for Charles and for Jack.c) Comment on the process they are using in this meeting.d) What would you say now?

Charles (chair) OK, I circulated the agenda. Our task is to select a pump for the MS plant. I trust that all havebrought the company design manual and that we are ready to proceed. I suggest that we spend thefirst 20 min. ensuring that we have defined the correct problem. Then, we work on sizing andselecting the pump for the next 30 min.

Jane Sounds good. But shouldn't we wait for Jack. He is supposed to be at this meeting. He always has goodideas.

Charles He knows about the meeting. I like to start on time. Alright, we have to move 20 L/s of butane from thestorage tank on the ground level up 15 m to the feed tank. Seems pretty straight forward.

Jane I still think we should wait for Jack.

Charles He may show up in 1/2 an hour. What do we do in the meantime? We all have busy schedules. Do youwant to do this job or just wait for Jack? Jack's OK but let's just get started.

Jane (reluctantly) Ok let's get started... On this problem since this is a liquid, I think we use 1 m/s, size thepipe and estimate the friction loss. Oh, we should consider the process control because we have acontroller in the system and we need to allow a large pressure drop, about 40 kPa is what I wouldsuggest. Take that info and plug it into Bernoulli and we've got our pump. Josephine, you can helpwith the process control system.

Jack (Meandering in with his coffee cup and a smug look on his face). Morning all. Didn't get your agenda,Charlie, until this morning. You need to get those agendas out earlier old boy. I have so much stuff todo. I need to plan carefully and giving me agendas at the last minute is the pits.

Charles Good afternoon! The name is Charles. The agendas were delivered personally to your office one weekago. Don't give me that *#!|! Jane, shall we continue.

Jane I think we should define the real problem. Let's hear what the summer student, whose had all that stuff justlast semester, thinks about that... ( and Jane turns to you for your response...)

You Write out your response

*******************************end of question 1 ****************************************

Time 7. You have been asked to chair a 50 min "design meeting" to introduce your colleagues13 to the short cut design procedures for fluidized beds. (Given in Chapter 3 of the bookmin "Selecting Process Equipment"). The five engineers have agreed that the meeting will be held on

December 30, 1993.

8-46

a) Create the agenda.

b) List three actions, relative to this meeting, you personally would take between now and the meeting.

c) Estimate the time required for each of the actions listed in part b).

****************** end of question 7 ******************

Time 8. Chairperson

20 min Your company has decided to explore the possibility of using a fluidized bed reactor for the production of1000 Mg/d Phthalic anhydride. You have been assigned the role of Chairperson for the "Quickie designMeeting." You are familiar with PDEP and realize that you can probably do a quickie sizing of the reactorand a costing to test the waters. The meeting is to last 2 h and is 1 week from today.

a. Create an agenda for the meeting.

b. List any activities you would do between now and then and rationalize. ******* end of Question 8 *************

Time 5. Self awareness and understanding of inventories about personal style You are on a team consisting ofthe people listed in Table 2. Also listed are the

20 the Jungian typology and Kirton inventory and for the other people. Add your ownmin scores.

What are the implications of these results of these inventories on

(a) Potential conflict: be concrete and identify the people, their behaviour and explain the conflict and the "degree of conflict".

(b) Team "blind spots" where the team may not be as effective.

Table 2: Your team (for question 5)

Member I S T P KAI

YOU

Andy 23 25 15 19 85

Lisanne 15 28 13 20 91

Jean 28 21 26 13 82

Tom 10 30 35 26 70

***********************************end of question 5 ***************************

8-47

Goals: Content is well identified, goals are challenging and achievable, goals are written in observable terms,goals are unambiguous, the "given" conditions are specified.

None of Few of these Most features All of thesethese behaviours but demonstrated behavioursbehaviours major omissions

G G G G G G G1 2 3 4 5 6 7

Criteria: Criteria are consistent with the goals and are measurable and practical. The criteria are challengingand achievable.

None of Few of these Most features All of thesethese behaviours but demonstrated behavioursbehaviours major omissions

G G G G G G G1 2 3 4 5 6 7

Evidence: The type and quality of evidence gathered is consistent with the goals and criteria. The evidencehas been gathered conscientiously over a long enough period of time. The evidence is well organized. Thequality and extent of evidence is sufficient to allow me to judge the extent to which the goals have beenachieved.

None of Few of these Most features All of thesethese behaviours but demonstrated behavioursbehaviours major omissions

G G G G G G G1 2 3 4 5 6 7

Process: The assessment process has been applied and as an independent assessor I concur with the decisionas to the degree to which the goals have been achieved.

None of Few of these Most features All of thesethese behaviours but demonstrated behavioursbehaviours major omissions

G G G G G G G1 2 3 4 5 6 7

Strengths Areas to work on________________________________________ ________________________________________________________________________________ ________________________________________________________________________________________________________________________________________________________________ from D.R. Woods, "How to Gain the Most from Problem-based Learning" (1994)

Table 4 Feedback about assessment

8-48

Table 5: Example results from form 2804

Grouprating: 8.3(8.3)

As seen by Score for individual andstandard deviation

Maria Jason Suzelle Brad Margarite

Maria 9.5 1 8.95 1 9.1 1 9.2 1 8.6 1 9.07 [0.33]

Jason 8.5 2 8.55 3 7.9 4 6.9 3 7.2 4 7.81 [0.75]

Suzelle 8.4 3 8.95 1 8.7 2 7.6 2 7.8 3 8.28 [0.59]

Brad 6.8 5 7.43 4 7.6 5 6.2 4 4.9 5 6.6 [1.1]

Margarite 7.9 4 7.43 4 8 3 6.2 4 8.3 2 7.57 [0.82]

Ratingscale usedbyindividual.

8.22 [0.98]

8.26 [0.78] 8.26 [0.62] 7.22 [1.25] 7.36[1.47]

average of ratingstandard deviations 1.02

8-49

Assessing student performance in problem-based learning

Donald R. Woods and Liz Bayley, Chemical Engineering Department and Health SciencesLibrary, McMaster University

Hamilton ON Mar, 2006

Abstract:

Five principles of assessment are outlined. The importance of published goals, measurablecriteria, multiple sampling and various forms of evidence is emphasized. Examples andsuggestions are given as to how these might be handled effectively in the context of small group,self directed, self assessed problem-based learning. Examples are given in health sciences,engineering and business.

Keywords:Assessment, lifelong learning, objectives, goals and criteria, forms of evidence, rating forms, problem-basedlearning, McMaster Problem Solving program.

8-50

Problem-based learning embodies most of the principles suggested by Chickering and Gamsonto improve student learning [1]. Overall evaluations to prove that PBL is effective, by comparisongroups, are still tentative [2]. Graduates of PBL program display different behaviours, especiallythey take more time with patients [3]. Studies in Physics have shown that, compared withtraditional lecture-based courses, students in active-learning [4] and in PBL environments [5] showimproved pre-post gains based on the Force Concepts Inventory [6,7] (gains of 0.23 for lecture;0.48 for active and 0.45 - 0. 65 for PBL). PBL environments develop deep learning instead ofsurface or rote learning [8,9,10], provide improved learning environment as measured by the CoursePerceptions Questionnaire [9, 11, 12, 13], provide a professional context for learning new knowledge[14], develop lifelong learning skills [9,11] and develop student’s self confidence [9,15]. Exit surveys,using Queen’s University Exit survey [16], has shown that graduates from our PBL program selfrate their acquisition of problem solving, critical judgement and team work higher than graduatesfrom a traditional program.

In PBL students learn subject knowledge and concurrently are given the opportunity to developskills in lifelong learning, problem solving, group work, communication, self assessment, changemanagement and critical thinking. We refer to the latter skills as “process skills”. Yet, if we areusing small group, self directed, self-assessed problem based learning, some common concernsinclude how do we assess the student’s performance relative to the quality of the knowledgeacquired and the process skills developed?

Graham Gibbs[17] says “Whoever owns the assessment, owns the learning”. The impact ofassessment on learning has been reported elsewhere[18, 19]. Savin-Baden[20] reminds us that“Assessment is an area that is often problematic on programmes which espouse self-directionand learner-centred approaches because of issues of power and control between tutors andstudents..... For example, the form of assessment can encourage students to adopt methods oflearning that ensure they pass the course with high grades, rather than adopt learning approachesthat would be in their best interests as an individual and a group member.” Many have addressedthe issue of assessment in the context of PBL [21-26] . Here we will do our best to summarize thevarious approaches and share our experience. We have developed and used a variety ofassessment methods in our PBL program that started in 1982.

In this paper, we review the principles of assessment, describe how the context is established forassessment and list goals and criteria and suggest options for gathering evidence. Examples aregiven of the assessment methods we used. This is described in the context of assessing studentperformance in the context of PBL.

1: Principles of Assessment

We define assessment as “a judgment based on the degree to which goals have been achievedusing measurable criteria and pertinent evidence.” Five principles embedded in this definitioninclude: [27-30]

Principle 1: Assessment is a judgment based on performance - not personalities. We need to helpa student realize that a poor mark does not mean he/she is a bad person. The judgment is made

8-51

about performance in completing a task. It has nothing to do with his/her value as an individual.This is an issue, especially for students with attitudes characterized by Perry’s level 2, whobelieve that a bad mark means he/she is a bad person. More details about Perry’s levels andtheir implications to teaching and learning are given elsewhere. [27, 31, 32]

Principle 2: Assessment is a judgment based on evidence - not feelings. We might intuitivelyfeel that a student is skilled at team work. However, we need to replace that intuitive feeling withphysical written evidence.

Principle 3: Assessment should be done for a purpose with clearly-defined performanceconditions.

Principle 4: Assessment is a judgment done in the context of published goals, measurable criteriaand pertinent, agreed-upon forms of evidence. That is, all involved in the assessment processhave received a written copy of the goals, the criteria and the forms of evidence. In conventionalprograms this is usually suppled by the teacher. In PBL, the students create the goals, criteria andforms of evidence.

Principle 5: Assessment should be based on multidimensional evidence: static and dynamicsituations; small assignments and lengthy projects; academic, social and personal contexts; undera variety of performance conditions (exams and homework, written and oral, performance as anindividual and as a member of a group); using formative and summative data and with differentpersons being the assessors (self, peer, teacher and trained external observers)[33].

To remove ambiguity from the assessment, the following six issues in practice should beaddressed.[27, 29, 30]

1. Goals: What is being assessed? Knowledge in chemical engineering? Skills? Attitudes? Havethe goals been expressed unambiguously in observable terms? Who creates the goals? Are thegoals explicit and published?2. Criteria: Do the criteria relate to the goals? Can each criterion be measured? Who creates thecriteria? Are the criteria explicit and published?3. Form of evidence: Is evidence consistent with the criteria? Do both the assessor and thestudent know that this form of evidence is acceptable? 4. Resources: Are the goals and the collection of the evidence possible to achieve in the timeprovided and with the resources available? 5. Assessment process: What is the purpose of the assessment? Under what conditions is thestudent’s performance assessed? Who assesses? What type of feedback is given by the assessor?(For example, Pass/ fail? A grade? A list of the five strengths and two areas to work on?) Whatis the form of feedback? Verbal? Written? What is the timing of feedback? Who delivers thefeedback? 6. Training in the assessment process: Have both the student and the assessor received training inassessment?

Failures of assessments to accomplish their purpose can usually be traced to violations of anyone of these five principles or to the incorrect application of the six issues in practice. Doing an

8-52

1 Details are available from the authors.

assessment well is not easy. For example, we reviewed 26 refereed papers describing theassessment of team work that were published in notable education journals between 1999 and20011. The assessment process (and hence the results) was invalid in all of the papers. Several mistakes were made. In none of the work were observable, unambiguous goals presented, indeedno goals (other than “good teamwork” whatever that means) were given in 20 papers. The goalsgiven in 6 papers were ambiguous. Measurable criteria related to the goals were not published inany of the papers. Ambiguous or criteria unrelated to the goals were given in 3 papers. Nocriteria were published for 23 papers. As another example, we reviewed the assessment process used in a residency program1. Thegoals and outcomes were published, but were expressed in ambiguous and unobservable terms.The rating form was poorly designed, considered only about half of the stated goals and includedgoals that were not listed as goals. On a positive note, our evaluation of an assessment process used in another program showedpublished, unambiguous goals with consistent measurable criteria. They published the details ofone major form of evidence1. Yes, assessment can be applied successfully.

2. Setting the context

For student performance, whether assessed by faculty, peers, external evaluators or self, thecontext must be clear. From principle 4, the context is published goals, measurable criteria andpertinent and agreed-upon forms of evidence. We will first consider goals and criteria, and thenthe elements in the assessment process are discussed.

2.1 Published goals and criteria

The goals apply to the discipline-specific knowledge and process skills.

2.1-1 For discipline-specific knowledge and skills

For a traditional lecture-based course, learning objectives that satisfy accreditation requirementsare created by the faculty and given to the students through the syllabus and throughout thecourse. Contrast this with a PBL course. For each session faculty start with learning objectivesand educational goals (similar to the ones they created for a traditional course) but now thefaculty use these objectives as the basis for crafting professionally significant problems/cases.Other criteria for crafting PBL problems are listed elsewhere [34 -36]. In PBL the key is that theproblem must be cued and so crafted that the students, by engaging with the problem, willgenerate, as a minimum, the same learning objectives as the faculty used in creating the case.The difference is that the students now feel ownership of the learning. Gaining this degree of

8-53agreement between learning goals used by faculty to create the cases and student-generatedlearning goals is not a trivial task.

In groups where a tutor is present, the trained tutor guides the process until the target learninggoals have been identified and prioritized. In this way, the tutor ensures that the student-generated learning objectives are consistent with the program and accreditation goals.

In tutorless groups [37], Dolmans et al.[38] report that these groups usually can identify about 60%of the target learning objectives. In our 25 years of experience with PBL and tutorless groups, weprovide explicit training[39, 40] in goal identification. Furthermore, the goals meeting consists oftwo parts: identifying and prioritizing the learning goals and secondly, contracting and allocatingresources. Before any group tackles the latter, the list of student-generated goals must bevalidated by the tutor. An example validation form is given in Table 1. A tutor and teachingassistant can handle about a dozen autonomous groups. In our experience, trained autonomousgroups can usually identify about 90% of the target learning objectives.

The criterion is usually “that the learning objectives shall agree within 95% of those identifiedby the tutor”. At first glance, this might appear to violate one of the tenants of PBL - namelyempowering the students with the task of identifying their learning needs. However, PBL is notgiving student license to study whatever they want. Rather, the responsibility is on the instructorto craft the problem so that the students identify learning objectives consistent with the course(and accreditation) expectations. If the student’s learning objectives do not agree with theinstructor’s learning objectives, then the problem has been poorly crafted.

2.1-2 For the process skills

Our experience has been that it is wise to publish the goals and criteria for the process skills(instead of asking students to generate them). These should be based on research evidence andnot on intuition[9, 39, 41] . An example of the evidence we have found related to lifelong learningskills is given in Table 2 [42-49]. This evidence can then be converted into learning objectives andcriteria, as illustrated in Table 3. Goals and criteria have been published for other process skills such as problem solving, criticalthinking, learning, personal awareness-acceptance, self assessment, time and stress management,change management, group skills, chairperson skills and conflict resolution [9, 39, 41].

Some discussion has occurred in the literature as to whether process skills exist separate from acontext. For example, Norman suggests that problem solving skills are not independent ofknowledge [23]. On the other hand, Stewart[50] says"...all problem solving is based on two typesof knowledge: knowledge of problem-solving strategies and conceptual knowledge." We seethis as a semantic issue. First one needs to distinguish between exercise solving and problemsolving[ 27, 39, 49, 51 - 54]. In exercise solving one recalls and uses solved examples from pastexperience, examples of how the person solved similar situations in the past. Experiencedprofessionals rarely use problem solving; they are exercise solving because they have had somuch professional experience[26, 55]. Problem solving occurs when one doesn’t have any previouspast solutions to problems similar to the one being addressed. When discussing problemsolving, our stance is that there are elements of the problem solving process that skilled problem

8-54solvers display that are independent of the knowledge, and these can be made observable andmeasurable. These include being active, focusing on accuracy, monitoring the process, andmaintaining a set of about a dozen hypotheses in clinical situations or trouble shooting problems[27, 49, 56-58]. An elaboration of such problem solving skills and options and evidence for assessingproblem solving skills is available [57,58].

For the learning goals, there must be measurable criteria.

2.2 The assessment process

Four issues important to the assessment process include who does the assessment, the astutedesign of the forms of evidence, what training is provided to the assessors and addressing theissue of ownership of the process.

2.2-1 Who does the assessment

With tutored groups, some use assessment by the tutor. This is challenging because Swanson etal. [8] and de Stephen[59] report that it is difficult for one person to provide assessment of more thetwo people at the time. Since tutorial groups usually range in size from 5 to 11 members, placingmuch emphasis on tutor assessment is questionable. Peer and self assessment are probably morereliable forms of evidence. For large classes (one faculty member with 10 to 100 students) onecan use tutorless groups. Our experience with tutorless groups[60] has been that peer and selfassessment has been a reliable form of evidence. For example, the difference between finalwritten exam mark and the self assessment mark was -0.8 % with a standard deviation of 5.84 for n = 265 students.

2.2-2 The design of the forms of evidence

The form of evidence must be well-designed.

For written examinations, the tasks must be consistent with the published goals and criteria,questions clearly stated, reasonable in expectation[61,62] , appropriate in intellectual expectation(using Bloom’s taxonomy or SOLO)[63, 64, 28], with the appropriate complexity[65] and degree ofexplicitness[66] .

For written multiple choice questions, MCQ, Case and Swanson offer suggestions[67].

Consider rating forms. Whether the assessor is observing or self assessing the performance ofknowledge or skills, often a rating form or scoring rubric is devised to attempt to providecriterion-based assessment.

The rating form should be valid and reliable. The items included in the form must be consistentwith the criteria. The rating form should not contain more items than are convenient for a rater toconsider at any one time, and the form must be related to the goals and criteria which aredescribed in some detail on the form.

8-55

2 More information about these rating forms can be obtained from the authors.

An observer assessing a group should not be expected to rate more than two or three people atone time [24, 59] and the form should be organized in convenient chunks around the main elementsof the process being observed. Compare, for example, the two forms in Figure 1 that have beenused to assess an individual’s performance in a group. Form A shows only the first entry of 26items; Form B shows the first of 8 items. First of all, expecting an observer to keep track of andrate 26 different items is asking for super-observers. We think, in form B, that an observer has achance to remember and rate only 8 items. Secondly, Form A places these in the category of"helpful behaviors". Research suggests that Task and Morale are the two major issues toconsider in group work. These descriptors in Form A sound like task issues but “arriving on timeand prepared” can be construed as a Morale issue. Such ambiguity makes it difficult for the rater.Form B clearly identifies Task as being the issue and identifies one of four Task items forassessment. Each item can have either positive or negative contributions. In summary, form B ispreferred over form A.

It is important to provide external descriptions of the criteria or standards. Otherwise, raters tendto create internal standards- standards selected based on one’s own performance or aperformance judged to be similar to one’s own[68, 69] . Consider forms developed to rate aclinician. Figure 2 shows four different rating forms. Form 1 gives no details of the goals; it onlylists " effective clinician" and provides no measure of the criteria. The rater is expected to picksome number between one and ten. What is acceptable or outstanding to one rater, will beinferior to another. No standards are given, and so each rater will create personal and differentinternal standards.

Form 2 offers at least some description of the skill being considered: the “knowledge” that theclinician brings to the patient-doctor encounter. However, the term “knowledge” is ambiguous;different raters will use different internal criteria in choosing the box "meets objectives".

Form 3, in Figure 2, gives more description of the goals by listing three attributes under“knowledge” that an effective clinician is expected to display. Some criteria are given to help arater understand what "meets objectives" means. However, the term "solid knowledge" and"common problems" are ambiguous. What is solid knowledge to one rater is not solid knowledgeto another.

“I rated you low because you knew very little about diphtheria.” "Diphtheria is not a common problem in this area. Why should I be expected to know about

diphtheria?" responds the resident being evaluated.

Although form 3 is a marked improvement over forms 1 and 2, the form still has ambiguouswords.

Form 4, in Figure 2, gives a more detailed and less ambiguous description of what is expected.True, a list is not given of the 38 illnesses but, at least we know that we are expected to know 38and not just 5. This elaboration on the goal allows the rater to check or circle the particulardeficiency from the list that forms the basis for the rating.2

8-56Criteria-referenced, rather than norm-referenced, assessments[69] should be used. The use ofnorm-referenced assessments means that one’s performance is compared to another’s instead ofbeing compared to external standards defined by the criteria. Norm-referenced leads tocompetition that is contradictory to the cooperative atmosphere expected especially in PBL.

2.2-3 Training the assessor

No matter who does the assessment, the assessors should receive training in assessment. Someuse videotapes of PBL sessions as training devices to train observers in completing well-designed rating forms. We have used a 6 hour workshop on assessment to train either tutors orstudents[60].

2.2-4 Addressing the issue of self-directedness

Savin-Baden[20] and Barrows[21] raise the issue of assessment in the context of self-directedlearning. Is the assessment consistent with the student-generated learning objectives?Technically this is a non-question if the PBL experience is well-designed. As mentionedpreviously, the program design starts with the faculty and their creation of the learningobjectives and criteria for the learning experience. The faculty translate those objectives into aproblem statement that must be well-cued so that the students generate learning objectives thatare identical to the faculty’s. The issue then becomes one of perception. Do the students perceiveownership of the learning process with assessment being part of the process? Ourrecommendation is to use self and peer assessment as much as possible.

3. Forms of evidence: options for documenting and measuring knowledge acquired

A wide variety of forms of evidence have been used[26] . These include - Written tests and exams: such as multiple choice questions (MCQ), essays, modified essayquestions (MEQ), written problems to solve and written concept maps.- Written forms of evidence related to their learning-teaching-problem solving process: such as plans and use of resources, study and teach notes, elaboration of the knowledge, a solution to theproblem, and written learning contracts and plans.- Rating of individualized activity such as the portable patient problem pack and computersimulations or of interactions with simulated patients/systems via simulated patient or troubleshooting triads.- Direct observation and rating of the performance such as oral presentations, oral exams.- Direct observation and rating of a group tutorial or specialized group activity. - Interrupted problem solving activity such as the triple jump, objective structured clinicalexamination OSCE or PBEE.- Pre and post tests.- Reflective journals, portfolios.These can be used as summative of formative assessments.

In presenting some of these options we hope to provide multiple forms of evidence from whichyou might select the ones most appropriate for your environment. The first seven options relateprimarily to the quality of the knowledge learned. Options eight and eleven provide evidence for

8-57both the knowledge learned and the process. Options twelve to nineteen relate primarily toprocess skills associated with PBL.

!Option 1: Individual Test and Exams of the knowledge, TETK.

Exams and tests can be created to give individuals a chance to demonstrate that they understandthe new knowledge. These tests can be written or oral and given as spot-checks or formally builtinto the program. The TETK must be consistent with the learning goals created by the students. The tests can be created, and assessed by, the teacher, peers or self.

Some popular formats included MEQ and MCQ. In the MEQ students are given an initial caseand asked to respond to a progressive series of questions or tasks about the case. For example,given a brief problem, task: list hypotheses. Next task: list questions you would ask. Next: giventhe following new information, what are your hypotheses now? Students are not allowed tobacktrack to previous tasks.

Barrows[21] describes the use of faculty-generated MEQ to assess the information the studentshould bring to mind while working with the patient immediately following each case.McPherson et al.[70] describe the use of modified essay questions, MRQ.

Our experience has been to use a faculty-generated three-hour written examination at the end ofthe course and to have student-generated and student-marked tests at the end of each PBL case.For the latter, after each PBL case cycle, groups of students prepare test questions based on thestudent-generated learning objectives. Tests questions from one group are given to another groupto solve in writing. The group’s answer is then marked by a peer from the group that posed thequestion. The teacher monitors the process and grades the quality of the posed question andanswer and checks the validity of the peer mark given to the group that “solved the problem”.Monitoring the questions is important because students may not be skilled in crafting questions. For example, some groups pose obscure questions unrelated to the learning objectives or givequestions that are too complex to answer in the time available. For the class of 2005, for the firstcycle, the average marks for the questions posed by 11 groups was 8.8; cycle 2 was 8.1 but bycycle 3 it was 9.4.

Another option is to randomly call on one individual from each PBL group. The individual’sperformance becomes the group’s performance. This approach is used in cooperative learninggroups to ensure that all of the individuals have learned the subject knowledge.[50]

! Option 2: Progress test [72-76]

This is a special form of TETK that is a summative MCQ written examination of the full rangeof knowledge expected from the complete program. What is unique is that all students in alllevels write the same examination about three times a year. Students at the first level may havefew questions that they can answer but the number of questions and their success rate increasesas they progress through the program. This provides students and faculty normative informationabout the acquisition of knowledge. Studies[75] suggest that this form of assessment does not

8-58interfere with the self-directedness elements of PBL.

The option has been used in medical schools where PBL is used throughout the whole program.

! Option 3: Student summary of the quality of knowledge learned. Based on the student-created learning objectives, an individual could write a summary of theknowledge learned, its relationship to previous knowledge and elaborate that knowledge. Thiscould be marked by peers, self or tutors.

Alverno College[77] uses a variation called the “critical learning incident”. Here the student identifies a learning experience where something noteworthy occurred. In this reflection thestudent identifies where the incident occurred, who was involved, how typical this incident wasin their experience, what happened, what they thought and felt at the time, what was intended,what they think and feel now, and what they learned.

! Option 4: Individual concept maps of the knowledge.

Individuals could create concept maps representing the new and previously known knowledge.Novak and Gowin [78] give suggestions, examples and methods of assessing. They emphasize theimportance of

- writing the relationship on the line connecting concepts,- using a hierarchy that follows the principles of valid classification: the classification doesnot have faulty coordination or faulty subordination,- including cross links between different sections of the hierarchy,- providing concrete examples that are valid instances of the concept.

More details are available[79, 80].

Angelo and Cross[81] describe the use of concept maps to monitor classroom learning, CAT 16. Akinsanya and Williams[82] describe their use of concept maps in nursing. A variation on the concept map is the symptom-cause map important in clinical problems inhealth sciences[83] and in trouble shooting problems in engineering[49, 84].

! Option 5: Individual teach notes In PBL, individuals contract with the group that they will learn new knowledge pertinent to oneor two of the learning objectives and teach the group members that knowledge. As an aside, we have found it useful, prior to this activity, to provide a training workshop[23] andto provide all members of every group with the scores of inventories related to learning styles(Perry, Lancaster approaches to studying, Jungian typology and Kirton inventory[ 39 - 41]). Thishelps each student address the teaching task[40]. Our students create teach notes that are given to the group members. These teach notes areassessed by the tutor. Teach notes have been used as evidence for students in a business program[25] . ! Option 6: Peer feedback about the quality of the knowledge brought to the teaching task.

Peers can give feedback about the quality of the knowledge demonstrated when a peer taught.

8-59The form we use[26, 36] asks peers to rate

the Quality of the knowledge: good intellectual understanding of the topic, the materialsupplied was complete and appropriate.

Other dimensions can be included in this form that relate to the process. This is discussed inOption 15.

The feedback can be given directly to the instructor/tutor or to the student. In our MPS program,the student collects all the feedback and includes this as evidence in his/her assessment journal.The assessment journal is described in Option 16.

! Option 7: Group solution to the problem.

Once the students have contracted with each other to learn the required knowledge and to teacheach other that knowledge, the group then uses the knowledge to solve the problem. We coulduse the quality of the group solution to the problem as evidence of the knowledge learned. Thisis a group mark. Individual contributions can be assessed based on self and peer assessment[85] using a form similar to Table 4.

4. Forms of evidence: options for documenting and measuring knowledge and skillacquired

The next five options gather evidence about both the product (the knowledge learned and appliedto solve a problem) and the process (such as the lifelong learning skills, problem solving, criticalthinking). The first option was developed for testing individuals one-on-one; the second option,for individuals in the context of a large class.

! Option 8: Evidence about both the process and the knowledge learned : the Triple Jump.

For PBL, the McMaster Faculty of Health Sciences developed three structured activities[24, 26, 39, 86,

87] to assess the process and the knowledge. These three activities are:1. In response to a problem situation, students define the problem and make a prioritized listof new knowledge that they need to learn to solve the problem. This is a 30 minute oralinteraction between the tutor and the student. 2. Students self study and privately synthesize the new knowledge they selected as their toppriority. Students are allowed two hours. Each writes outline notes that will be judged on theefficiency in sticking with a realistic task, the knowledge resources consulted and the abilityto locate key information and the quality of the note-taking and referencing.3. Students meet again with their tutor for 30 minutes. They respond to further questionsfrom the tutor to illustrate the process used, the new knowledge gained and how they mightuse that knowledge to solve the problem. Example criteria include the ability to present andsynthesize ideas, the ability to pose questions about the learning issues that were notexplored and the quality of the students self assessment.

A rating form is used by the tutor. The evidence would be the list of learning issues, the writtenoutline notes and the tutor and self assessments.

8-60

! Option 9: Evidence about both the process and the knowledge learned: Branda=smodification of the Triple Jump for large classes, PBEE.

Branda[88] modified the Triple Jump for use with one teacher and a class of 100 to 150 students.In a 50 minute class, in response to a problem situation, each student lists four learning issuesand prioritizes and justifies his/her top two issues. For the next class, each researches his/her toptwo issues and prepares a two-page summary. In the next 50 min class, each answers two testquestions, prepared and personalized by the instructor. In class they can consult only their two-page summary notes. Branda acknowledges that this is a lot of work but that, for any problemscenario, a bank of one to two dozen test questions covers all of the different learning issuesgenerated by a class of 150 students.The criteria developed by Branda are:For Stage 1: Issue identification: clarity of issues and correctness of context statements;

relevance to the learning objectives for the overall course and to the problem situation.For Stage 1: Research and study issues: appropriateness of the issues identified and justification

of the choice of issues. For Stage 2: Knowledge and understanding of the area of research: quality of the answers to thetwo questions.

! Option 10: Evidence both about the process and knowledge: critical instance cases

Critical instance problems provide an opportunity to assess student’s performance relative toknowledge and process. In health sciences these are referred to as clinical events[33, 89]; inengineering, as trouble shooting problems[49, 90]. For the process skills, target behaviors havebeen described for health professionals[89,56] and for engineers[41, 49] and goals and criteria havebeen published[41]. To gather evidence about student performance, the performance situationinvolves three elements: a) the student clinician-trouble shooter, b) the patient or system thatdisplays symptoms and responds to tests, and c) the assessor, observer. The variation is usuallyin the format of b. The “system” could be a person trained to be a simulated patient[33]; or trainedto respond as the “expert system”[49] . The system could be a deck of cards as in the portablepatient problem pack[91] or the patient management problems. Difficulties in using the latter forthe purpose of determining a “mark” have been described by Swanson et al.[92].Another option for presenting the case is a computer simulation[33, 90].

The forms of evidence relate to the script of questions and actions requested, a rating preparedby the observer, a comparison of the cost and the questions/actions posed with those ofsuccessful/experts and reflections about the process. The choice depends on the purpose of theassessment: to provide a “mark” or grade or to develop confidence and skill. In our MPSprogram the assessment was primarily to develop confidence and skill. The “mark” was based onthe reflective journal, Option 16.

! Option 11: Evidence about both process and knowledge: The OSCE

The Objective Structured Clinical Examination, OSCE, is similar to a modified essay question,described in option 1, except here students rotate through a series of timed, 5 minute stations. At

8-61each station, students are given a different task to do or asked to perform part of a physicalexamination or required to take a focused history or interpret data. Information is divulged in apreset sequence. For more see Fliegal et al.[93].

5. Forms of evidence: options for documenting and measuring process skill acquired

The next options offer evidence mainly about process skills. For the sake of this discussion, thefocus is on the acquisition of lifelong learning skills.

M Option 12: Evidence about individual’s attitude toward learning and approach to learning.

Perry’s inventory[27, 31, 32] and the Lancaster Approaches to Studying Questionnaire,[93] LASQ (orequivalent[94,95]) are questionnaires about:- One’s role in learning (Perry). The first five scales are the most appropriate. Typically studentsleave four year colleges with levels 3 to 4. Level 5 is most consistent with an attitude helpful forlifelong learning. For example, Pavelich and Moore [96, 97] report that freshman engineeringstudents rated average values of 3.27 and they graduate as seniors with scores of about 4.27.These students are in the unique EPICS[98] program at the Colorado School of Mines. Thesefreshman scores are higher than those reported by Allen[99] of 2.3 to 3.1 and by Marra et al.[100] of2.9. Marra et al. found that registration in their design course increased the freshman scores to3.29. Fitch and Culver[101] report values for seniors as being 2.8 to 3.1. For the MPS program,[9]

scores for sophomores have been in the range 2.7 to 3.4; for juniors, about 3.5 to 4; and forseniors, 4 to 4.7.

- The inventory (LASQ) explores one’s preference among three options in learning: 1) surface orrote learning; 2) deep learning or a search for meaning and 3) strategic or work hard and usewhatever approach will allow one to succeed. Typically students graduate with a preferencetoward rote learning whereas one hopes a lifelong learner will search for meaning. Forexample, students enter college with the distribution 15, 13 and 15 for rote, meaning andstrategic respectively. If the college program continues with primarily formal lectures, studentsgraduate with a statistically significant increase in rote learning. On the other hand, if theprogram includes experiential activities, projects and problem-based learning, the students tendto shift toward a search for meaning. For example, in a longitudinal three year study for the samecohort (N = 60) in the MPS program[9] that includes PBL and experiential components, in threeyears that rote decreased (p < 0.035), meaning increased (p< 0.027)[102] . Coles[103] compared theeffect on cohorts of students in one year in lecture program versus those in one year of a PBLprogram. Over one year those in the lecture program showed an increase in rote (p<0.01) and adecrease in meaning (p<0.001). Those students in the PBL program showed a decrease in rote(p<0.05) and no change in the search for meaning. Students in a five year, inquiry programcalled Engineering and Society, showed a decrease in rote and an increase in meaning withratings 12.6 and 16 respectively. Students in a PBL Theme school had ratings of 13.3 and 15.5for rote and meaning respectively.[104] Other inventories useful for assessing other process skillsare available[39] .

M Option 13: Test and Exams in Process Skills, TEPS.

8-62Based on the goals for process skills given in section 2, create examinations that test theachievement of the published objectives for lifelong learning skills. Some example questions aregiven in Resources Chapter D [23] and in Table 5.

M Option 14: Self assessment of achievement of target skills for lifelong learners.

Form 3600, given in Table 2, lists the target skills and attitudes for lifelong learning. This formcan be completed by individuals periodically throughout the semester and throughout theprogram to reflect their journey toward skill in lifelong learning.

M Option 15: Peer and self assessment of individual contributions to the learning.

As an extension of option 6, where the focus was on the quality of the knowledge, questionsenlarging the scope to include the process of lifelong learning can explore:

the Quality of the instruction: he/she was here on time, the presentation focused on the newknowledge with good choice of material and medium; effective communication and goodresource material was supplied.

the Additional follow-up needed: responses range from “no additional self study needed”to “I must study this subject on my own; I learned nothing from the presentation”.

Peers also list the five strengths of the presentation and two areas to improve. As was done in Option 6, the feedback can be given directly to the instructor/tutor or to thestudent. In the MPS program, the student collects all the feedback and includes this as evidencein his/her assessment journal. The assessment journal is described in Option 16.

M Option 16: Reflective assessment journals

Reflective self assessment journals can be written for any of the process skills. For example, for lifelong learning skills, the goals and criteria are created, as suggested in section 2. Individualsgather evidence about his/her own achievement of the goals: reflections, peer feedback, learningcontracts, teach notes. Individuals gather evidence after each PBL cycle or case. Examples areavailable.[23] The elements included in the a reflective assessment journal typically are:

- a comparison of how well one achieved each objective before and after the workshop,- a critique of the pre- and post test results,- the validation form given in Table 1,- a summary of new ideas and behaviors discovered from the workshop,- for each objective, a list of the pertinent evidence and a claim as to the degree to which onehas demonstrated completion of the objective,- evidence of how the skills have been used in other educational contexts and courses,- evidence of how the skills have been transferred and used in everyday life,- conclusions and summary of achievements.

The evidence used by the students includes Options 5, 6, 14, and 15.

8-63M Option 17: Portfolio about skill in lifelong learning.

A portfolio is a collection of evidence of “best” work. Mourtos,[105] for example, asks students toinclude assignments, tests, enrichment activities and an overall reflection about the learning.Whereas Option 16, reflective assessment journals, is structured around faculty-generated goalsfor skill in lifelong learning and assessment, the portfolio tends to allow students to selectmaterials that represent their best work. Ideally the Portfolio should have a clear summary by thestudent of the skills they have achieved as demonstrated by the materials included in theportfolio. Mourtos recommends the development of extensive marking criteria and provides ageneral format for the portfolio. He found that markers spent 30 to 45 minutes marking eachportfolio. We would add that the marking criteria should be published and given to the studentsat the beginning of the semester.

M Option 18: Learning plans for lifelong learning.

Learning plans that identify learning goals, learning questions/objectives, course objectives,learning resources and strategies, evidence and assessment and target date[106] are another form ofevidence that has been used effectively in the nursing program at McMaster University.

M Option 19: Other options.

The CATs listed by Angelo and Cross[81] have been popular possibilities to assess the “softskills”. The CATs are a collection of options to monitor classroom learning, but these can oftenbe converted to forms of evidence if the appropriate goals and criteria are made explicit. Forlifelong learning skills, few CATS, besides concept maps, are applicable. The use of CAT 35(assessing students’ awareness of learning goals) and CAT 36 (self-assessment of ways oflearning) were described in courses (p. 79 ff) and may have potential in some contexts.

6. Example assessment of student performance in PBL

Since 1982, in the Chemical Engineering program at McMaster University students learn thefive principles of assessment and develop skill in creating observable objectives, identifyingmeasurable criteria, completing assessment evidence and writing self assessment reflectivejournals in a six hour workshop[39, 41, 60]. This addresses the sixth issue in practice described at thebeginning of this paper. Before they experience PBL students also have workshops on teamskills (2 h), identifying personal uniqueness especially as it applies to group work and learningstyles (2 h), conflict resolution (2 h), stress and time management (4 h) and problem solving (14h). Immediately before the first PBL sessions, we introduce students to the unique characteristicsand expectations of the PBL learning environment. The amount of detail we use has evolvedover the years. Initially we showed a videotape showing students in PBL, and provided a onehour overview of the expectations. This had mixed success. In 2005 we had a one-hour lecture-interaction in which we rationalized the choice of PBL as a learning environment and describedthe underlying principles of how case problems were created. We basically addressed thequestion “What’s in it for you, the students?” We related how, traditionally, instructors publishlearning objectives for every course and how in PBL, instead of handing students the courselearning objectives, we pose professionally- significant problems that are cued so that the

8-64

3 The four year program leads to an accredited degree in Chemical Engineering; the five yearprogram, to accredited degrees in Chemical Engineering and Management, ChemicalEngineering and Society and Engineering and Chemical Engineering and Bioengineering.

students will generate learning objectives equivalent to the instructors. Teachers monitor thisprocess to ensure that each group is learning the “right material”. The case problem shouldgenerate about 6 to 7 learning objectives each requiring about one to two hours of “learningtime” and one to two hours to create effective teach notes for a total of about 2 to 4 hours ofhomework per group member. This provoked a lot of discussion and very probing questions.What seemed to be helpful to the students was a sense of direction behind the activity, theteacher’s validation of the learning objectives and the clarification about the time commitment.The required text for this course is Woods’s “How to gain the most from PBL”[27].

We provided a one-hour workshop on how to search for cues, generate learning objectives, howto research and teach and how to prepare effective teach notes [39]. In addition, we had a one hourclass time for the group norms meeting. The research on the effectiveness of this approach isreported elsewhere[40].

So that students get the most from their PBL experience, we recommend these activities be doneregardless of the form of PBL used. Without such preparation, our experience is that studentshave the misconceptions that assessment is “licence to give yourself (and your friends) 100%”and that PBL is a learning environment where the teacher doesn’t do his/her job and the studentcan learn whatever they want.

Now consider the form of PBL we use. In our particular context, we use small group (5 to 6students), self directed, self assessed, interdependent problem-based learning in two requiredcourses in an otherwise conventional four to five year curriculum3. The PBL sections are forknowledge in engineering economics and process safety and, for each, there are 4 to 6 caseproblems with one case being considered each week. These are tutorless groups since oneinstructor is available for classes of 60 students. More details about our context of using PBL inan otherwise conventional program are available[9, 107,108].

We assess both knowledge acquisition and process skill development. The learning objectivesand criteria for the subject knowledge are generated by the student groups and monitored by theinstructor (as described earlier). The learning objectives for the process skills were developed bythe faculty based on research evidence and given to the students. These were challenging todevelop and evolved over 13 years. They are available[39, 41, 109].

Concerning the forms of evidence, in this paper we have listed, with succinct descriptions, awide range of options for assessing the performance of students in PBL programs: how well dothey understand the knowledge and how well can they apply that knowledge? how skilled arethey with the process skills?

Some options for small group, self directed, tutored PBL that is used in every course (HealthSciences, especially at McMaster University) include options 2, 8, 11 (the progress test, thetriple jump and OSCE). There is tutor and peer assessment but we am not aware of the details.

8-65For our small group, self directed, self assessed, interdependent and tutorless PBL the forms ofevidence include options 1, 5, 6, 12, 13, 14, 15, and 16 (Tests and exams, teach notes, peerfeedback, attitude measures, self assessment and reflective journal writing).In practice, four elements have been used as the basis of the final mark or grade in the course.These are 1) a written, three hour conventional examination, 2) the student-generated tests, 3) the reflective journal and 4) an interview.

The written, three-hour conventional examination is created and marked by the instructor andincludes questions related to subject knowledge and process skills. All questions were created inthe context of the learning goals an criteria (whether they have been generated by the studentgroups or by faculty). This is a combination of Options1 and 13 since both knowledge andprocess skills are tested. The questions are not multiple choice; they are usually problem tosolve. These have been monitored by visiting external engineering accreditation teams.

The student-generated tests for knowledge that are produced at the end of each PBL cycle havebeen described in Option 1 earlier in this paper. The team marks (for creating the test question,developing and answer and marking scheme and for answering another group’s questions)represent about 10% of the term mark.

The self assessment reflective journal, Option 16, is written by the student for each process skill.In the penultimate course, the focus varies from year to year but always includes lifelonglearning, and group skills and may include conflict resolution and asking questions. In the finalyear course, the focus includes skill in chairperson, lifelong learning and trouble shooting. Anjournal is written after each PBL cycle, after each meeting when the student was chair, and aftereach trouble-shooting triad activity. Although the total number of journals and the marks varyfrom year to year, these usually represent about 40% of the individual’s final term mark. (Theother term marks are for projects not directly related to PBL since only a portion of this course isPBL.) The reflective journal is Option 16 but in our context also includes forms of evidencedescribed in Options 5, 6, 12, 14, and 15.

The personal interview addressed the evidence the student provided for the knowledge andprocess skill development. This usually accounted for 25% of the term mark. The personalinterviews considered all of the evidence from Options 1, 5, 6, 12, 13, 14, 15 and 16 (except thewritten final exam since the interview was before the exam was written). The personal interviewwas used for the first 10 years of our journey with PBL but was discontinued because the markbased on the interview was within 3% of the marks acquired from the other options. Since weuse ± 3% as a guideline for interpreting marks from conventional written examinations, and asthe class size increased toward 60 the commitment the time by instructor became onerous, weelected to drop this option. Our experience is, for example, that the difference between finalwritten exam mark and the self assessment mark was -0.8 % with a standard deviation of 5.84 for n = 265 students.

For the final mark in the course, the student would receive the best of the 60 - 40 weighting forthe written final exam mark and the term mark. An alternative to this default weighting was thatstudents could contract to have a weighting anywhere between 10 and 90%. Whatever weightingthe students chose, for over 75% of the students there was less than 5% difference among the

8-66different marks. Details of some of the earlier results are given in Woods, Chapter 5[26].

In summary, students should receive formal training about the principles of assessment and howto self assess. Then, the actual forms of evidence used should be selected based on what wouldwork in your environment. We described the forms we have used since 1982. Our approacheshave changed over the years. .

7. Summary

1. Five principles of assessment provide a framework for developing and using evidence forthe assessment of student performance.

2. Any assessment requires published goals and measurable criteria. Some examples weregiven.

3. Assessment should be based on a wide variety of forms of evidence.4. Examples are given of forms of evidence, and their use in different programs, for the

assessment of knowledge acquisition, for the combination of knowledge acquisition andprocess skills used to apply that knowledge and for the process skills alone.

8. Acknowledgments

Vic Neufeld and Luis Branda, Faculty of Health Sciences; Tom Marlin and Andy Hrymak,Chemical Engineering Department, Bob Hudspith, Engineering and Society Program, McMasterUniversity, David Boud, University of Sydney formerly of the University of New South Walesand Georgine Loacker, Alverno College helped us in devising this approach. to assessment. Thanks.

9. References

1. Chickering, A.W., and Z.F. Gamson (1987) “Seven Principles for good practice inundergraduate education,” AAHE Bulletin, Mar., 3 - 7

2. Woodward, C. (1997) “What can we learn from programme evaluation studies in medicaleducation”, in Boud, D. and Feletti, G. (eds.) The challenge of problem based learning. London:Kogan Page, 294 - 307.

3. Woodward, C.A. et al. (1990) “A comparison of the practice patterns of general practitionersand family physicians graduating from McMaster and Other Ontario Medical Schools,”Teaching and Learning in Medicine, 2, 79-88.

4. Williams, B.A. and B.J. Duch (1997) “Cooperative PBL in an Undergraduate PhysicsClassroom,” in “Student-active science: models of innovation in College Science Teaching”,A.P. McNeal and C. D’Avanzo, eds. Saunders publishing

5. Hake, R. (1998) "Interactive-engagement vs Traditional Methods in Mechanics Instruction: Asix-thousand Student Survey of Mechanics Test Data for Introductory Physics Courses,”Am. J. Phys. 66(1), 64-74

8-676. Halloun, I., R.R. Hake E,P Mosca, D. Hestenes (1995), “Force Concept Inventory”, revised

7. Hestenes, D., M. Wells and G. Swackhamer (1992) “Force Concept Inventory,” Phys. Teach.,30, 141 - 158

8. Coles, C. R. (1985) “Differences between conventional and problem-based curricula in thestudent’s approach to studying”, Medical Education, 19, 308-309

9. also old 25. Woods, D.R., et al. (1997) “Developing Problem Solving Skill: the McMasterProblem Solving Program”, J. Engineering Education, April, 75-91

10. Regan-Smith, M.G. et al. (1994) Rote Learning in Medical Schools,” PULSE JAMMA 272,(17) 1380 - 1381

11. Duch, Barbara J., S. E. Groh and D.E. Allen (2001) “The Power of Problem-based Learning:a practical “how to” for teaching undergraduate courses in any discipline,” Stylus Publishing,Sterling, Virginia

12. Ramsden, P. (1983) “The Lancaster Approaches to studying and Course PerceptionsQuestionnaire: lecturer’s Handbook,” Educational methods Unit, Oxford Polytechnic (nowOxford Brookes University, Oxford, UK

13. Woods, D.R. et al. (2000) “Approaches to learning and Learning Environments in Problemn-based versus lecture-based learning,” paper presented att Session 2213, ASEE conferenceproceedings, St Louis MO

14. Schmidt, H.G. (1983) “Problem-based Learning: rationale and description”, MedicalEducation, 17, 11- 16.

15. Personal communication from chemical engineering students in the PBL program atMcMaster University, Hamilton, 2005

16. Queen’s University, Kingston ON, personal communication.

17. Gibbs, Graham (undated) “The A to Z of student focused teaching strategies”, OxfordBrookes University (formerly Oxford Polytechnic), Heddington, Ox, UK

18. Newble, D. and K. Jaeger (1983) “The effect of assessment and examinations on the learningof medical students”, Med. Ed. 17, 165-171

19. Frederiksen, N. (1984) “The real test item bias”, American Psychologist, 39, 193 - 202

20. Savin-Baden, M. (2000) “Problem-based learning in higher education: untold stories”, TheSociety for Research into Higher Education, London, p 94

21. Barrows, H.S. (1994) “Assessment in Problem-based Learning”, Chapter 14 in “Practice-

8-68based learning: problem-based learning applied to medical education”, Southern IllinoisUniversity School of Medicine, Springfield, IL

22. Barrows, H.S., and R. Tamblyn (1980) “Problem-based Learning: an approach to medicaleducation”, Springer, New York

23. Norman, G.R. (1997) “Assessment in problem-based learning”, in “The Challenge ofproblem-based Learning”, D. Boud and G. Feletti, ed., Kogan Page, London

24. Swanson, D.B., S.M. Case and C.P.M. van der Vleuten (1991) “Strategies for StudentAssessment”, Chapter 27 in “The Challenge of Problem-based Learning”, D. Boud and G.Feletti, eds., Kogan Page, London

25. Wee Keng Neo, L. and M Kek Yih Chyn (2002) “Authentic Problem-based Learning”Prentice Hall, Chapter 5

26. Woods, D.R. (1995) “Questions and Answers about Assessment” Chapter 5 in “Problem-based Learning: helping your students gain the most from PBL” Woods, Waterdown, ON andhttp://www.chemeng.mcmaster.ca/innov1.htm

27. Woods, D.R. (1994) “Problem-based Leaning: how to gain the most from PBL”, WoodsPublisher, Waterdown ON Canada distributed by McMaster University Bookstore, Hamilton,ON (1994)

28. Boud, D. (1993) “Enhancing Learning through Self Assessment”, Kogan Page, London

29. Alverno College (1985) “Assessment at Alverno College”, Alverno College Publications,3401 39th St., Box 343922, Milwaukee, WI. 53234-3922

30. Alverno College (1994) “Student Assessment-as-Learning at Alverno College”, AlvernoCollege Publications, 3401 39th St., Box 343922, Milwaukee, WI. 53234-3922

31 Perry, W.H, Jr. (1968) “Forms of Intellectual and Ethical Development in the College Years”,Holt Rinehart and Winston, New York, NY

32. Woods, D.R., R.M. Felder, A. Rugarcia, and J.E. Stice (2000) “The Future of EngineeringEducation: Part 3. developing critical skills”, Chem. Eng. Ed., 34 (2), 108-117

33. Swanson, D.B., G.R. Norman and R.L. Linn (1995) “Performance-based assessment: lessonsfrom the Health Professionals”, Educational Researcher, 24 (5) 5-11, 35

34. Dolmans, D.H., et al. (1997) “Seven Principles of Effective Case Design for PBL”, MedicalTeacher, 19, 185 - 189

35. Drummond-Young, M, and E. A. Mohide (2000) “Developing Cases for use in PBL: fromconcepts to application”, Chapter 8 in “Transforming Nursing Education through Problem-based

8-69Learning”, E. Rideout, ed., Jones and Bartlett, Sudbury, MA

36. Woods, D.R. (2004) “Preparing for PBL” McMaster University, Hamilton

37. Woods, D.R., F.L. Hall, C.H. Eyles, A.N. Hrymak and W.C. Duncan-Hewitt (1996) “Tutoredversus Tutorless Groups in Problem-based Learning”, Am. J. Pharmaceutical Education, 60,231-238

38. Dolmans, D.H., et al (1993) “Problem Effectiveness in a course using problem-basedlearning”, Academic Medicine, 68, 207 - 213

39. Woods, D.R. (1999) “Problem based learning: resources to gain the most from PBL”,Woods, Waterdown, Chapter B, D and downloadable from http://www.chemeng.mcmaster.ca/innov1.htm and click on PBL and download from theresources book.

40. Woods, D.R. (2006) “Helping students gain the most from their PBL experience,” chapter in“The PBL book: the transformation process to PBL”, Erik de Graaf and A. Kolmos, eds.

41. Woods, D.R. (2003) “The MPS program”, McMaster University, Hamilton, ON, Canada andhttp://www.chemeng.mcmaster.ca/innov1.htm and click on MPS for a summary of the researchfindings and some details for many of the units.

25. Woods, D.R., et al. (1997) “Developing Problem Solving Skill: the McMaster ProblemSolving Program”, J. Engineering Education, April, 75-91

42. Knowles, Malcolm (1975) “Self-directed Learning”, Follett Publishing Co., Chicago, IL

43. Tough, Alan M. (1978) “Major Learning Efforts: recent research and future directions”,Adult Education, 28, 4, 250-263 ____________ (1971) “Adult’s Learning Projects”, Ontario Institute for Studies in Education,Toronto, 147-166

44. Albanese, M.A. and S. Mitchell (1993) "Problem-based Learning: a review of the literatureon its outcomes and implementation issues," Academic Medicine, 68, 52-81

45. Vernon, D.T.A., and R.L. Blake (1993) “Does problem-based learning work? a meta-analysisof evaluative research”, Academic Medicine, 68, 550-563

46. Nooman, Z.G., H.G. Schmidt and E.S. Ezzat (1990) “Innovations in Medical Education: anevaluation of its present status”, Chapters 1 to 7, Springer, New York

47. Longworth, N., and W.K. Davies (1993) “Lifetime Learning”, Kogan Page, London

48. American Association of Colleges of Pharmacy (1991) “Position Paper from theCommission to Implement Change in Pharmaceutical Education”, Washington, DC., Nov.

8-7049. Woods, D.R. (2005) “Successful trouble shooting for process engineers: a complete course incase studies,” John Wiley and sons, NY

50. Stewart, James (1982). Two Aspects of Meaningful Problem Solving in Science. ScienceEducation, v66, no. 5, pp 731-749

51. Patel, V.L and C.L. Frederiksen "Cognitive processes in comprehension and knowledgeacquisition by medical students and physicians" in Tutorials in PBL new directions in teachingthe health professional, H. Schmidt and M. deVolder, eds. (1984) They call it "typical" case(exercise) versus "atypical"

52. Frederikson, N., "Implications of Cognitive Theory for Instruction in Problem Solving"Review of Educational research, Fall 1984, vol 54, 363 to 407. Greeno, J.G. in Frederickson'sreview calls exercises reproductive problems (exercises) vs Productive problems vs ill-definedsituations.

53. Smith, M.U and R. Good "A proposed developmental sequence in problem solving ability inclassical genetics the trial and error deductive logic continuum" 1985 NSTA Annual Meeting,Cincinnati, April 54. Kurlik, S., "Problem Solving in School Mathematics" 1980 NCTM Yearbook, Reston VA1980

55. Norman, G.R. (1984) Faculty of Health Science, McMaster University, personalcommunication

56. Elstein, A.S., L.S. Shulman, and S.A. Spratka (1978) “Medical problem solving: am analysisof clinical reasoning,” Harvard University Press, Cambridge, MA

57. D.R. Woods, T. Kourti, P.E. Wood, H. Sheardown, C.M. Crowe and J.M. Dickson (2001)“Assessing Problem Solving Skills: part 1 the context for assessment”, Chem. Engng. Education,35 (4) 300 - 307

58. D.R. Woods, T. Kourti, P.E. Wood, H. Sheardown, C.M. Crowe and J.M. Dickson (2002)“Assessing Problem Solving Skills: part 2: assessing the process of problem solving”, Chem.Engng. Education, 36 (1) 60 - 67

59. De Stephen, R. (1985) personal communication, Hilton Head Conference on TeachingThinking,

8-7160. Woods, D. R., R .R. Marshall and A. N. Hrymak (1988) “Self assessment in the context ofthe McMaster Problem Solving Program”, Evaluation and Assessment in Higher Education, 12,2, 107 – 127

61. Stice, J.E. (1997) “Goals and Assessment”, Workshop at the NSF Workshop, University OfWisconsin, Madison

62. Wankat, P.C. (1997) Personal communication, workshop at the Technical University ofNova Scotia, Halifax, NS

63. Bloom, B.S. et al. (1956) “Taxonomy of Educational Objectives: classification of theeducational goals: Handbook I: Cognitive Domain,” McKay Publishers, New York, NY

64. Biggs, J.B. (1987) “Student approaches to learning and studying,” Australian Council forEducational Research, Melbourne.

65. Johnstone, A.H. and H. El Banna (1986) “Capacities, demands and process - a predictivemodel for science education”, Education in Chemistry, May, 80 ff

66. Kimbell, R., et al. (1991) “The assessment of performance in design and technology,” SchoolExaminations and Assessment Council, London W11 3JB

67. Case, S.M. and D.B. Swanson (1998) “Constructing written test questions for the basic andclinical sciences”, National Board of Medical Examiners, 2nd ed. http://www.nbme.org 68. Saavedra, R., and S.K. Kwun (1993) “Peer Evaluation in Self-managing work groups”, ofApplied Psychology, 78, 3, 450-462

69. Brown, S., C. Rust, G. Gibbs (1994) “Section 2: Setting Standards”, in “Strategies forDiversifying Assessment in Higher Education”, Oxford Brookes University, http://www.lgu.ac.uk/deliberations/oscd_pubs/div-ass2.html April (1999) 70. McPherson, J. et al. (1997) “Assessment in an integrated, problem-based, self-directedcurriculum”, Chapter 16 in “Imperatives in Medical Education: the Newcastle approach” R.Henry, K. Byrne and C. Engel, ed., Faculty of Medicine and Health Sciences, University ofNewcastle, Australia

71. D.W. Johnson, R.T. Johnson and K.A. Smith (1991) “Active Learning: cooperation in the

8-72college classroom”, Interaction Book Company, Edina, MN

72.Verwijnen, G.M, T. Imbos, H. Snellen et al. (1982) “The evaluation system at the medicalschool at Maastricht,” Assessment and Evaluation in Higher Education, 7, 3, 225 - 244.

73. Albano, M.G. et al., (1996) “An international comparison of knowledge levels of medicalstudents: the Maastricht Progress Test”,@ Medical Education, 30, 239 - 245

74. van der Vleuten, C.P.M., G.M. Verwijnen, and W. Wijnen (1996) “Fifteen years ofexperience with progress testing in a problem-based learning curriculum”, Medical Teacher, 18,2, 103 - 109

75. Blake, J.M., G.R. Norman et al. (1996) “Introducing progress testing in McMasterUniversity’s problem-based medical curriculum: psychometric properties and effect onlearning”, Medical Education, 71, 9 1002 - 1007

76.Boshuizen, H.P.A. et al. (1997) “Measuring knowledge and clinical reasoning skills in a PBLcurriculum”, Med Ed. 31. 115 - 121

77. Alverno College (undated) “Critical learning incident” Milwaukee, WI

78. Novak J.D. and D.B. Gowin (1984) “Learning how to Learn”, Cambridge University Press,Cambridge, England

79. Lanzing, J.W.A. (2000) “The Concept Map Homepage”,http://www.to.twente.nl/user/ism/lanzing/cm_home.htm Dec.

80. White, H.B. (2000) “How to Construct a Concept Map”,http://www.udel.edu/chem/white/teaching/ConceptMap.html Dec. (2000)

81. Angelo, T.A. and K.P. Cross (1993) “Classroom Assessment Techniques: a handbook forcollege teachers”, 2nd ed., Jossey-Bass, San Francisco, CA

82. Akiunsanya, C., and M. Williams (2004) “Concept Mapping for meaningful learning”, NurseEducation Today, 24, 41 - 46

83. Patel, V.L. (1990) “A cognitive science perspective on instructional issues,” Chapter 15 in“Innovation in Medical education: an evaluation of its present status”, Z.G. Nooman, H.G.

8-73Schmidt and E.S. Ezzat, ed, Springer, New York, 189 ff.

84. Vigil, A., R.L. Miller, E.D. Sloan (2002) “Structured Trouble shooting in process design,”paper presented session 2313, at the ASEE national conference, June

85. Woods, D.R. (2001) “They just don’t pull their weight” Chapter 22 in “Problem-basedLearning: case studies, experience and practice,” P. Schwartz, S. Mennin and G. Webb, eds.,Kogan Page, London.

86. Paivin, C., V.R. Neufeld, G.R. Norman and L. Walker (1979) “The Triple Jump Exercise - ameasure of problem solving and self-directed learning”, Proc. 18th Conference on Research inMedical Education, Nov. Washington DC

87. Neufeld, V.R. (1999) Faculty of Health Sciences, McMaster University, Hamilton, personalcommunication (1975) and reported in “Problem-based Learning: resources to gain the mostfrom PBL”, D.R. Woods, Waterdown ON and distributed by McMaster University Bookstore,Hamilton ON, p. F-45

88. Branda L.A. (1999) AThe Problem Based Evaluation Exercise, PBEE,@ p F-45ff in AProblem-based Learning: resources to gain the most from PBL,@ D.R. Woods, Waterdown ON anddistributed by McMaster University Bookstore, Hamilton ON (1999).

89. Barrows, H.S. and G.C. Pickell (1991) “Developing Clinical Problem-solving skills” NortonMedical Books, New York, NY

90. Woods, D.R. (1980) Using Trouble Shooting Problems”, Chem. Eng. Ed., Spring 8 – 92and summer, 130 – 136

91. Barrows, H.S. (1982) Faculty of Health Sciences, McMaster University, the P4 deck,personal communication

92. Swanson, D.B., J.J. Norcini and L. Grosso (1987) “Assessment of clinical competence:written and computer simulations”, Assessment and Evaluation in Higher Education, 12, 220 -246

93. Fliegel, J, J.G. Frohna and R.S. Mangrulkar (2002) “Computer based OSCE station tomeasure competence in evidence-based medicine skills in medical students”, AcademicMedicine, 77 (11) 1157-1158

8-7494.Ramsden, P. (1983) AThe Lancaster Approaches to Studying and Course perceptionsQuestionnaire: Lecturer=s Handbook,@ Educational Methods Unit, Oxford Polytechnic (nowOxford Brookes University), Oxford, UK

95. Woods, D.R. A.N. Hrymak and H.M Wright (2000) AApproaches to Learning and LearningEnvironments in Problem-based versus lecture-based learning,@ Session 2213, conferenceproceedings, ASEE, St. Louis, MO

96. Pavelich, M.J. and W.S. Moore (1993) AMeasuring Maturing Rates of Engineering StudentsUsing the Perry Model,@ Frontiers in Education Annual Meeting

97. Pavelich, M.J and W.S. Moore (1996) AMeasuring the Effect of Experiential EducationUsing the Perry Model,@ J of Engineering Education, 85, 4, 287-292

98. Pavelich, M.J., R.S. Culver and J.T. Hackos (1984) AEPICS: Professional projects forfreshman and sophomores,@ Engineering Education, 74,5 279-283

99. Allen, R.D (1981) AIntellectual development and Understanding of Science: application of

100. Marra, R.M., B. Palmer and T.A. Litzinger (2000) AThe effects of a First-year EngineeringDesign Course on Student Intellectual Development as measured by the Perry Scheme,@ J.Engineering Education, 89, 1, 39-46

101. Fitch M.A. and R.S. Culver (1984) AEducational Activities to Stimulate IntellectualDevelopment in Perry=s Scheme,@ Proceedings 1984 ASEE Annual Conference, 712-717

102. Woods D.R. (2000) AMotivating Teachers to Improve Learning,@ Manuscript for text,McMaster University, Hamilton ON

103. Coles C.R. (1985) ADifferences between conventional and problem-based curricula in thestudent=s approach to studying,@ Medical Education, 19, 308-309

104. Hudspith R.C (2000) AImproving the Bridge: Making Engineering Education Broader andLonger,@ Engineering and Society Program, McMaster University, Hamilton, ON

105. Moutros, N.J. (1999) “Portfolio assessment in aerodynamics”, J. of Engng. Ed. 88, (2) 223– 229

8-75

106. Undergraduate Nursing Education (2003) “Guidelines for the Development of LearningPlans” in “Undergraduate Nursing Education”, McMaster University, Hamilton, ON, Canada

107. Woods, D.R. (1997) “Issues in implementation in an otherwise conventional programme,”Chapter 17 in “The Challenge of problem-based Learning,” D.Boud and G. Feletti, eds. KoganPage, UK

108. Woods, D.R. (1996) “Problem-based learning for large classes in Chemical Engineering”,Chapter 10 in “Bringing Problem-based learning to higher education: theory and practice,” No.68, Jossey Bass publishers, San Francisco.

109. Woods, D.R. (2003) “Introducing the MPS program: the McMaster Problem solvingprogram,” Chemical Engineering Department, McMaster University, Hamilton, Ontario, Canada

8-76

Table 1. Validation form for learning objectives

Case problem ________ Date _____________Issues (to be completed by the tutor)Number identified: 1 2 3 4 5 6 7 >7

Agreement with tutor <50% 50% 60% 70% 80% 90% 100%

Knowledge/skills to be learned (to be completed by the student)Consensus among group little some a lot complete

(to be completed by the tutor)Agreement with tutor's list little some a lot complete

Learning objectives (to be completed by the tutor)Quality poor fair OK good excellent

Learning (to be completed by the student)Quality of questions asked during the teach session none some astute excellent

Willingness to continue to contribute <50% 50% 60% 70% 80% 90% 100%

Your Attitude (to be completed by the student)

Perry shift 2 3 3.5 4 4.5 5******************************************************************************

8-77Table 2. Some evidence-based targets for Lifelong learning skills, 8 copyright, Donald R.Woods, 1999

Lifetime learning we define as learning how to learn so that whatever comes our way weempower ourselves to master the new knowledge.

Evidence-based targets Progress toward internalizing these targets

20% 40% 60% 80% 100%

M Consider peers and classmates as resourcesto help me see my learning needs, to help meplan my learning and to provide new ideas so

that I can learn from them.

M Able to assess learning needs realistically.

M Able to create observable, unambiguous andachievable learning objectives to match or

satisfy my needs. Use these to assess progress.

M Can relate to teachers and instructors asresources, facilitators and helpers rather than asthe sole source of knowledge. Have acquired an

attitude toward learning comparable to Perrylevel 5.

M Able to identify people and materialresources needed to achieve my learning

objectives.

M Able to shift from being a dependent learnerthrough being an independent learner to being

an interdependent learner.

M Able to devise a time plan and stick to itreasonably well.

M Willing to assume responsibility andownership for the tasks in learning: (goalsetting, resource identification, learning,

assessment).

M Meets contract commitments to teach others.

M When teaching others, uses the principles oflearning and addresses differences in learningstyles (instead of Areporting information@ and

8-78expecting the learner to sort it out).

M When learning a Adifficult@ topic, willing toaccept the challenge to unravel the complexity

(instead of skipping over it and Ahoping it won=tbe on the exam@).

8-79Table 3. Goals for lifelong learningMPS 36 Self-directed learning or lifetime learning

1.1 given a term listed under "concepts introduced", you should be able to give a word definition, list pertinent characteristicsand cite an example.

2.1 Given a problem, you will ask questions, explore issues so that within 30 min, as a group of five, you will be able to identifyall of the five to six major issues, and these shall agree within 95% of those identified by the tutor.

2.2 Given a problem, you will be able to list the possible knowledge you would need to know to solve the problem; your listshould agree within 80% of the list of others in your group and within 85% with the list of the facilitator/tutor.

3.1 Given a problem, a list of the possible knowledge and resource available, you will create learning objectives and methods ofassessment. These will be 90% acceptable according to the criteria for self-performance assessment (in MPS 3)

3.2 Given the learning objectives and methods of assessment, you will be able to identify reasonable and pertinent sources ofinformation and be able to allocate the resources to achieve the objectives in the time available.

3.3 Given the learning objectives, you will create for yourself examination questions that are consistent with the objectives.These will be judged by peers and or tutor to be 90% acceptable.

3.4 Given that other members of the group have acquired key knowledge pertinent to the situation, you will ask questions so thatyou can learn from them the knowledge you need. You will interact such that they will rate a willingness above 70% to continueto share information with you.

Concepts introduced

Peers as resources, Perry's inventory, learning objectives

8-80Table 4. Form 2804, Self and peer assessment of contributions to team work (developed by C.M. Crowe and usedwith his permission)

______________________________________Name

Assessment of the group

(10) Group worked well together, making the project more interesting. (5) Group usually worked well together, with occasional difficulties (0) Group could not work together at all, making the project more

difficult

Please enter a mark from 0 (worst) to 10 (best) in each row, for yourself and the other members of your team or group. If you cannot make anassessment, enter U for Unable to Observe. Your assessment should be about performance and should be based on evidence you have seen orheard.

Enter your initials in the first cell to the right and those of the other members ofyour group in the remaining cells

self

Leadership

(10) Provided direction, inspired the others ( 5 ) Willing follower, took direction easily

( 0 ) Frustrated the group, blocked progress, criticized the othersCooperation

(10) Worked readily with the others, contributed to the group work ( 5 ) Cooperated with occasional prompting

( 0 ) Rarely contributed, worked mostly alone, had to be coerced into cooperating or would not cooperate.

Initiative

(10) Produced good ideas which helped others, solved problems ( 5 ) Accepted other’s ideas and improved on them

( 0 ) Criticized other’s ideas, never contributed his/her own ideas’Attitude

(10) Positive, encouraging the others to work better ( 5 ) Neutral, worked well with the group without enthusiasm or grumbling

( 0 ) Negative, complained about the project, worked unwillingly with the group

Computation

(10) Helped others to understand and use computer tools better ( 5 ) Good but not innovative in using the computer tools

( 0 ) Uninterested or unable to use computer tools effectively; neededconstant help to become useful at computing

Effort

(10) Worked very hard on tasks assigned by the group or on his/her own ( 5 ) Worked reasonably hard, given other courses and commitments

( 0 ) Did not work much at all, tasks were either not done or were unsatisfactory

Research

(10) Found many additional published papers, contacts or internet sites which greatly helped the project

( 5 ) Read only the assigned material and used that knowledge effectively ( 0 ) Had not read any material about the project, relied on others for

information

8-81Communication and written reports

(10) Organized and wrote major parts of the report ( 5 ) Contributed his/her share of the writing of the report ( 0 ) Made little or no contribution to writing of reports.

Individual assignments

(10) Did all of them on his/her own, helped others ( 5 ) Did most of them, perhaps with some help from others

( 0 ) Either did few of them or most of them handed in were copied. Pertinent technical/subject Knowledge

(10) Excellent grasp of the technical fundamentals for this project and shared and used this effectively

( 5 ) Reasonable understanding of the basics and used this effectively ( 0 ) Little or superficial understanding, relied on others

8-82Table 5 Test and Exams in Process Skills, TEPS, for ALifelong Learning@MPS 36: Self-directed learning: Example assessment tasks:

1. For the learning goal "to explore the environmental issues related to the Red Hill expressway" (the problem posed in PBLCase 3)

(a) If pertinent, break this into further subgoals.(b) Rewrite the subgoal in "observable terms."(c) Create at least one measurable criterion that you be used to measure success in achieving the subgoal.(d) Write out one type of evidence you would collect to show progress toward your subgoal.

2. Given in Table Z are the results of the Jungian Typology and the LASQ inventories and the Perry level for all the members ofyour group. The handout that Ralph has prepared for the teach meeting is given in Table R. Critique the handout based on yourinterpretation of the learning needs and preferences of the members of the group.

3. List the learning preferences and level of Perry development for each member of your group. Document the evidence that youused to reach this assessment.

4. For the PBL Case given in Table X,a. List ten issues you think are pertinent;b. Prioritize these in the context of this course.

5. Given in Table Z are the results of the Jungian Typology, the LASQ inventories and the Perry Level for all the members ofyour group. For the PBL case 3, reproduced in Table X,

a. Identify the topic you contracted to teach the group.b. From the information in Table Z, outline how you would teach the group. Assume you have 30 min.

Create the handout sheets you would use.

8-83Figure 1: Example options for assessing group and individuals within groups

Form A: First of 26 entries

Helpful behaviors

Initiates and contributes ideasArrives on time and is preparedCompletes assigned tasksArrives with questions or commentsShares knowledge

Form B: First of 8 behaviorsFirst of four related to Task

Observer of the task process: Positive: orients the group, monitors, summarizes, seeks direction, identifies phases in the problem solving processNegative: ignores phases, asks whatever he/she wants, blocks Is unaware of his/her contributions

8-84Figure 2: Forms for rating a clinicianForms:1 unsatisfactory outstanding

1 10Effective clinician: G G G G G G G G G G

2Effective clinician: Unsatisfactory area meets area NAknowledge base requires objectives strength

attentionG G G G

3. Effective clinician:

Domain Unsatisfactory Needsimprovement

Meetsobjectives

Exceedsobjectives

Knowledge base:growth and

development;family systems;diagnosis andmanagement of

common pediatric andadolescent problems

Occasional major orunacceptably frequent

minor gaps inknowledge base

G

Frequent gaps inknowledge

G

Solid knowledge basefor common clinical

problems

G

Thorough knowledge ofall basic areas and goodknowledge in additional

areas

G

Domain Unsatisfactory Needsimprovement

Meetsobjectives

Exceedsobjectives

Knowledge base: Occasional major or Frequent gaps in Solid knowledge base Thorough knowledge of

4.Effective clinician: Knowledge base: has knowledge base in fundamentals, can recall correctly values from diagnostic tests for healthy infants, children teens, adults,elderly and can give casual relationship between potential diseases and numerical values from diagnostic tests, can list positive and negative testsfor 15 diseases, can list 2 dominant and 3 inferential symptoms for 38 illnesses, can list 15 history-taking questions and prioritize them, cancorrectly hold 13 different instruments and use them with patient satisfaction greater than 95%. ..

None of Few of these Most features All of thesethese behaviours but demonstrated behavioursbehaviours major omissions G G G G G G G

8-85

Assessing Problem solving skills. Woods, D.R., T. Kourti, P.E. Wood, H. Sheardown, C.M. Crowe, J.M. Dickson (2001)“Assessing Problem Solving skills: part 1, The context for Assessment,” Chem Eng. Education ,35, no. 4, Fall, p. 300 - 307

The five principles of assessment provide a framework for developing and using instruments for the assessment forstudent performance and of the evaluation of the program effectiveness. Crucial to any assessment is the creation ofpublished goals and measurable criteria that form the context for the performance of the student. Evidence should begathered and assessed in the context of these goals and criteria. Four example sets of goals nd criteria werepresented.

Assessment is based on evidence of performance. Seven options for gathering evidence were described in thispaper. Eight options that focus more on measuring the problem-solving process will be given in Part II. The first twooptions of evidence (mark the answer and tell the student the process is important) provided exam scripts in subjectdiscipline, where correctness of the answer, subject knowledge and problem solving were being marked. Weelaborated on the challenges of using conventional exams as a measure of problem-solving skill.

Woods, D.R., T. Kourti, P.E. Wood, H. Sheardown, C.M. Crowe, J.M. Dickson (2002)“Assessing Problem Solving skills: part 2, Assessing the process of problem solving,,” ChemEng. Education, 36, no. 1, Winter, p. 60 - 67

Assessment is based on evidence of performance.

Fifteen options for gathering evidence were described, seven in Part 1 and eight in this paper. Data from student’sperformance in the McMaster problem Solving program were used to suggest that:! Tests and Examinations of Problem Solving skills (TEPS) can be developed that are consistent with the publishedgoals and criteria. These results, however, are least highly related among the measures.! Self-assessment and journal writing are highly related measures of problem solving skill.! The use of a written script gathered for the sole purpose of displaying ten to twenty minutes of thought processesused in problem solving was not a statistically significant measure.! We recommend using a variety of forms of evidence. We also recommend the ue of a contract mark in which thegoals and criteria are published and regarding which the students have some choice in the relative weighting of theevidence.! The results from exam scripts in subject discipline - where correctness of the answer, the subject knowledge andproblem solving were being marked (Option 2) - gave one form of evidence, but it should not be used as the soleform of evidence. The challenges of using conventional exams as a measure of problem solving skills w3eredescribed in Part 1. Even addressing these challenges as best we could, we found no relationship between the exammarks in chemical engineering subject discipline courses and any of our measures of problem solving skills.For program evaluation, the results from such published tests as Basadur’s attitude toward ideation, Heppner’smeasure of confidence in problem solving, and Billings-Moos’ measure of problems solving performance andavoidance are useful for pre- and post data about the effectiveness of programs. We recommend that instruments beused, including the items on exit surveys, where benchmark data are available for comparable populations.

8-86

a Chapter for the book “Case Studies on PBL,” P. Schwartz and S. Mennin, eds., Kogan Page

They just don’t pull their weight! a180

Case Reporter: Don Woods.

Issues Raised

Individual accountability in small tutorless groups. A student complains that she is doing all the work. She says thatothers commit to “teach” subjects but come ill-prepared.

Background

The third year course in Chemical Engineering includes six weeks (18 hours of class contact time) of small group,self directed, self-assessed problem-based learning. I handle a class of 30 to 50 students and assign the students togroups of five or six. The groups have no tutor assigned to each group because there is only one instructor. Duringthe first two weeks students are introduced to the skills and attitudes through a series of four, 1 ½ hour workshops.The topics for the workshops are 1) what is PBL? Why is it important for you? and managing change; 2) problemsolving skills; 3) how to work effectively in groups and 4) how to be an effective teacher. To remind students of their learning attitudes, styles and preferences, all students share with their group members theresults of the Perry inventory (an instrument to explore the student’s attitude about their role in the learning), theJungian typology (which suggests student preference for theory versus worked examples, generalities versus details)and the Lancaster Approaches to Studying (which explores the student’s predilection for rote learning, search formeaning or work hard in the context of the published syllabus).In each of the weeks following the preparation workshops the students cycle through three meetings for each of thefour sequential “problem cases.” I had worked successfully with fourth year students for the past six years, but at the students’ request have movedsome of the PBL material into a third year class. This is the first time I have tried this approach with third yearstudents.

For the PBL case problems, each group holds three scheduled meetings. During the first, the Goals Meeting, thestudents read the case, list the learning issues and have these validated by me. They contract the teachingresponsibilities from the learning issues with each person contracting to return and teach the other their topic.Although I provide a written contract form, most groups create their own contract. The second meeting is the“Teach Meeting” where each person teaches with each others the topics they have researched. Each student is givenwritten feedback from everyone in the group about the quality of the “teaching.” The final meeting is the “FeedbackMeeting” where each group selects a reasonable “exam” question based on the case. Each group’s question is givento another group to solve. I monitor the results of the exam questions posed and the answers created.After each cycle of three sessions, each student writes a Reflective Report of how well their lifetime learning skillsare developing. The analysis includes the feedback sheets from the teach meeting, the example test question from theFeedback meeting, the learning objectives, and a personal analysis of the strategies used to teach and the shift inattitude about PB and the acquisition of lifetime learning skills.

Part One

A knock on my door. Maria, a conscientious B student in my third year class asked, “Can I see you a moment aboutsomething that is bothering me about PBL.” “Please come in and tell me about it,” I encouraged.Maria sat down and launched into her story.“For the first case, everything seemed to go OK. Some of the teaching done by the others was “fair,” but I put thatdown to our first attempt at this new form. The second time around, everyone came to the teach meeting. Althoughwe all contracted to prepare and distribute “teach notes” I was the only one with “teach notes” to distribute. I tried

8-87

my best to teach them, but they didn’t seem to be paying much attention. When it came their turn, they all did alousy job of teaching. No notes. Very superficial. No worked examples.”

“I’m sorry to hear about that.”

M What do you think I should do?M What are the issues being raised by this situation?M What is my role in this scenario? Do I go into class and tell them what should be happening? Do I move Maria

to another group? Do I ask permission to sit in on her group to see for myself Maria’s problem?

Part Two

Quickly, I thought through the range of issues that might pertain:Is Maria expecting too high a standard? Does she have a lousy group?Did I not give them enough pre PBL training or perhaps what I offered was ineffective?Is the method of contracting insufficient to get a commitment from all?Are the others living up to the contract but Maria is expecting more?Are they ready for PBL? Are the students too comfortable with the “lecture system” and they really are notready for or are unwilling to accept PBL?Is this an issue that will ‘resolve itself’ with maturity with PBL? Is the problem with Maria? with the others in her group? with trying to introduce this at too low a level? withme and how I set it up?

I concluded that it was time to get more information to help identify a root cause.“Who are the other members of your group?” I ask.“ Jason, Brad, Margarite and Suzelle.”

I reflect on how I set up the groups. This was typical group with a mix of academic backgrounds.Jason, an A+ student, mature and headed for graduate school eventually. Brad, a C- student. Active in sports. Rather casual attitude about school and not working to his potential.Margarite, C+ student. Very active in campus activities. Conscientious student working close to her potentialacademically. Suzelle, B student. Transfer into the program from a work term. An unknown quantity for me and her classmates.Seems to work hard. OK this seems like a reasonable group.“How do you feel about your group?”“I don’t know Suzelle very well but she seems OK. I expected Jason to be a conscientious driving force in the group.The others are OK. But they’re just not pulling their weight.” “Did it seem to work out OK the first time?”“Yes, but none of us knew what to expect. In hindsite, we probably didn’t do a good job.”I could follow up on this, but this seems like a group that should work well. I wonder if it is the contracting (from theGoals meeting), their understanding of the teaching task, the delivery in the Teach Meeting or their generalresistance to change? To check out the potential for resistance, I asked for the Perry scores.Maria responded, “Mine is 4.2; Jason is 3.8; Suzelle is 4.0; Margarite is 2.9 and Brad doesn’t remember his. I wouldput him about 3.”OK, I thought. Margarite at 2.9 I expect to be struggling. She probably wishes I would lecture. She’s not as ready asI would hope. Brad. Oh, that’s Brad not to have his score. He’s probably a little higher than 3 but... This is a typicalmix. I wish they were all 5 but...Jason is a surprise. Perhaps he has succeeded so well by having a lot of mentalhorsepower and having learned how to use the system effectively. He may feel a little threatened with this new formand be resisting.

“How about the Lancaster scores?” “Which ones are those?”

8-88

“The three numbers in the range 15 to 30 about strategic, surface and deep learning.”Maria looked through her notes and handed me the summary sheets that all students completed for their groupmembers:

Maria, high on deep learningJason highest on StrategicSuzelle, Margarite and.. Brad’s is missing... these two show highest on surface or rote learning. I would expectBrad’s to be the same.

“Thanks,” as I hand the paper back to Maria without further comment.

To check out the contracting, let’s hear what they did during the Goals Meeting.

“Can you lead me through the Goals Meeting for the second case?” “Brad was assigned to chair the meeting. He didn’t circulate an agenda although we all had an idea about what to doso that probably wasn’t a big deal. We spend the first 20 minutes brainstorming the issues and prioritizing them.Then you checked them as 80% complete and helped us to see the additional learning objective we should include.” “Yes, your group did a good job on identifying the learning issues. Then, how did you contract?”“ We checked over the list; each identified a topic they would work on. We said that we would teach the same as thelast time. Each person completed the feedback form about how the meeting went and gave it to Brad, and we left. ” “ The contract was the hunk of paper saying “We the undersigned agree to meet out obligation to research, study andteach our assigned.....” Everyone was expected to sign it to help meld the contract among group members. Also, ifyou recall I suggested that each group discuss individual member’s preferred way of learning and that each prepareteach notes to be distributed to all members of the group. Did your group do any of that?” “We did a bit of that the first meeting. The discussion about teaching style was brief. We just had the completed list Ishowed you... except, of course, for Brad.”“What about the teach notes?”“The first time we did agree by everyone nodding their heads. We didn’t sign anything at either meeting.” ... pause...“I’d really like to transfer to Tony’s group. Everyone in their group brought teach notes to both meetings. I looked atthem. They were good. I must get on the Dean’s honour list this year to retain my scholarship. I may want to go tograd school. I really don’t want to have my grades go down the drain because the others don’t care. You have controlover that, not me!”

M What does this information tell you about the situation?M What other questions would you like to have asked?M Did I respond empathetically? How could the responses to Maria be improved?M Should I have followed up on Maria’s insights about the first cycle?M What would you do next?M How do you respond to Maria’s request to transfer to Tony’s group?

Part Three

So far I realized that although Maria thought there was a contract to provide “teach sheets” this contract does notseem to have been created or signed. The other members did not have the expectations.

The students in this group, and probably in the others, are not as prepared as I had hoped. For example, sophomore students, when given the choice to select a PBL environment, are compared, in thefollowing table, with third year students in traditional programs where I tried to introduce PBL in the followingtable.

8-89

second year third year cohort Comments about the thirdcohort who where I introduced year cohort relative to theelect PBL PBL ideal

Perry > 4.5 3. 74 unwilling to see the teacher as resourceLASQ Strategic 16 16

LASQ surface 13 15 stronger preference for surface learningLASQ deep 15 14 less preference for deep learning

Based on the relatively low value of the Perry inventory I should add more structure and individual accountability.The LASQ scores help students approach the teaching task. I should not directly interfere with group processing. Idecided to have a brief “State of the PBL” talk with the class as a whole. I would

1. Remind all the student groups of the elements I have found critical to the success: “contracts” and “teach notes.”Some examples from the fourth year class were posted to illustrate options. Remind them of the expectation thatthese would be addressed in the Reflective Report.

2. Remind all the student groups of the implications of the Perry and LASQ data and describe how this informationwill help them.

3. Explain that I would try to help them by monitoring the individual progress by having class ombudspersons andby creating a feedback form for peer and self rating about individual contributions to the team.

In addition, I decided that Maria’s request to be transferred to another group should be denied promptly now. Here’show I handled it.

“Maria, you recall that when I set up the groups I announced that I was assigning people to groups where Ihoped each would encounter new people with completely different styles from your own. I hoped that youwould have to learn to deal positively with differences. I wanted you to learn this skill while you were still inschool. When you are working as a professional, you may be assigned to work with a team - a team containingmembers with whom you have difficulty working. You cannot request to be assigned to another team! Norwould I be helping you here if I transferred you to Tony’s group. I realize you are on the path to graduateschool, and I want to help you get your scholarships. On the one hand, I see that I have succeeded in putting youin a group where you are encountering differences in approaches.. in particular differences in acceptingresponsibility. On the other hand, let’s see what aspects from the workshop on coping with conflicts andassertiveness might apply to this situation. Which ideas might be pertinent?”

Maria thought for a moment and then suggested, “I probably should express my feelings assertively by sayingsomething like “When I don’t receive written teach material in the Teach Meetings, I feel frustrated because Ithought we had all agreed to supply that material.” and then see how they respond.”

“That sounds good. You have modelled a good assertive statement that should open up some communication. Youmight add to your statement a request, “Can we talk about this?” rather than just waiting to see how they respond. Inaddition, since you might not be the only one experiencing frustration, I will be telling all the groups about theimportance of written contracts and teach notes, and of the value of the LASQ and Perry inventories to guide inpreparing to teach. I also will probably ask everyone to complete feedback forms about contributions to the group. Is there anything else I can do to help you?”

“No, I’ll try the assertiveness and hope that your general comments will help our group.”

8-90

“Please keep me posted on how it works it.”

The options I rejected were:- To have all groups hand in “teach notes.” This would give me direct evidence about the quality of the

preparation by each person. This option would make each person accountable. However, I believe that the groupneeded to be able to decide on their own how best to handle the teach meetings. I did not want to require themto create “teach notes” just for a mark or just for me.

S To have all groups hand in “contracts” because my experience with the final year students had suggested thateach group handled this is different and imaginative ways.

Here is how ombudspersons worked. I asked the class to appoint four class representatives who would give meoccasional feedback about how well the learning was progressing and offer suggestions for improvement. Thisworked extremely well. They raised issues that were easy to explain, rationalize and change. Above all, the studentsfelt that I was concerned and aware of what was happening in class.

Here is the individual accountability form, the data I received and how well this worked. For the form, I listed a setof desired behaviours and criteria. Some elements I included were leadership, cooperation, initiative, attitude, effortand individual assignments. The form included example rating criteria. For example, for leadership a rating of 10means: provided direction and inspired others, 5: willing follower, took direction easily and 0: frustrated the group,blocked progress and criticized others. Similar descriptors were given for each element. Each student was to ratehim/herself and all of the other members. The rating forms were given directly to me and not seen by anyone else.

What wee the results? Unless you see the actual numbers it is hard for colleagues to see that this really works. Hereis an example of the results for Maria’s group. First, the students rated themselves as a group 8.3 out of 10. Thearithmetic average of the scores for all elements on the form are given in the body of the table with the self-ratinggiven in bold. Thus, Maria rated herself highly at 9.5 yet her overall average rating for all members of the group was8.22. Maria was rated highly by the other members with all ranking her as first. I have added the ranking to eachentry. That is, Brad rated Marie as 9.2/10, Jason 6.9/10, Suzelle 7.6/10, Margarite 6.2/10 and assigned the sameaverage score for himself. Based on these ratings by Brad, Marie is ranked first; Suzelle, second and so on.

Grouprating: 8.3(8.3)

As seen by Score for individual andstandard deviation

Maria Jason Suzelle Brad Margarite

Maria 9.5 1 8.95 1 9.1 1 9.2 1 8.6 1 9.07 [0.33]

Jason 8.5 2 8.55 3 7.9 4 6.9 3 7.2 4 7.81 [0.75]

Suzelle 8.4 3 8.95 1 8.7 2 7.6 2 7.8 3 8.28 [0.59]

Brad 6.8 5 7.43 4 7.6 5 6.2 4 4.9 5 6.6 [1.09]

Margarite 7.9 4 7.43 4 8 3 6.2 4 8.3 2 7.57 [0.82]

Ratingscale usedbyindividual.

8.22 [0.98]

8.26 [0.78] 8.26 [0.62] 7.22 [1.25] 7.36[1.47]

average of ratingstandard deviations 1.02

These results were obtained early in the semester. Once everyone realized that this form wassome measure of accountability, this group started to communicate with each other and theirperformance improved. This table is representative of the results from the seven other

8-91

autonomous groups. This particular self and peer rating form worked well in that:- The scores are relatively consistent. In this example, except for Margarite, the self rankingswere consistent with the peer rankings.- Just the fact that such rating forms were completed and the results seen by the tutor meant animprovement in individual accountability. Individuals could not get away with poor performancewithout receiving low ratings from their peers. The diligent workers feel satisfied that theirdissatisfaction with their peer’s performance is reflected through their ratings that, in turn, areseen by the tutor.

One issue remained for me. What would I do with the data? I could return the original ratingforms to the groups; I could summarize the results in Tables, as illustrated above, and give theseto each group; I could use this as evidence about individual performance and assign a “mark” toeach individual; I could use the data in conjunction with subsequent collections of the same datato help see individual progress; or I could take no action and treat this as “gathered data.” Imulled this over for a while, and thought that I might bring it to the class (or to the classombudspersons) for advice. As time passed, no one asked about the results so I just took noaction. The information provided interesting benchmarks for me to see the team performance.The students knew that I knew about their performance. I think they no longer felt that they couldhide in the group.

Maria saw me in the hall about three weeks later and reported that her group was really workingwell. I didn’t ask her to elaborate so I don’t know whether it was the use of ombudspersons, herassertive statements and work within the group, my “State of PBL” address, the use of the ratingform or the Reflective Report. Perhaps I should have asked her.

M How do you feel about the way I handled this situation? M If you elected to use a rating form, how would your form differ from the one I used?M How frequently do you think a rating form might be used during a semester?M In my situation the students seem to have responded positively by just knowing that I saw the

results. What would you do with the results from the rating form? M What other ways might be used to improve individual accountability within tutorless groups?

Discussion

The two key issues in this case are how can PBL be used in large classes where individualstudent groups do not have a tutor and what might be done to overcome uneven participation ingroups.

Working with tutorless groups:

PBL has been successfully used in large classes where the student groups are without a tutor (1-4). Don’s particular approach is to:

M Spend about 6 hours of workshop time upfront to develop the student’s confidence in their

8-92

skills. He believes that the students need these because the groups are autonomous, and thestudents should have the skills needed to solve their own problems. By way of contrast, inprograms where a tutor is available for each group, the tutor is trained in these skills:problem solving, group process, conflict resolution and sensitivity to how to teach. Fortutorless groups, Don develops the skill for each group member. These are seen a valuedoutcomes of the program. In this Case, Don could use Marie’s familiarity with and skill inassertiveness as a suggestion for dealing with the apparent conflict. Details about how to runthe workshops are available (5).

M Empower students with self awareness of their preferred styles of learning and those of the

peers in their group. Don has considered a variety of questionnaires and has settled on two: Jungian Typology (6) and the deep versus surface learning style using the LancasterApproaches to Studying Questionnaire (7). In Part Two of this Case, the Lancaster scoresprovided Don with key information that helped him diagnose potential areas for conflict. Atthat time, Maria did not seem to appreciate the significance of this information. Later in PartThree, Don again used the LASQ data to help him realize the difference between theapproaches taken by his third year students and the target approaches. Little is said in thisCase about the implications of the Jungian typology. Don uses the S-N dimension of thattypology to help students identify their preference for teaching by examples and details (apreference for dominant S students) compared with a preference for teaching theory andglobal issues (a preference for dominant N students). More is given in the section B.6 on aworkshop on Self-directed Learning (5).

M Try to give the PBL approach the best chance to succeed. Unless students have chosen toenter a PBL program, most prefer the “same old lecture song and dance.”(8) Students havelearned how to make the familiar lecture system work well for them. They resist change, andPBL is certainly a change! Don took five effective actions but missed on three additionalcomponents that, for this class, were critical. The five effective actions were:

+ He explained why PBL was such an excellent learning environment and used avideotape to show the expected behaviours in the three PBL meetings. + He ran a 1 ½ hour workshop on how to manage change (details are available, 5 ).+ All students completed the Perry inventory,(9) and he used the results to help thestudents understand the attitude shift expected and Don’s role as a tutor. More about thisa little later.+ He assigned a text written to help students understand PBL (10).+ Together with the class he did a “force-field analysis” of the factors that would makethis a great success and those that impede the success (as recommended in the text, 10).

But Don missed on three issues that are critically important when introducing PBL. - Don should have set up ombudspeople right at the beginning to provide him withongoing feedback about how well things were progressing. In the Case, he didn’t do thisuntil after Maria came to see him, in Part Three.

- He should have paid more attention to the differences in Perry scores between the

8-93

“ideal” and his third year students. He had the data, as described in Part Three. He used ita bit but he should have gone further. Here is some background about Perry and the Perrymodel. For any change in learning environment, Perry’s Model of IntellectualDevelopment (9, 10) (or an equivalent model such as King and Kitchener’s Model ofReflective Judgement (11)) provides a good framework to help students cope with theexpectations of the new environment. According to William Perry, a Harvardpsychologist, people progress through some or (in rare cases) all of nine stages ofdevelopment. Most college students can be found on Levels 2–5:

• Level 2 (Dualism). Every point of view is either right or wrong. All knowledge isknown and obtainable from teachers and texts, and the student’s task is to absorb whatthe teacher presents and demonstrate having done so by repeating it back. Confusionoccurs if the text and the teacher do not agree. Dualists want facts and formulas anddon’t like theories or abstract models, open-ended questions, or active or cooperativelearning found in PBL.

• Level 3 (Multiplicity). Most information is known, but there are some fuzzy areaswith questions that still have no answers but eventually will. The teacher’s role is both toconvey the known answers and to tell students how to learn. Students start usingsupporting evidence to resolve issues rather than relying completely on what authoritiessay, but they count preconceptions and prejudices as acceptable evidence and once theyhave reached a solution they have little inclination to examine alternatives. Open-endedquestions and cooperative learning are still resented, especially if they have too much ofan effect on grades.

• Level 4 (Transition to relativism). Some knowledge is known, but some is not andprobably never will be. Students feel that almost everything is a matter of opinion andtheir answers are as good as the teacher’s. The teacher’s task is to present knowninformation and to serve as a role model that can be discounted. Independent thought isvalued, even if it is not substantiated by evidence, and good grades should be given tostudents who think for themselves, even if they are wrong.

• Level 5 (Relativism). Students in relativism see that knowledge and values depend oncontext and individual perspective rather than being externally and objectively based, asLevel 2–4 students believe them to be. Using real evidence to reach and supportconclusions becomes habitual and not just something professors want them to do. Different knowledge is needed in different contexts; there is no absolute truth with goodanswers existing once the conditions are known. The student’s task is to identify theconditions and to choose the best ideas, with the teacher serving as a resource. Studentsat this level are comfortable with both positive and corrective feedback.

A student at one level (in this Case the third year cohort is between level 3 and 4 as noted in PartThree) being asked to function at a higher level (ideally in PBL the students should be close tolevel 5) is likely to be under a great deal of stress, especially if the required level is more than

8-94

one removed from the student’s current level. Their reactions to this stress account for theresistance and occasional hostility instructors often encounter when they begin to use PBL (10).In this Case, Don experienced some of that anger in Maria’s request to transfer to another groupand her concern about losing her scholarship. On the positive side, Don (and Maria) used thePerry information, in Part Two, to help diagnose that the group might have an attitude problem.In particular Margarite and Brad were probably just not ready to accept their new role.

Don could have helped by spending more time upfront in the change management workshopaddressing the issues raised from the results from the Perry inventory. For example, he couldhave had brief role-play activities for the full class to illustrate typical responses of persons withPerry level 3 and contrast that with behaviour at Level 5. He could have worked through moreactivities in Chapter 1 of the required text (10).

- Don’s third mistake was that he did not anticipate that uneven participation would be a majorissue in tutorless groups. Ironically, he had coauthored an article that identified this as a major issue(12). Part of the role of the tutor is to create, from the beginning, a learning environment wherestudents are both empowered with most of the elements in the learning process and are accountableto their peers and to the tutor. In this Case, Don faced the reality of students being able to hide in thegroup only when Maria brought it to his attention. Then he had to back-peddle and try to set up theground rules after-the-fact. Student performance and contributions need to be made visible to thetutor -especially when the tutor is not sitting with the group! What should he have done? At thebeginning, he should have indicated the means that would be used to make behaviour visible to alland the importance placed on such evidence.

Overcoming uneven participation in groups.

Uneven participation and lack of attendance are the top two major issues for tutorless groups (12).The options Don considered explicitly, especially in Parts Two and Three, all dealt with makingindividual contributions explicit and visible. He considered, and rejected, requiring that individualshand in the learning contracts or the teach notes.

Make individual contributions visible: Learning contracts: Knowles (13) advocates the use of learning contracts. For engineering students, Don had found thatmost groups were more willing to create their own contracts than to use a prescribed version such asthe one proposed by Knowles. The emphasis needs to be on the importance of the creation of thewritten contract. The contract can then be used as evidence in student’s Reflective Reports or handedin separately as evidence of performance in the Goals Meeting Make individual contributions visible: Teach notesAgain this idea was rejected by Don. Teach notes or concept maps provide excellent evidence aboutthe quality of the teaching provided by individual students in the Teach Meeting. The notes can beused in the Reflective Reports, handed in to the tutor, marked by peers from other groups or puttogether as a set of notes from each cycle of PBL.

8-95

Make individual contributions visible: Use a self and peer rating formDon had success with this approach. He realized that the rating scales needed to have descriptors oneach criterion. The form was carefully designed. The student response was very encouraging. Asoutlined in the Case, many different uses can be made of the data.

An option implicit in this Case involved Maria’s assertiveness and her willingness to bring the issueto the group, as described in Part Three. The explicit activity Don should have used was to ask thegroups to create their own guidelines about how to handle potential problems before they occur.Maria had to do this after the fact. Some options to use before the formal PBL session begin are:

Use commitment charting to initiate the discussion

Commitment charting is a technique where each individual privately notes his/her personal priorityand hours willing to devote to this particular PBL activity. Then, within the group this information isshared and used for a discussion and resolution of differences. In this Case, the rest of the groupcould have heard of and understood Maria’s desire to retain her scholarship. They would havediscovered that Jason has to balance his school work with a part-time job and that Brad is captain ofthe basketball team that seems headed for the championship this year.

Use a norms meeting to address the issues

Require each group to have a two-hour “norms” meeting during which the group decides how it willhandle such major issues as fair share of the workload and attendance. A listing of 17 major issues isgiven in the Resources Book (5) and a less structured collection of issues is published (12).

Questions for Personal Reflection

M What have you discovered about working with tutorless groups and how does this affect whatyou do in the classroom?

M What ideas from this Case apply to tutored groups?M Don placed much emphasis on the Perry and LASQ inventories. What is your reaction to this

type of information? Can it help the groups? Which data might be most helpful in your context? M Does this Case have any other implications for your own teaching practice?M If you were writing this Case addressing the issues of tutorless groups and uneven performance,

how would it differ? What issues would you emphasize? What main messages would you wish toleave with the reader?

References

1. D. E. Allen, B. J. Duch and S. E. Groh (1996) “The Power of Problem-based Learning in TeachingIntroductory Science Courses,” in “Bringing Problem-based Learning to Higher Education: Theoryand Practice,” L.Wilkerson and W.H. Gijselaers, eds., New Directions for Teaching and Learning,No. 68, Jossey-Bass Publishers, San Francisco.

8-96

2. P. K. Rangachari (1996) “Twenty-Up: Problem-based Learning with a Large Group,” in“Bringing Problem-based Learning to Higher Education: Theory and Practice,” L.Wilkerson andW.H. Gijselaers, eds., New Directions for Teaching and Learning, No. 68, Jossey-Bass Publishers,San Francisco

3. D. R. Woods (1996) Problem-based Learning for Large Classes in Chemical Engineering,” in“Bringing Problem-based Learning to Higher Education: Theory and Practice,” L.Wilkerson andW.H. Gijselaers, eds., New Directions for Teaching and Learning, No. 68, Jossey-Bass Publishers,San Francisco

4. N. Winslade (1994) “Large Group PBL: a revision from traditional to pharmaceutical care-basedtherapeutics,” Am. J. Pharma. Educ., 58, 64

5. D. R. Woods (1999) “Problem-based Learning: resources to gain the most from PBL,” WoodsPublishing, Waterdown ON distributed by McMaster University Bookstore, Hamilton ON andpartially available from the WWW at http://chemeng.mcmaster.ca/innov1.htm and click on PBL, Jan2000.

6. Jungian Typology, Myers Briggs Type Inventory (MBTI) Consulting Psychologists Press, 3803 E.Bayshore Rd., Palo Alto, CA 94303. Other variations are available, see C. Hogan and D. Champagne, "Personal Style Inventory," University Associates Annual, San Diego, CA (1974) or D. Keirsey andM. Bates "Please Understand Me," Gnosology Press, Del Mar, CA. (1984) andhttp://www.keirsey.com 7. Lancaster Approaches to Studying Questionnaire: P. Ramsden (1983) “The Lancaster Approachesto Studying and Course Perceptions Questionnaire,” Educational methods Unit, Oxford Polytechnic,UK.

8. M. A. Benvenuto (1999) “In an Age of Interactive Learning, Some Students Want the Same oldsong and dance,” The Chronicle of Higher Education,” June 4, B9.

9. W. G. Perry, Jr (1968) , “Forms of Intellectual and Ethical Development in the College Years,”Holt Rinehart and Winston, New York, NY

10. D. R. Woods, (1994) "Problem-based Learning: how to gain the most from PBL," Woods,Waterdown, ON, distributed by McMaster University Bookstore, Hamilton ON.

11.P. M. King and K. S. Kitchener (1994) “Developing Reflective Judgement: understanding andpromoting intellectual growth and critical thinking in adolescents and adults,” Jossey-Bass, SanFrancisco, CA

12. D. R. Woods, W. Duncan-Hewitt, F. Hall, C. Eyles and A.N. Hrymak (1996) “ Tutored versustutorless groups in PBL,” American Journal Pharmaceutical Education, 60, 231-238

8-97

13. Malcolm Knowles (1975) “Self-directed Learning,” Follett Publishing Co., Chicago, IL

9-1

9. How to evaluate the effectiveness of your PBL course: Donald R. WoodsMcMaster University, Hamilton, CanadaOct 3, 2002

Case 7: “I liked the course and the students liked it” ““How did your course go this term?” asked Lim. “I liked it and the students seemed to like it. Iguess I won’t change.” “But was your course cost effective? Was it the most effective learning environment for yourstudents? How well did the students achieve the program goals?” asked the administrator.

An embarrassed silence followed. “Well the students passed the exams, and they were difficultexams this year. And 100% passed the course.”

Case 8: Is it worth it? This past year the Polytechnic spent over $500,000 S to help different schools introduce andextent PBL. Is it worth it? Is that how we should be spending our money?

*******************

Activity: In small group of 5 or 6, with chair _______________________________; reporter ________________________, brainstorm the issues this case raises. Identify what youknow already. Identify what you need to learn.

Prioritize the issues: criterion: what do you want to gain from this workshop in the context of theissues raised.________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

9-2

Feedback about the group work. Form 2802

Task: Problem defined, many issues and hypotheses explored, criteria listed and the issues prioritized. Refrainedfrom early closure. Task carried out and looked back at the result to assess it. Group agreement as to goals. Processwas active with monitoring. Completed task on time. The accuracy in the group’s answer matched the time available.Group avoided contributing excessive information. None of Few of these Most features All of thesethese behaviours but demonstrated behavioursbehaviours major omissions

G G G G G G G1 2 3 4 5 6 7

Morale: Group relaxed; enjoyed working together. They gave emotional support to each other and were able toexpress disagreement or disappointment directly. Seven fundamental rights preserved. Members are enthusiastic andinvolved. None of Few of these Most features All of thesethese behaviours but demonstrated behavioursbehaviours major omissions

G G G G G G G1 2 3 4 5 6 7

Individual Contribution to Task and Morale

____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Group Strengths Group Areas to work on_____________________________________ ________________________________ ____________________________________ _____________________________________________________________________________________________________________________________________________

from D.R. Woods (1995)

9-3

Feedback from the Goals Meeting

Case 7 or 8:IssuesNumber identified: 1 2 3 4 5 6 7 >7

Agreement with tutor <50% 50% 60% 70% 80% 90% 100%

Knowledge/skills to be learned Consensus among group little some a lot complete

Agreement with tutor's list little some a lot complete

Learning objectivesQuality poor fair OK good excellent

LearningQuality of questions asked during the teach session none some astute excellent

Willingness to continue to contribute <50% 50% 60% 70% 80% 90% 100%

Your AttitudePerry shift 2 3 3.5 4 4.5 5******************************************************************************

Reflections:______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________-

9-4

“Assessment and Evaluation are judgements made to decide how well goals have been achievedbased on measurable criteria and based on evidence”. When we judge the performance of aperson, traditionally we call this “assessment”. When we judge the performance of a program,school, company or institution, traditionally we call this “evaluation.” Assess people; evaluateprograms.

The same five fundamental principles apply to both assessment and evaluation:

Principle 1. The judgement is based on performance - not personalities. Principle 2. The judgement is based on evidence - not feelings.

Principle 3. The judgement should be done for a purpose with clearly-defined performanceconditions. Before and after? At the end of a series of courses? This year versus last year?

Principle 4. The judgement done in the context of published goals, measurable criteria andpertinent, agreed-upon forms of evidence.

Principle 5. The judgement should be based on multidimensional evidence. For example, in Canada, Maclean’s magazine annually evaluates universities. The forms ofevidence they use include: about the students: the average entering grade of the students, theproportion who have marks higher than 75%, the proportion of students entering who graduate,the number of “out of province” students in the undergraduate program, the number of foreignstudents in the graduate program, and the number of awards received by students. For classes:the class size, and the number of classes taught by tenured faculty. For the faculty: number ofprofessors with PhD’s, awards per full time faculty member; amount of research grant fundingreceived. For the budget: operating budget, the % of the budget that goes to scholarships andbursaries and the % of the budget that goes to students services. For the library, the totalholdings, the holdings per student, the acquiistions and the expenses. For reputation: alumnisupport and survey of major industries and institutions about the reputation.

Following Principle 3, we should evaluate programs to continually improve the learningeffectiveness and to help us decide on priorities and funding.

1. Evaluation to monitor a course while it is in progress

We should use methods to give us feedback while the course is in progress. I think this shouldbe done in every course, but especially in PBL courses where the students percieve the learningenvironment to be different than expected.We can monitor our courses by:

- class ombudspersons- one minute papers- any of the methods suggested by Angelo and Cross “Classroom assessment techniques”

9-5

Jossey Bass.This allows us to make mid-course adjustments and to show students that we really believe thatteaching and learning is a combined effort. *****************************************Action: Which approach will you take?______________________________________________________________________________________________________________________________________________________________________________________

2. Evaluation of the overall effectiveness of a course

Many teachers do not take time to evaluate the overall effectiveness of their courses.Unfortunately, when it comes to teaching many professor “diddle around”. They try one thing inthe classroom; then they try another. Their approach evolves like Topsy with them never sure asto what works and what doesn't. They would never do research in our subject discipline that way.In subject-discipline "research" they create a hypothesis, create models or experiments and testthe hypothesis and draw conclusions. With just a little more effort, we can bring that samescholarship to teaching. The hypothesis is that “by making a change in how you teach yourstudents will learn more effectively and/or they will develop process skills of value to them asprofessionals.” Your intervention should make a measurable change. Reflect on what went on.Keep your own learning journal. Gather evidence. Write journal articles and seek grants. Beingscholarly in your approach means that right from the start you plan how to test and monitor theeffectiveness of what you do.

How might you gather evidence?

! We can use data about student performance as a measure (for any course):- % who passed the course;- quality of the exam (degree of difficulty, number of questions requiring higher levelBloom performance, fairness [or relationship between the published course objectivesand the exam questions; the time it takes a professor to write the exam compared with thetime allowed for the student {should be in the range 1:2 to 1:4}].- student mark improvement from entry to exit.- number of scholarships or awards your students receive.- feedback about student performance from instructors in subsequent courses.

! We can gather data about the development of deep learning instead of surface learning (forany course):- use Course Perceptions Questionnaire, CPQ (about the quality of the learning environment).- use pre and post Lancaster Approaches to Studying Questionnaire, LASQ.

! We can use the Peer Evaluation of Educational Programs inventory whereby peer facultyevaluate the planning that went into the course (for any course).

9-6

! We can measure changes in “processing skills” such as problem solving, team work, selfassessment, lifelong learning, change management, communication (especially for PBL courses).Many well-developed instruments can be used for Pre and Post tests.

Consider using some simple and easy-to-administer pretests and post-tests. Examples for processskills have been cited (Woods, 1999e; Woods et al., 2000b, Woods et al., 1997). Downloadsome from http://www.chemeng.mcmaster.ca/innov1.htm and click on PBL and download E.

Perry Inventory, Heppner’s inventory, Billings Moos Basadur creativity

! Gather data from exit surveys. Queen’s University has developed an excellent survey(Queen’s, 1994) (for any program).

********************Action:Which approaches are you using or will you take?_______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

3. ExampleExample: for McMaster’s chemical engineering program: MPS and PBL program seehttp://www.chemeng.mcmaster.ca/innov1.htm & click on MPS and PBLThe program: The program consists of four required courses: 2nd year: 48 tutorial hrs. required PS homewk.3rd year: about 20 hrs. required for application PS3rd year: 45 tutorial hrs. required teams; PBL4th year: 30 h. required PS, team, PBL

Our published outcomes for the program: problem solving, team, self assessment, lifelonglearning.Constraint: can’t continually add new courses & still have 4 year programgive more fish -> teach to fish

MPS (four courses) 20% of total curriculumPBL component: “safety” engineering economics” “process operability” 2% of totalcurriculum.Develop process skills before PBL. 6 - 10 groups & 1 teacher: tutorless groupsProgram evaluation:

9-7

M Marks improve in other courses compared with controlM Course Perceptions Questionnaire: 28 to 35 compared with 15 to 20 with controlM Confidence in PS skills: 90 -> 70

control 90 -> 85M Skill in PS: Billings-MoosM Attitude toward lifelong learning: Perry; Deep learningM Self assessment skillM Alumni & recruiters surveys and responses.

Enrichment:1. Please help me understand how best to help you2. MRIQ3. Inventories and scoring CPQLASQHeppnerBillings Moos

4. Exit Questionnaire

References1. Woods, D.R., A.N. Hrymak and H.M. Wright, “Approaches to Learning And lerningEnvironments in PBL versus lecture-based learning,” Conference proceedings, ASEE Meeting,St. Louis, MO June 2000.

2. “MPS program” downloadable from http://www.chemeng.mcmaster.ca/innov1.htm and MPS

9-8

9-9

9-10

10-1

10. Myths and misconceptions about PBL

Many myths and misconceptions have developed about PBL. Probably the most prevalent one Ihear is “How can students solve a problem if I haven’t given them a lecture on the backgroundknowledge?” Other issues include I’m doing PBL already, there is only authentic version forPBL, isn’t there a dilution of content? PBL can only be used for certain disciplines, my studentscannot do PBL, all students will love it, it’s easy for teachers to use it, and by using PBL you’regiving students licence to study whatever they want. What’s happening to standards? Considereach in turn.

10.1. Myth? Prior subject knowledge is needed before students can solve PBL problems

Problems can be used for several purposes. Students - and teachers - are used to assigningproblems from the end of the chapter or after a teacher has lectured. Such problems are used inthe context of.. this is the knowledge you need to know, I have given you the basic ideas! Use problems after they know stuff; synthesize knowledge.

! Use problems to drive the search for knowledge (student ownership; solving in the context ofprofessionals; learned in the context of future need)

Myth? is this possible? Biggest leap!Yes... you have done this many time.. been confronted with a problem, had to figure out theissues, called people, learned stuff and solved it. The hardest hit when you let students do this isyour ego.. you are no longer the source of all information.

Yes, all MDs from Mac since 1972; ChE from Mac since 1982Mac: Nurses, OT/PTMD, U New Mexico, Newcastle U, Maastrict U.Delaware: Samford, Temasek

10.2 Myth? I doing PBL already

Many think because they are using problems, then they must be doing problem based learning.“Don’t students really learn the stuff I lectured on when they are doing the homework problemsat the end of the chapter?” Yes. That’s probably true that students learn when they are doing thehomework; that they didn’t pick up much knowledge from lectures. However, assigningproblems is not PBL.

10.3 Myth? there is only one form of PBL

Luis Branda, a colleague from McMaster, used PBL effectively in his class of 120 with oneinstructor. His unique version of individual, self-directed, teacher assessed, independent PBL. Inthis context students given a problem and identified what they needed to know to solve theproblem. Each individual created his/ her learning objectives, researched to locate and learn the

10-2

new knowledge. They summarized that knowledge for himself/herself. Branda provided eachperson with examination questions on each person’s learning objectives. He marked it. Is thisPBL. Certainly!Jim Harvey, a fictional colleague, gave a group of students a design problem to do in their senioryear. The project required them to draw on all the courses they had taken, to identify any newknowledge they needed and then to solve the design problem. Is this PBL? Partly. This approachlacks structure. Students are not formally asked to (or marked on) identify the new learning thatis required. The students don’t usually share or teach each other the new information. Usuallypeople in the group are responsible for different tasks and there is no expectation that everyone isresponsible for learning the new knowledge. The amount of new knowledge varies with theproject and the previous courses that were taken. For projects with reactors, heat exchangers anddistillation columns (subjects that most chemical engineering students know already) then thereis relatively no new knowledge needed. Is this PBL? Definitely not; usually this is problemsynthesis. Here we consider the key characteristics of PBL and describe some options where PBL has beenused.

10.3-1 PBL: What are the key characteristics of small group, self-directed, self-assessed,interdependent PBL?

! an extremely effective way to improve student learning. Other methods OK too, but PBL usesthe most of the proven ways.

! method that develops “Deep learning” not “Rote learning”. Straight lecture promotes “Rotelearning” whereas the use of small group, self-directed, self-assessed PBL develops deeplearning.

! develops skills needed in your student’s career especially Lifelong learning. Others too:problem solving, group work, critical thinking. but these can also be developed by methods otherthan PBL.

1. Presentation of knowledge on a “need to know” basis instead of “trust me, you’ll need thisstuff sometime.”2. Pose the problem first.3. Empower students with most, if not all, of the elements of the learning process: scrutinize asituation, identify new knowledge needed to address the situation, create learning objectives,identify key resources, critique the resources, learn and teach each other, assess the integrated setof knowledge the group learned, apply it to solve the problem, assess the knowledge and skillsdeveloped, elaborate the knowledge to put it into context and see where else you can use theknowledge. 4. No “lecturing” by students or by teacher unless requested with clear learning objectives givenby the students and requested by the students.

Any form that includes all four of these characteristics is what I would call authentic PBL. Some

10-3

versions that maintain four characteristics include:- Socratic version where small groups are present in the room; the teacher tutor facilitatesoccasionally. This is described in “PBL: helping your students gain the most from PBL”;downloadable from http://www.chemeng.mcmaster.ca/innov1.htm and click on PBL. exampleMD program at U of Adelaide, Australia. - Using Branda’s approach with individual, self-directed PBL; described in “PBL: Resources togain the most from PBL”. example, Luis Branda’s course in biology.- Using extensive pretraining and autonomous student groups; example, Chem. Eng atMcMaster.- Using a tutor with each group; example MD, Nursing, Physio and occupational therapy atMcMaster.

10.1-2 Myth? There is only one authentic form of PBL

The popularity of PBL has spread so rapidly and extensively that a) many don’t really know thefundamental underpinnings of PBL, b) they think that all you do is pose a problem and turnstudents loose. This sloppy implementation of “something” in the name of PBL has provided abacklash from those who are indeed trying to use small group, self-directed, self assessed,interdependent PBL who suggest that there is one form of “authentic” PBL. Incorrect, there is alot of flexibility in implementing a range of options that satisfy the fundamentals. Here are some:a) use Individual, self directed PBL ( the assessment component and small group (cooperativeelement) are missing), but this is a very exciting and, in my view, valid way to use PBL.b) use small group (size 5 to 8 max), self-directed, PBL (assessment component andinterdependent elements are missing). Here the teaching element is minimized, students reporttheir research, more than one student may be “studying the same topic” or indeed all might bestudying the same topics and the “teach meeting” is a critique of the information.c) small group, self-directed, self-assessed interdependent PBL with a tutor in each group.d) small group, self-directed, self-assessed interdependent PBL without a tutor in each group.There might be a tutor-facilitator for two to 50 groups of five students; alternatively, the studentsmay function as empowered autonomous groups.

10.3 Myth? Dilution of Content

Dilution... Reduction of subject knowledge? Yes. In a fixed three or four year curriculum thereare only so many hours available per course. In the conventional courses this is 120 h, for classactivities, self study, homework, studying for tests/exams and doing the exams. PBL is a learningenvironment that draws on students problem solving, group, self assessment, goal settingabilities. It provides an opportunity to develop these skills further. Recall that effective teamsspend 25% of their meeting time on “process activities” and 75% of the time actually solving theproblem that they were given to solve (Reddy, W.B. 1994, “Intervention skills: processconsultation for small groups and teams, Pfeiffer & Company). So it is not surprising that thesubject knowledge that can be mastered is about 75% of the knowledge that is “covered” in aconventional lecture environment. You note, I do not use the term mastered in the conventionallecture environment.

10-4

This really is not a problem because:1. in preparing to use PBL we know that the amount of topics “covered” needs to be reduced.We identify the fundamentals; ensure these are embraced in the PBL experience. The other frilltopics students can learn on their own when they need to know it. Since we introduced PBL, noalumni, no company has ever said that graduates of our program “didn’t know thefundamentals”. Instead they have said, “your graduates are ready to work as soon as I employthem; graduates from other schools require about 2 years of training.” 2. graduates with have the fundamentals plus skill in lifelong learning, and usually skill inproblem solving, listening, communication, locating and assessing information, self awarenessand team skills.

To help you identify the fundamentals in your discipline you might ask the question: In the year2020 do the students need to know “everything” you lecture about now?Focus on the core fundamentals of your discipline = 50 to 70% of what you currently “cover”in class.

How important to you are the task of developing:problem solving,team work,change management,communication,lifelong learning.

10.4 Myth? My Students cannot do PBLIf they think so & you think so.. then you are right.

Attitude shifts for both students & you

Prepare them

10.5. Myth? PBL can only be used for certain disciplines

So far PBL has been used successfully in MedicineNursingPharmacyOT/ PTDentistry QueenslandBusiness Temasek, Samford, George Brown, SenecaEngineering Monash, Manchester, McMaster, Oxford BrookesScience Delaware, LaTropeLaw Calgary, WollongongArchitecture Newcastle, AdelaideTeacher training

10-5

Vet Medicine GuelphIT Linkoping, SenecaPolice NSW, Queensland

10.7. Myth? It’s easy for teachers to use it

It is not easy. It requires a major attitude shift: you are no longer the sage on the stage, you arethe guide on the side. A shift that students can actually learn and teach each other instead of Ihave to tell them my knowledge. A shift that they can self assess.

Next, implementing PBL requires a lot of work. You need to create the problems and pretestthem to ensure that the cues are well-designed. You need to gather the resources they will needso that a new subject can be located and learned in about 2 h per student per week.

10.8. Myth? All students will love it.

Not all students will love it. Some will struggle.

10.9 Myth? you’re giving students licence to study whatever they want. What’s happeningto standards?

The students are not given licence to study whatever they want. The problems are crafted so thatthe students will create your learning objectives for your course. The difference is that thestudents feel ownership for the objectives and for the learning.

A-1

Self assessment journals

Marking/feedback on self assessment journals for process skills

Chairperson skillsLifelong learning skillsTrouble shooting/clinical skills

Examples:PBL Deirdre SchroderPBL Fahim AhmedChairperson Kathryn Grundy

A-2

Feedback on Chairperson self assessment journal

name__________________________

Group number: _____________________ /30

Analysis:

Numerical summary of Task and Morale scores:chair’s role: + x 2; neutral x 1; negative x -1 and total: 5/30

Listing of strengths & areas to work on: 5/30

Arithmetic average of "satisfaction with decision" 3/30

Anecdotal summary of events and comments (and about growth in subsequent meetings) 6/30

Quality of the agenda: 4/30

Relate evidence to goals and achievement of goals: 5/30

Overall communication: 2/30

DRW, Sept 19, 2005

A-3

/30 Feedback on First PBL self assessment journal on lifelong learning skillsName ______________

attitude and risk: 5understanding of interdependent learning, Perry attitude and shifts for you and for group thathave occurred or that you might have to address; [form p 105, p. 106]include checklist of attitudes [form 3600, p 109] 2

overall plan and strategy: 10have one?skill in identifying learning issues: [form 106]include learning contract or equivalent if you generated oneQuality of the teach notes:Quality of "test question" and answer :

analysis: 10 relate to learning objectives of MPS 36 [p. 103]summary table of numerical values of communication, knowledge & feedback for this meeting:

(form p. 119 to 142 using scale of 8)table of comments about strengthstable of comments about weaknessesinclude all feedback sheets from all members including self:comment on group and learning style, attitude: [form p 105]help us understand what really happened in your meeting:is PBL working for you and for your group?

communication: 5

If the memo only is marked, the marking will be based on how well all of the issues weresummarized in the memoA120

A-4

/40 Feedback on PBLName ______________

attitude and risk: 5understanding of interdependent learning, Perry attitude and shifts for you and for group thathave occurred or that you might have to address; [form p 105, p. 106]include checklist of attitudes [form 3600, p 109] 2

overall plan and strategy: 10have one? 2address learning issues: 1

list of topics generated?explore issues? resource? learning objectives?

feedback checklist included: 3include learning contract or equivalent: 2include teach notes: 1include "test questions": 1

analysis: 10 relate to learning objectives of MPS 36 [p. 103]summary table of numerical values of communication, knowledge & feedback for this meeting:

(form p. 119 to 142 using scale of 8)table of comments about strengthstable of comments about weaknessesinclude all feedback sheets from all members including self:comment on group and learning style, attitude: [form p 105]help us understand what really happened in your meeting:is PBL working for you and for your group?

improvement: 10summary table of numerical values of communication, knowledge & feedback for all meetings:maintain strengths:

table of commentsnote and discuss

shift one weakness:table of commentsnote

communication: 5

Forms: p 105, 106, 109, five or six from 119 to 142

If the memo only is marked, the marking will be based on how well all of the issues weresummarized in the memo

A-5

Feedback on Trouble Shooting self assessment journalname ___________________

/150process: 40 evidence of:

organized approach: (objective 3.1: evidence observer form/reflections)monitoring: (objective 3.2: evidence observer form/ reflections)hypotheses: (objective 4.1, 5.1; evidence observer forms, notes)data analysis: (objective 4.2, 4.3; evidence observer forms, notes) select an appropriate approach: (objective 3.4, evidence observer forms, notes)clarity of questions written out (objectives: 3.3; 5.2; evidence observer form;green sheets)

improvement: 25

maintain strengths: 9/25note and comments

shift one weakness: 12/25note and comments

Overall 4/25

analysis of the triad: 30what did you learn from playing each role?how did this vary from case and triad to triad:what did you learn about trouble shooting from interacting with others

communication: 10

relate evidence to learning objectives: 35

skill in self assessment 20

evidence to back up claims or just wishful thinking?

Forms: Learning objectives, 145Written evidence sheets from role as TS, 171 ffFeedback from observer, p 149Reflections: Papers used to make notes during the activity

220,5.5

1

Index to “Preparing for PBL” Donald R. Woods, © copyright 2003, 2006

Accountable, making students, 3-5; need for studentsto be, 7-10. Active learning, need for to improve learning, 1-6, 4-5;30 ideas on how to:1-7; and PBL, 1-1; Dale’s cone oflearning, 1-6; comparison of lecture vs PBL, 4-5;

Advantages of PBL: 8-50, 1-2

Agenda, example, A-35

Ahmed, Fahim, A-19 to -32

Albert Haber Anxiety achievement, as diagnosticinstrument, 7-7;

All courses shift to PBL see PBL, full.

Alverno college, cited, 2-5;

Assertiveness, skill in, need for in PBL, 4-18; how todevelop, 4-18;

Assessment: options for student performance, 1-11;Principles of, 2-6, 4-23, 7-11, 8-4, 8-50; of knowledgelearned, list of options: 1-11, 8-56; and forms ofevidence for problem solving, 2-7; checklist reflectionabout, 2-20; and accountability, 3-6; ownership bystudents as factor to improve learning, 4-5; lecture vsPBL, 4-6; importance of good assessment in PBL, 7-11;Gibbs “ownership” and learning, 8-5; who does theassessment, 8-8; and list of forms of evidence, 8-8;assessing problem solving skills, two papers, 8-85;assessing team skills, two papers, 8-26ff; assessinglifelong learning skills, paper, 8-65 ff

Asthana, Deepti, 7-26

Attention span, 1-6; and lecture vs PBL, 4-6;

Autonomous groups, see Tutorless groups.

Basadur inventory for creativity, as diagnosticinstrument, 7-7; use for program evaluation, 9-6.

Billing Moos inventory, use of 2-5; as diagnosticinstrument, 7-7; use for program evaluation, 9-6.

Bloom’s taxonomy, important for improved learning, 1-6; as guide for assessment, 8-5; activity to identify levelfor assessment, 8-7;

Branda’s PBEE, option for individual PBL, 3-5.

Bunn, Jeff, 7-26 Careful preparation needed, see Preparation needed, 1-10;

Cases see Problems.

Chairperson skills, feedback form, 3-11; need for inPBL, 4-18; how to develop, 4-18.

Challenges to different options of PBL, 3-4; 7-9;

Change management, skill needed for PBL, 6-4; andrelated skills, 6-4; feedback form, 3-11; learningenvironment for development, 4-3; use of, to preparestudents for success, 7-6; how to develop, 6-4. see alsoProcess skills.

Characteristics of PBL, list of 1-1, 3-4. Chickering and Gamson, citation, 4-24; cited 7-9, 4-5;

Claims of PBL, list of, 1-2; and CPQ, 4-7; and containelements to improve learning, 4-5, -6, -7;

Class time, scheduling, 4-14, 4-15; length of 75 min, 5-1;

Communication skill, feedback form about, 4-2;learning environment for development, 4-3;

Confidence, developing student’s: importance and howto, 2-8; reflective checklist about ideas, 2-20; need inPBL, 4-18; how to develop, 4-18;

Conflict resolution, skill in need for in PBL, 4-18; howto develop, 4-18; Cooperative learning, as part of PBL, 1-1; 17 ideas onhow to: 1-7; need for to improve learning, 4-5; lecturevs PBL, 4-5;

Coverage of material, 1-1; less “coverage” in PBL, 1-2; covering material is not learning, 7-4;

CPQ, course perceptions questionnaire, use 1-2; resultsfor “lecture”, 1-1; results for PBL, 1-2, 4-7; use forprogram evaluation, 9-5

2

Creating problems see Problems.

Creativity, skill in, need for in PBL, 4-18; developmentof, 4-18; use Basadur to measure development, 9-6.

Critical thinking, skill in, need for in PBL, 4-18;development of, 4-18.

Cues, the importance of in problem statement, example2-2;

Deciding on a version of PBL that suits you, see Selecta form for you.

Deep learning, and PBL, 1-2; in lectures, 1-6; list offactors affecting, 4-5; LASQ as measure, as diagnosticinstrument, 7-7;

Definition of PBL: 1-1.

Effectiveness of PBL, see Evaluation

Elaboration, definition, 5-4; importance of 2-12;reflective checklist about, 2-21; need for skill in PBL,4-18; how to develop, 4-18; need for, 5-4.

Empowering students, 14 ideas on how to: 1-9.Environment to expect success, see Expect studentsuccess.

Evaluation of effectiveness of PBL, need for, 1-6;reflective checklist about, 2-21; include, 5-4;fundamental principles of, 9-4; example data forMcMaster’s chemical engineering, 9-6 to 9-7; exampleclaims/data, 1-2, 8-50. Expect student success, important issue cited, 1-9;factor to improve learning, 1-9, 4-5; lecture vs PBL, 4-6

Facilities, classroom for PBL, 5-1;

Faculty ready? see Teachers ready?

Feedback, suggestions on how to give, 8-5; importanceof prompt feedback to improve learning, 7-9, see alsoPrompt feedback.

FIRO-B: as diagnostic instrument, 7-7; cited 8-7;

Forms of evidence, for assessment, selecting, 2-7; listof options for problem solving, 2-7; identifying, 3-6;how to design, 8-7, 8-54; list of options, 8-8, 8-56.

Forms: quality of assessment, 2-18, 3-13; for goalsmeeting, 3-3, 6-3, 7-3, 8-3, 9-3; group process, 1-3, 3-11, 3-17, 6-2, 6-38, 7-2, 8-2 3-2, 9-2; general processskills, 3-11; for problem solving, 3-11, 3-14; for stressmanagement, 3-15; need for consistency among, 5-1;for individual behavior in groups, 6-33 in Chpt 6; howto design, 8-7; for chairperson, 3-24; A- 36, -37, -38;for teaching in PBL group, for creating learning goals,1-3, for change management, 3-11; for interpersonalskills, 3-1; for listening in TAPPS, 6-11for team skills,; individual contribution to group, 3-18,3-19, 6-32; for chairperson skills, 3-11, 7-24; forlifelong learning skill, 3-12; teach skills, 1-12, 2-18, 3-21, 7-23, A-10 to -15, A-28 to -32; ; objectivesidentification, 1-13, 3-22, 6-3, 7-3, 8-3, A-9; learningstyles, 1-14, 3-23, 7-8, 7-25; learning contract, 7-26; forself assessment, 3-12; for knowledge structure, 3-12;for experience knowledge, 3-12 related topic Forms ofevidence.

Goals, learning see Learning objectives, related termLearning issues.

Goals meeting, importance of, 1-10; feedback tostudents about creation of objectives, 3-3; scheduling,4-14;

Goals (related term Learning objectives), need for, forassessment, 2-7; need for to improve learning, 1-6, 4-5,4-6; creation of, 3-6; learning objectives for subjectknowledge and problem creation, 4-17; for lecture vsPBL, 4-6; number (30) for a course, 8-5; and criteria, 8-6; examples, team skills, 8-6 to -7;

Goal setting, skill in, needed in PBL, 4-18; workshopsto develop skill, 4-18;

Grieving process, as model of the transition, 7-5; asmodel of change, 7-5

Groups, size of 5 to 6, 5-3; assigning students to, 5-3;length of time students are in the same group, 5-3

Group skills, feedback form, 1-3, 3-11, 3-17, 3-2; Listof published goals, 2-16; individual contribution togroup, 3-18, - 19; learning environment fordevelopment, 4-3; Workshop, MPS 28, 6-26

Grundy, Kathryn, A-33 to -39

Guided design, as option for PBL, 3-5;

3

Heppner PSI, use of 2-5; as diagnostic instrument, 7-7;and evaluation of program, 9-6, 9-7.

Holmes-Rhae stress instrument, as diagnosticinstrument, 7-7.

How to shift from lectures to PBL, see Journey

Implementing PBL: 9 issues, 1-9; details, 5-1 ff. Improving learning, see Learning, improving.

Interpersonal skills, and RIGHTS, 2-8, 6-26 in Chpt.6; four killers of, 2-9, 6-26 in Chpt. 6; feedback form,3-11; needed for PBL, 4-18.

Johnson’s conflict resolution, as diagnosticinstrument, 7-7;

Journal, see Reflective journal

Journey toward PBL, suggested stages, 1-6; PBLoptions, 3-4, 3-5;

Jungian typology, MBTI: as diagnostic instrument, 7-7; table for students, 7-8.

Just-in-time”, and PBL, 1-1; “Trust me, you’ll needthis sometime” vs just-in-time, 1-1;

KAI (Kirton adaptor-innovator), as diagnosticinstrument, 7-7; table for students, 7-8;

Kellner-Sheffield instrument, as diagnostic instrument,7-7;

Knowledge structure, developing lecture vs PBL,compare, 4-4;

Knowledge, subject: need to reduce amount “covered”,5-1; planning “systems” of knowledge, 4-12; assessing,8-5; options to assess, 8-53

Kolb learning cycle, as diagnostic instrument, 7-7. LASQ (Lancaster approaches to studying), asdiagnostic instrument, 7-7; table for students, 7-8; usefor program evaluation, 9-5.

Learning environment, in PBL, 1-2; measuring, 1-2;and deep learning ; and CPQ, 1-2;

Learning, improving: list of 14 ideas, 1-6; list of 20

factors, 4-5; and PBL, 1-2; and Dale’s cone of learning,1-6;

Learning goals, see Learning objectives, related termLearning issues.

Learning objectives are written, unambiguousstatements of goals complete with measurable criteria:students generation of, 2-2; correct generation of by thestudents? 2-2; for problem, 4-17; examples, team skills,8-6, strategies for problem solving, MPS 4, problemssolving strategies, 6-7, -8; MPS 28, group skills, 6-27, -28; MPS 29, chairperson skills, 7-20; MPS 36, lifelonglearning, 7-18, A-23

Learning issues are topics that we need to know tosolve a problem; learning issues are then converted into“Learning objectives”, student creation of, 2-2;feedback about quality of student-generated, 3-22, 3-3;difficulty students have in identifying more than 60%,5-2; training in Goal setting workshop, see Goal setting.

Learning style see Personal style.

Lectures, weakness of, 1-6; and Dale’s cone oflearning, 1-6; transition from lectures to PBL, seeJourney; compared with PBL see Lecture vs PBL.

Lecturing to PBL, how to shift: 1-9; see also Journey

Lecture vs PBL, and factors to improve learning, 4-5, -6, -7; teacher’s role in, 7-4;

Lee, Craig, 7-26

Lifelong learning skills, definition, 7-10; claim forPBL, 1-2; feedback form about, 3-12; target skills, 3-20; feedback about quality of teach in , 3-21; feedbackabout creation of learning objectives, 3-22; learningenvironment for development, 4-3; sharing of learningstyle information, 3-23, 7-8; importance as outcome, 8-5, 4-1;

Listening, feedback form in TAPS, 6-11

Matrix planning for location of activities, 5-1; forlocation of skill development, 4-4; for problemdevelopment, 4-13;

Meetings, select types, 4-14; scheduling, 4-17, 4-14;visualizing, 5-3; Goals meeting, 5-3; Teach meeting, 5-3; Feedback meeting, 5-3;

4

Misconceptions about PBL, 10-1 ff

Monitoring, of student response and the teaching-learning teamwork, 7-5; use of ombudspersons, 7-5;teacher’s role of monitoring student’s progress andapplication of process skills, 7-4, 7-10; guide on theside, 7-4; the transition, 7-5; use in PBL evaluation, 9-4; monitor skill development in curriculum, 5-1, 4-13.

MPS, McMaster Problem Solving program, list ofunits, 4-18;

MRIQ: form and scoring 1-4, 4-9 to -11; score data, 1-6; use of 7-5; interpretation, 1-6

Myths about PBL, 10-1 ff

Options for PBL: 3-4, 3-5; (related topic “Select aform for you)

Osterman feedback lecture, as option on journey toPBL , 3-5;

Outcomes of program, 3-7, 3-8; selecting, 4-1;publishing, 5-1; process skills as, 6-4, 4-1.

PBL, ideal version, list of characteristics, 1-9; welldesigned? issues, 1-10;

PBL full (every course is using PBL), 4-12; planningthe system, 4-12; checking the resources, 4-13;scheduling the sessions, 4-14.

PBL in part (only in your course is PBL used), 4-12

Perry, W.G, citation, 4-25.

PEEP, peer evaluation of educational programs, usefor program evaluation, 9-5.

Perry inventory: check on student’s attitude, 1-11, 7-6;and assessment, 2-6; and students ready? 4-7; cited, 4-22, 4-23; use of to help transition, 7-5; as diagnosticinstrument, 7-7, 7-8;

Personal choice of form of PBL, see Select a form ofPBL for you.

Personal style, how to identify your, 2-8; example listof diagnostic instruments, 7-7; Learning style, sharingof information among group, form, 3-23; factor toimprove student learning, 4-5; lecture vs PBL, 4-6.

Plan for success using PBL, 1-11, 4-8; nine issuesrelated to your success, 7-4; reflective checklist about,2-21.

Preparation for PBL, need for careful preparation, 1-10; example approaches, 7-14; and evidence ofeffectiveness, 7-14ff

Problem solving, list of skills for, 6-4; assessment of,list of forms of evidence for, 2-7; feedback form, 3-14,3-11; learning environment for development, 4-3; skillneeded in PBL, 4-18; list of workshops how to develop,4-18; how to develop, 6-4. see also Process skills.

Problems, (cases) how to create, checklist 2-19, 4-16,4-17, 5-2; description of how to create, 5-2; criteria, 2-2, 1-10, 5-2; example critique of creation, 2-2ff;connected to learning objectives, 5-5 to 5-10; example,1-1, 1-3, 1-10, 2-1, 3-1, 5-4, 5-5 to 5-10, 6-1, 7-1, 8-1,8-5; 9-1; development in PBL, 1-2; do they reflectprofessional practice? 7-10; examples with learningobjectives from, 2-4 ff, 5-5 to 5-10.

Problems, how to create, 1-10; posed first, 1-1; criteriafor good problems, 1-10; checklist about process ofcreating, 2-19; use of matrix to keep track of learningobjectives, 4-13;

Process skills, list of, 1-11, 3-7, 6-4; developingprocess skills, 1-11, 4-1; embed in PBL experience, 2-5,3-7, 7-11; checklist of how to embed, 2-19; reflectivechecklist of process skill development, 2-19; how todevelop, 2-4 and 2-5, 3-7, 6-4; when to develop, 2-5, 3-7, 4-21; assessment of, 2-5, 8-61; and your PBLactivity, 2-6; Goals for group skills, 2-16; exampleworkshops, list and source, 3-9; learning environmentsfor different, table of, 4-3; chart where each skill isdeveloped, 4-4; process skills needed, list, 4-18;schedule of workshops to develop, 4-20; need foroutcomes, 6-4; need for PBL, 6-4; example learningobjectives for group skills, 6-27, -28 in Chpt 6, 8-6 and-7; example learning objectives for problem solving, 6-7, -8 in Chpt 6,; target skills: for group skills, 6-31 inChpt 6, for problem solving, 3-14; example workshops,for strategies, 6-6 ff in Chpt 6; for group skills, 6-26 ffin Chpt 6; list of activities in typical workshop, 6-4, 7-9;track skill development in curriculum, 5-1, 4-13.

Prompt feedback, 15 ideas on how to: 1-8; need for todevelop confidence, 2-8; need for to improve learning4-5; lecture vs PBL, 4-6;

Proof that PBL works, see Claims.

5

Quality of teacher-student interaction, factor toimprove student learning, listed, 4-5; lecture vs PBL, 4-6; building relationships, 2-9;

Queen’s Univ. exit survey, 9-8, 9-9

Ramsden, Paul, citation, 4-25; cited 7-9

Recall, improve, claim of PBL, 1-2;

Reflection, definition, 5-4; need for in PBL, 5-4;

Reflective journals, use of 2-8; use for studentaccountability, 7-10; 8-56, 8-35, 8-62, 8-65. ExamplesA-6 ff.

Relationships, building between students and teachers,2-9

Resources, student learning for PBL, creating, 4-16, 4-17

RIGHTS, list of and importance, 2-8; list, 6-26 in Chpt6.

Rogers, Karen, 7-26

Rotter locus of control, as diagnostic instrument, 7-7;

Scheduling the meetings, full PBL, 4-14; forworkshops to develop process skills before PBL, 4-20;for workshops to develop process skills concurrentlywith PBL, 4-22; Select version for you, see Select aform of PBL for you.

Schroder, Deirdre, 7-26, A-6 to -18

Select a form of PBL for you: 1-6; are you ready? 1-6;large class, 3-1ff; list of 26 steps in process fo deciding,4-1ff; need for, 1-9; seven steps in changing, 1-9;reflective checklist on how to select, 2-20; identifyingissues to improve learning and relate to your version, 7-9; challenges to selecting different options, 3-4, 7-9; listof options, 7-9, 3-5, 3-4; .

Self confidence, development of, 1-2; issues fromBandura, 5-4, 7-6; nine steps to, 5-4, 7-6;

Self assessment, use of for process skill assessment, 2-8; feedback from for, 3-12, 2-18; learning environmentfor development, 4-3; use as part of PBL, 7-8; use forstudent accountability, 7-10;

Self-directed, interdependent learning, see Lifelonglearning.Shifting from lectures to PBL, see Journey to PBL. Skills, see Process skill.

Socratic PBL, option, 3-5; 4-17;

Strategies for problem solving, skill in applicationneed in PBL, 4-18; how to develop, 4-18; exampleworkshop, 6-6; strategy “board” 6-17

Stress management, feedback form, 3-15; 3-11;

Success, expect student, see Expect student success.Success, see Plan for success.

Students ready? issue in success of PBL, 1-9, 1-11, 4-7, 7-3, 7-5; how to help them get ready, 7-5 to -9.

Student’s roles, 5-15 to -18

Succeed, helping students: see Plan for success.

TAPPS (talk aloud pairs problem solving), cited, 7-7;used in strategy workshop, 6-8

Target skills, need for, 8-5; list of for groups skills, 2-17, 3-16, 6-31; for lifelong learning, 3-20, 7-22, A-9, A-22; for problem solving, 3-14, 6-13; for stressmanagement, 3-15

Teach, see Teach, skill of students.

Teach meeting, learning styles in preparation for, 3-23;scheduling, 4-14, 4-17;

Teach, skill of students to, list of component skills, 6-4; need in PBL, 4-18; feedback about student’s skill, 3-21; how to develop, 6-4. see also Process skills.

Teacher (tutor), role as guide on side, 1-1; to monitorstudent’s activities, 1-1; key questions to consider, 4-8;training of, 4-19; being, 4-21; use of MRIQ to helpdecide if ready, 4-8; role in lecture vs PBL, 7-4;activities to help, 7-5; Teacher ready for PBL?important issue cited, 1-9; 1-11; use of MRIQ, 4-8;issue in success of PBL, 7-3; importance of monitoring,4-8; teacher’s roles, 5-15 to -18

Teacher ready for PBL? see Teacher.Team skills, see Group skills.

6

TEPS, tests and exams of process skills, 2-7; forproblem solving, 6-20 in Chpt. 6; for group skills, 6-41in Chpt 6; use 8-61

TETK; Test and exams of knowledge, 1-11; use, 8-57

Time-on-task, clear, need for to improve learning, 4-5;lecture vs PBL, 4-6;

Timing sheets, for workshops, need for, 6-4; examplefor strategy workshop, 6-22

Track skill development in curriculum see Processskills.

Transition to PBL, managing, 7-1 ff; list of nine issuesto address, 7-4. “Trust me, you’ll need this sometime” vs just-in-time,1-1;

Trust, how to build or destroy, 2-9;

Tutor see Teacher.

Tutorless groups and PBL, 3-5; challenges, 3-5, 4-23,making students accountable, 3-5; selecting, 4-18;

Workshops to develop process skills: schedule of, 4-20; list of 4-18, 3-9; description of, 2-4; example,problem solving strategies, 6-6 in Chapt 6; group skills,6-26 ff in chapt. 6; list of activities in, 7-9, 2-4, 6-4;

Your unique style, see Personal style.

Yue, Ben, 7-26