ABCOF

LEARNING ANDTEACHING IN

MEDICINE

ABCOF

LEARNING ANDTEACHING IN

MEDICINE

Edited by Peter Cantillon, Linda Hutchinson and Diana WoodEdited by Peter Cantillon, Linda Hutchinson and Diana Wood

ABC OF LEARNING AND TEACHING IN MEDICINE

ABC OF LEARNING AND TEACHING IN MEDICINE

Edited by

PETER CANTILLONSenior lecturer in medical informatics and medical education, National University of Ireland, Galway, Republic of Ireland

LINDA HUTCHINSONDirector of education and workforce development and consultant paediatrician,

University Hospital Lewisham

and

DIANA WOODDirector of medical education, University of Cambridge School of Clinical Medicine,

Addenbrookes Hospital, Cambridge

© 2003 BMJ Publishing Group

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording and/or otherwise, without the prior written permission of the publishers.

First published in 2003 byBMJ Publishing Group Ltd, BMA House, Tavistock Square,

London WC1H 9JR

www.bmjbooks.com

British Library Cataloguing in Publication DataA catalogue record for this book is available from the British Library

ISBN 07279 16785

Typeset by BMJ Electronic ProductionPrinted and bound in Spain by GraphyCems, Navarra

Cover Image shows a stethoscope for listening to sounds within the body.With permission from Colin Cuthbert/Science Photo Library

v

Contents

Contributors vi

Preface vii

1 Applying educational theory in practice 1David M Kaufman

2 Curriculum design 5David Prideaux

3 Problem based learning 8Diana Wood

4 Evaluation 12Jill Morrison

5 Teaching large groups 15Peter Cantillon

6 Teaching small groups 19David Jaques

7 One to one teaching and feedback 22Jill Gordon

8 Learning and teaching in the clinical environment 25John Spencer

9 Written assessment 29Lambert W T Schuwirth, Cees P M van der Vleuten

10 Skill based assessment 32Sydney Smee

11 Work based assessment 36John J Norcini

12 Educational environment 39Linda Hutchinson

13 Web based learning 42Judy McKimm, Carol Jollie, Peter Cantillon

14 Creating teaching materials 46Richard Farrow

Index 49

vi

Peter CantillonSenior lecturer in medical informatics and medical education,National University of Ireland, Galway, Republic of Ireland

Richard FarrowDirector of problem based learning at the Peninsula MedicalSchool at the Universities of Exeter and Plymouth

Jill GordonAssociate professor in the department of medical education atthe University of Sydney, Australia

Linda HutchinsonDirector of education and workforce development and consultantpaediatrician, University Hospital Lewisham, London

David JaquesIndependent consultant in learning and teaching in highereducation

Carol JollieProject officer in the skills enhancement project for theCamden Primary Care Trust at St Pancras Hospital, London

David M KaufmanDirector of the Learning and Instructional DevelopmentCentre at Simon Fraser University, Burnaby, British Columbia,Canada

Judy McKimmHead of curriculum development at Imperial College School of Medicine, London and an educational consultant

Jill Morrison Professor of general practice and deputy associate dean foreducation at Glasgow University

John J NorciniPresident and chief executive officer of the Foundation forAdvancement of International Medical Education andResearch, Philadelphia, Pennsylvania

David Prideaux Professor and head of the Office of Medical Education in theSchool of Medicine at Flinders University, Adelaide, Australia

Lambert W T SchuwirthAssistant professor in the department of educationaldevelopment and research at the University of Maastricht in theNetherlands

Sydney SmeeManager of the Medical Council of Canada’s qualifyingexamination part II, in Ottawa, Canada

John SpencerGeneral practitioner and professor of medical education in primary health care at the University of Newcastle upon Tyne

Cees P M van der VleutenProfessor and chair in the department of educationaldevelopment and research at the University of Maastricht in theNetherlands

Diana WoodDirector of medical education, University of Cambridge Schoolof Clinical Medicine, Addenbrookes Hospital, Cambridge

Contributors

vii

Preface

Although we would never allow a patient to be treated by an untrained doctor or nurse, we often tolerate professional training beingdelivered by untrained teachers. Traditionally students were expected to absorb most of their medical education by attendingtimetabled lectures and ward-rounds, moving rapidly from one subject to the next in a crowded curriculum. Our junior doctorslearnt by watching their seniors in between endless menial tasks. In recent years the importance of active, self directed learning inhigher education has been recognised. Outcome led structured programmes for trainees are being developed in the face of reducedworking hours for both the learners and teachers. These all present new challenges for teachers in medicine of all levels of seniority.

Throughout the world there is great interest in developing a set of qualifications for medical teachers, both at the elementary“teaching the teacher” level and as part of progressive modular programmes leading to formal certification. In addition to acquiringnew qualifications and standards, teachers also need access to literature resources that describe essential components in medicaleducation and supply tips and ideas for teaching.

This ABC began as an expressed wish of the BMJ to publish an introductory and accessible text on medical education. It grewinto a book covering the more generic topics of learning and teaching in medicine with the aim of illustrating how educationaltheory and research underpins the practicalities of teaching and learning. The editors invited an international group of authors onthe basis of their acknowledged expertise in the particular topics assigned to them. Each chapter was edited and illustrated to ensuremaximum accessibility for readers and subsequently peer reviewed by two educational experts. Their suggestions have beenincorporated into the finished book.

The ABC of Learning and Teaching in Medicine would not have been possible without the tireless support of BMJ editorial staff,Julia Thompson, Eleanor Lines, Sally Carter, and Naomi Wilkinson. We would also like to thank Professor Paul O’ Neill and Dr EdPeile for their excellent and timely peer reviews for each of the chapters. Finally we would very much welcome comments andsuggestions about this ABC from its most important reviewers, you the readers.

PC, DW, LH

1 Applying educational theory in practiceDavid M Kaufman

How many times have we as teachers been confronted withsituations in which we really were not sure what to do? We “flewby the seat of our pants,” usually doing with our learners whathad been done with us. It would be useful to be able to turn to aset of guiding principles based on evidence, or at least on longterm successful experience.

Fortunately, a body of theory exists that can inform practice.An unfortunate gap between academics and practitioners,however, has led to a perception of theory as belonging to an“ivory tower” and not relevant to practice. Yet the old adage that“there is nothing more practical than a good theory” still ringstrue today. This chapter describes several educational theoriesand guiding principles and then shows how these could beapplied to three case studies relating to the “real world.”

Adult learning theoryMalcolm Knowles introduced the term “andragogy” to NorthAmerica, defining it as “the art and science of helping adultslearn.” Andragogy is based on five assumptions—about howadults learn and their attitude towards and motivation forlearning.

Knowles later derived seven principles of andragogy. Mosttheorists agree that andragogy is not really a theory of adultlearning, but they regard Knowles’ principles as guidelines onhow to teach learners who tend to be at least somewhatindependent and self directed. His principles can besummarised as follows:x Establish an effective learning climate, where learners feelsafe and comfortable expressing themselvesx Involve learners in mutual planning of relevant methods andcurricular contentx Involve learners in diagnosing their own needs—this will helpto trigger internal motivationx Encourage learners to formulate their own learningobjectives—this gives them more control of their learningx Encourage learners to identify resources and devise strategiesfor using the resources to achieve their objectivesx Support learners in carrying out their learning plansx Involve learners in evaluating their own learning—this candevelop their skills of critical reflection.

Self directed learningSelf directed learning can be viewed as a method of organisingteaching and learning in which the learning tasks are largelywithin the learners’ control (as with the adult learningprinciples described above).

It can also be viewed as a goal towards which learners striveso that they become empowered to accept personalresponsibility for their own learning, personal autonomy, andindividual choice. Success in the first view would lead toattaining the second.

Philip Candy identified in the literature about 100 traitsassociated with self direction, which he synthesised as the abilityto be methodical and disciplined; logical and analytical;collaborative and interdependent; curious, open, creative, andmotivated; persistent and responsible; confident and competentat learning; and reflective and self aware.

Andragogy—five assumptions about adult learningx Adults are independent and self directingx They have accumulated a great deal of experience, which is a rich

resource for learningx They value learning that integrates with the demands of their

everyday lifex They are more interested in immediate, problem centred

approaches than in subject centred onesx They are more motivated to learn by internal drives than by

external ones

Self directed learningx Organising teaching and learning so that

learning is within the learners’ controlx A goal towards which learners strive so that they

become able to accept responsibility for theirown learning

Learners need to feel safe and comfortable expressing themselves

1

How do we develop these traits in our learners? Mostimportantly, learners must have the opportunity to developand practise skills that directly improve self directed learning.These skills include asking questions, critically appraising newinformation, identifying their own knowledge and skill gaps,and reflecting critically on their learning process andoutcomes.

Self efficacyAccording to Albert Bandura, people’s judgments of their ownability to deal with different situations is central to their actions.These actions include what they choose to do, how much effortthey invest in activities, how long they persist in the face ofadversity, and whether they approach the tasks anxiously orassuredly.

These judgments, called “self efficacy,” may or may not beaccurate, but they arise from four main information sources. Indecreasing order of their strength, these sources are:performance attainments, observations of other people, verbalpersuasion, and physiological state. Successes raise our selfefficacy, while failures lower it. Failures are particularly likely tolower our self efficacy if they occur early in the learning processand are not due to lack of effort or difficult situations.

Observing other people similar to us performingsuccessfully can strengthen our beliefs that we can performsimilar tasks, especially when the tasks are unfamiliar. Verbalpersuasion from a credible source also can help.

Finally, we (both teachers and learners) need to re-interpretour anxiety or nervousness in difficult situations as excitementor anticipation, rather than as an ominous sign of vulnerability.

ConstructivismConstructivism has important implications for teaching andlearning. Firstly, the teacher is viewed not as a transmitter ofknowledge but as a guide who facilitates learning. Secondly, aslearning is based on prior knowledge, teachers should providelearning experiences that expose inconsistencies betweenstudents’ current understandings and their new experiences.Thirdly, teachers should engage students in their learning in anactive way, using relevant problems and group interaction.Fourthly, if new knowledge is to be actively acquired, sufficienttime must be provided for in-depth examination of newexperiences.

Reflective practiceThe theory of reflective practice is attributed primarily toDonald Schön, whose work is based on the study of a range ofprofessions. He argues that formal theory acquired throughprofessional preparation is often not useful to the solution ofthe real life “messy, indeterminate” problems of practice.

Schön labels professionals’ automatic ways of practising asprofessional “zones of mastery”—that is, areas of competence.Unexpected events or surprises trigger two kinds of reflection.

The first, “reflection in action,” occurs immediately. It is theability to learn and develop continually by creatively applyingcurrent and past experiences and reasoning to unfamiliarevents while they are occurring. The second, “reflection onaction,” occurs later. It is a process of thinking back on whathappened in a past situation, what may have contributed to theunexpected event, whether the actions taken were appropriate,and how this situation may affect future practice.

Self efficacy—roles for the teacherx Modelling or demonstrationx Setting a clear goal or image of the desired outcomex Providing basic knowledge and skills needed as the foundation for

the taskx Providing guided practice with corrective feedbackx Giving students the opportunity to reflect on their learning

The primary idea of constructivism is that learners“construct” their own knowledge on the basis of what theyalready know. This theory posits that learning is active,rather than passive, with learners making judgmentsabout when and how to modify their knowledge

Learners should identify their own knowledge gaps and critically appraisenew information

Teachers and learners need to view any anxiety or nervousness in difficultsituations as excitement or anticipation

“Reflection in action”

ABC of Learning and Teaching in Medicine

2

Through the process of reflecting both “in practice” and “onpractice,” practitioners continually reshape their approachesand develop “wisdom” or “artistry” in their practice. Activitiessuch as debriefing with peers or learners, seeking feedback fromlearners on a regular basis, and keeping a journal can providevehicles for reflective practice.

Converting theory into practiceEach of the educational theories presented here can guide ourteaching practices. Some theories will be more helpful thanothers in particular contexts. However, several principles alsoemerge from these theories, and these can provide helpfulguidance for medical educators.

Three cases studiesThe boxes (right) describe three “real world” case studiesrepresenting situations encountered in medical educationsettings. The educational theories described above, and theprinciples which emerge from them, can guide us in solving theproblems posed in these three cases.

Case 1 solutionYou could present an interactive lecture on the autonomicnervous system. You could distribute a notetaking guide. Thiswould contain key points, space for written notes, and two keymultiple choice or “short answer” questions requiring higherlevel thinking (principle 1, see box above). You could stop twiceduring the lecture and ask the students to discuss their responseto each question with their neighbours (principles 1, 3, and 5). Ashow of hands would determine the class responses to thequestion (checking for understanding) and you could then givethe correct answer (principle 5). Finally, you could assign alearning issue for the students to research in their own time(principle 4).

Case 2 solutionYou could assign the students to small groups of four to six, andask each group to submit two case studies describing clinicalethics issues in their local hospitals (principles 1 and 2). Theethics theory and approach needed to analyse these cases couldbe prepared by experts and presented on a website in advanceof the sessions (principles 4, 5). The first of the six blocks of twohours could be used to discuss the material on the website andclarify any misunderstandings (principle 5). You could thenshow the students how to work though a case, with participationby the class (principle 7). The other five blocks could then beused for each small group to work through some of the casesprepared earlier, followed by a debriefing session with thewhole class (principles 5 and 6).

Case 3 solutionYou could first invite the registrar to observe you with patients,and do a quick debrief at the end of the day (principles 2, 6, and7). With help from you, she could then develop her ownlearning goals, based on the certification requirements andperceived areas of weakness (principles 1, 3, and 4). These goalswould provide the framework for assessing the registrar’sperformance with patients (principles 5, 6). You could observeand provide feedback (principle 5). Finally, the registrar couldbegin to see patients alone and keep a journal (written orelectronic) in which she records the results of “reflection onpractice” (principle 6). She could also record in her journal thepersonal learning issues arising from her patients, couldconduct self directed learning on these, and could document

Seven principles to guide teaching practice1 The learner should be an active contributor to the educational

process2 Learning should closely relate to understanding and solving real

life problems3 Learners’ current knowledge and experience are critical in new

learning situations and need to be taken into account4 Learners should be given the opportunity and support to use self

direction in their learning5 Learners should be given opportunities and support for practice,

accompanied by self assessment and constructive feedback fromteachers and peers

6 Learners should be given opportunities to reflect on their practice;this involves analysing and assessing their own performance anddeveloping new perspectives and options

7 Use of role models by medical educators has a major impact onlearners. As people often teach the way they were taught, medicaleducators should model these educational principles with theirstudents and junior doctors. This will help the next generation ofteachers and learners to become more effective and should lead tobetter care for patients

Case 1: Teaching basic science

You have been asked to give a lecture on theautonomic nervous system to a first year medicalclass of 120 students. This has traditionally beena difficult subject for the class, particularly as ithas not been explicitly covered by faculty in theproblem based anatomy course. You wonder howyou can make this topic understandable to theclass in a 50-minute lecture.

Case 2: Ethics education

You are a member of a course committee in thedepartment of internal medicine, which ischarged with the task of integrating the topic ofethics into the third year medicine rotation. Yourcommittee has been given six blocks of twohours over a 12 week period. You wonder how tomake the material engaging, understandable, anduseful to the students.

Case 3: General practice training

You are the trainer for a first year registrar in herfirst year of a general practice trainingprogramme. Your practice is so busy that youhave very little time to spend with her. Youwonder how you can contribute to providing avaluable learning experience for your trainee.

Applying educational theory in practice

3

her findings in the journal (principles 1, 4, and 6). You couldprovide feedback on the journal (principle 5). If practical, thecohort of registrars could communicate via the internet todiscuss their insights and experiences (principle 6).

ConclusionsThis article has attempted to show how the gap betweeneducational theory and practice can be bridged. By usingteaching and learning methods based on educational theoriesand derived principles, medical educators will become moreeffective teachers. This will enhance the development ofknowledge, skills, and positive attitudes in their learners, andimprove the next generation of teachers. Ultimately, this shouldresult in better trained doctors who provide an even higherlevel of patient care and improved patient outcomes.

Further readingx Bandura A. Social foundations of thought and action: a social cognitive

theory. Englewood Cliffs, NJ: Prentice-Hall, 1986.x Candy PC. Self-direction for lifelong learning: a comprehensive guide to

theory and practice. San Francisco: Jossey-Bass, 1991.x Kaufman DM, Mann KV, Jennett P. Teaching and learning in medical

education: how theory can inform practice. London: Association for theStudy of Medical Education, 2000. (Monograph.)

x Knowles MS and Associates. Andragogy in action: applying modernprinciples of adult learning. San Francisco: Jossey-Bass, 1984.

x Schön DA. Educating the reflective practitioner: toward a new design forteaching and learning in the professions. San Francisco: Jossey-Bass,1987.

Curriculummaterials

Teachingmethods

Assessmentmethods

Clinicalsettings

Teacher Learner Outcome

Learningexperiences Skills

Bestpractices

with patients

Improvedpatient

outcome

Knowledge

Attitudes

From theory to practice

ABC of Learning and Teaching in Medicine

4

2 Curriculum designDavid Prideaux

The curriculum represents the expression of educational ideasin practice. The word curriculum has its roots in the Latin wordfor track or race course. From there it came to mean course ofstudy or syllabus. Today the definition is much wider andincludes all the planned learning experiences of a school oreducational institution.

The curriculum must be in a form that can becommunicated to those associated with the learning institution,should be open to critique, and should be able to be readilytransformed into practice. The curriculum exists at three levels:what is planned for the students, what is delivered to thestudents, and what the students experience.

A curriculum is the result of human agency. It isunderpinned by a set of values and beliefs about what studentsshould know and how they come to know it. The curriculum ofany institution is often contested and problematic. Some peoplemay support a set of underlying values that are no longerrelevant. This is the so called sabretoothed curriculum, which isbased on the fable of the cave dwellers who continued to teachabout hunting the sabretoothed tiger long after it becameextinct. In contemporary medical education it is argued that thecurriculum should achieve a “symbiosis” with the health servicesand communities in which the students will serve. The valuesthat underlie the curriculum should enhance health serviceprovision. The curriculum must be responsive to changingvalues and expectations in education if it is to remain useful.

Elements of a curriculumIf curriculum is defined more broadly than syllabus or course ofstudy then it needs to contain more than mere statements ofcontent to be studied. A curriculum has at least four importantelements: content; teaching and learning strategies; assessmentprocesses; and evaluation processes.

The process of defining and organising these elements intoa logical pattern is known as curriculum design. Curriculumwriters have tried to place some order or rationality on theprocess of designing a curriculum by advocating models.

There are two main types: prescriptive models, whichindicate what curriculum designers should do; and descriptivemodels, which purport to describe what curriculum designersactually do. A consideration of these models assists inunderstanding two additional key elements in curriculumdesign: statements of intent and context.

Prescriptive modelsPrescriptive models are concerned with the ends rather thanthe means of a curriculum. One of the more well knownexamples is the “objectives model,” which arose from the initialwork of Ralph Tyler in 1949. According to this model, fourimportant questions are used in curriculum design.

The first question, about the “purposes” to be obtained, isthe most important one. The statements of purpose havebecome known as “objectives,” which should be written in termsof changed behaviour among learners that can be easilymeasured. This was interpreted very narrowly by some peopleand led to the specification of verbs that are acceptable andthose that are unacceptable when writing the so called

Curriculum modelsPrescriptive modelsx What curriculum designers should dox How to create a curriculum

Descriptive modelsx What curriculum designers actually dox What a curriculum covers

Objectives model—four important questions*x What educational purposes should the institution seek to attain?x What educational experiences are likely to attain the purposes?x How can these educational experiences be organised effectively?x How can we determine whether these purposes are being attained?

*Based on Tyler R. Basic principles of curriculum and instruction. Chicago:Chicago University Press, 1949

The planned curriculum• What is intended by the designers

The delivered curriculum• What is organised by the administrators• What is taught by the teachers

The experienced curriculum• What is learned by the students

Three levels of a curriculum

Students

Education improvesclinical service

Clinical serviceimproves education

Curriculum Health servicescommunities

“Symbiosis” necessary for a curriculum. From Bligh J et al (see “Furtherreading” box)

5

“behavioural objectives.” Once defined, the objectives are thenused to determine the other elements of the curriculum(content; teaching and learning strategies; assessment; andevaluation).

This model has attracted some criticism—for example, that itis difficult and time consuming to construct behaviouralobjectives. A more serious criticism is that the model restricts thecurriculum to a narrow range of student skills and knowledgethat can be readily expressed in behavioural terms. Higher orderthinking, problem solving, and processes for acquiring valuesmay be excluded because they cannot be simply stated inbehavioural terms. As a result of such criticism the objectivesmodel has waned in popularity. The importance of being clearabout the purpose of the curriculum is well accepted.

More recently, another prescriptive model of curriculumdesign has emerged. “Outcomes based education” is similar inmany respects to the objectives model and again starts from asimple premise—the curriculum should be defined by theoutcomes to be obtained by students. Curriculum designproceeds by working “backwards” from outcomes to the otherelements (content; teaching and learning experiences;assessment; and evaluation).

The use of outcomes is becoming more popular in medicaleducation, and this has the important effect of focusingcurriculum designers on what the students will do rather thanwhat the staff do. Care should be taken, however, to focus onlyon “significant and enduring” outcomes. An exclusive concernwith specific competencies or precisely defined knowledge andskills to be acquired may result in the exclusion of higher ordercontent that is important in preparing medical professionals.

Although debate may continue about the precise form ofthese statements of intent (as they are known), they constitutean important element of curriculum design. It is now wellaccepted that curriculum designers will include statements ofintent in the form of both broad curriculum aims and morespecific objectives in their plans. Alternatively, intent may beexpressed in terms of broad and specific curriculum outcomes.The essential function of these statements is to requirecurriculum designers to consider clearly the purposes of whatthey do in terms of the effects and impact on students.

Descriptive modelsAn enduring example of a descriptive model is the situationalmodel advocated by Malcolm Skilbeck, which emphasises theimportance of situation or context in curriculum design. In thismodel, curriculum designers thoroughly and systematicallyanalyse the situation in which they work for its effect on whatthey do in the curriculum. The impact of both external andinternal factors is assessed and the implications for thecurriculum are determined.

Although all steps in the situational model (includingsituational analysis) need to be completed, they do not need tobe followed in any particular order. Curriculum design couldbegin with a thorough analysis of the situation of thecurriculum or the aims, objectives, or outcomes to be achieved,but it could also start from, or be motivated by, a review ofcontent, a revision of assessment, or a thorough considerationof evaluation data. What is possible in curriculum designdepends heavily on the context in which the process takesplace.

All the elements in curriculum design are linked. They arenot separate steps. Content should follow from clear statementsof intent and must be derived from considering external andinternal context. But equally, content must be delivered by

Behavioural objectives*Acceptable verbsx To writex To recitex To identifyx To differentiatex To solvex To constructx To listx To comparex To contrast

Unacceptable verbsx To knowx To understandx To really understandx To appreciatex To fully appreciatex To grasp the significance ofx To enjoyx To believex To have faith in

*From Davies I. Objectives in curriculum design. London: McGraw Hill, 1976

Clearly stated objectives provide a goodstarting point, but behavioural objectivesare no longer accepted as the “goldstandard” in curriculum design

Example of statements of intentAimx To produce graduates with knowledge and skills for treating

common medical conditions

Objectivesx To identify the mechanisms underlying common diseases of the

circulatory systemx To develop skills in history taking for diseases of the circulatory

system

Broad outcomex Graduates will attain knowledge and skills for treating common

medical conditionsx Students will identify the mechanisms underlying common diseases

of the circulatory systemx Students will acquire skills in history taking for diseases of the

circulatory system

Situational analysis*External factorsx Societal expectations and

changesx Expectations of employersx Community assumptions

and valuesx Nature of subject

disciplinesx Nature of support systemsx Expected flow of resources

Internal factorsx Studentsx Teachersx Institutional ethos and

structurex Existing resourcesx Problems and

shortcomings in existingcurriculum

*From Reynolds J, Skilbeck M. Culture and the classroom. London: Open Books,1976

Desiredoutcomes(students

will beable to...)

Content • Teaching• Learning

Assessment Evaluation

Outcomes based curriculum (defining a curriculum “backwards”—that is,from the starting point of desired outcomes)

ABC of Learning and Teaching in Medicine

6

appropriate teaching and learning methods and assessed byrelevant tools. No one element—for example, assessment—should be decided without considering the other elements.

Curriculum mapsCurriculum maps provide a means of showing the linksbetween the elements of the curriculum. They also display theessential features of the curriculum in a clear and succinctmanner. They provide a structure for the systematicorganisation of the curriculum, which can be representeddiagrammatically and can provide the basis for organising thecurriculum into computer databases.

The starting point for the maps may differ depending onthe audience. A map for students will place them at the centreand will have a different focus from a map prepared forteachers, administrators, or accrediting authorities. They all havea common purpose, however, in showing the scope, complexity,and cohesion of the curriculum.

Curriculum maps with computer based graphics with“click-on” links are an excellent format. The maps provide oneway of tracing the links between the curriculum as planned, asdelivered, and as experienced. But like all maps, a balance mustbe achieved between detail and overall clarity of representation.

Situationalanalysis

Statementsof intent

Monitoringand evaluation

Programme building(content)

Organisation andimplementation

Programme building(teaching and learning)

Programme building(assessment)

The situational model, which emphasises the importance of situation orcontext in curriculum design

Situation

Students

IntentsEvaluation

ContentOrganisation

Teaching and learningAssessment

Content

Clearly statedStudent feedback

AppropriateExplicit organisation

Student oriented

Clear blueprint

• Backgrounds• Abilities• Experience

• Aims• Goals• Outcomes

• Questionnaires• Focus groups• Participation

• Scope, sequence• Related to aims• Related to practice

• Blocks• Units• Timetables

• Variety of methods• Opportunity for self direction• Learning in real life settings

• Formative• Summative

Example of a curriculum map from the students’ perspective. Each of theboxes representing the elements of design can be broken down into furtherunits and each new unit can be related to the others to illustrate theinterlinking of all the components of the curriculum

Further readingx Bligh J, Prideaux D, Parsell G. PRISMS: new educational strategies

for medical education. Med Educ 2001;35:520-1.x Harden R, Crosby J, Davis M. Outcome based education: part 1—an

introduction to outcomes-based education. Med Teach1991;21(1):7-14.

x Harden R. Curriculum mapping: a tool for transparent andauthentic teaching and learning. Med Teach 2000;23(2):123-7.

x Prideaux D. The emperor’s new clothes: from objectives tooutcomes. Med Educ 2000;34:168-9.

x Print M. Curriculum development and design. Sydney: Allen andUnwin, 1993.

Curriculum design

7

3 Problem based learningDiana Wood

Problem based learning is used in many medical schools in theUnited Kingdom and worldwide. This article describes thismethod of learning and teaching in small groups and explainswhy it has had an important impact on medical education.

What is problem based learning?In problem based learning (PBL) students use “triggers” fromthe problem case or scenario to define their own learningobjectives. Subsequently they do independent, self directedstudy before returning to the group to discuss and refine theiracquired knowledge. Thus, PBL is not about problem solvingper se, but rather it uses appropriate problems to increaseknowledge and understanding. The process is clearly defined,and the several variations that exist all follow a similar series ofsteps.

Group learning facilitates not only the acquisition ofknowledge but also several other desirable attributes, such ascommunication skills, teamwork, problem solving, independentresponsibility for learning, sharing information, and respect forothers. PBL can therefore be thought of as a small groupteaching method that combines the acquisition of knowledgewith the development of generic skills and attitudes.Presentation of clinical material as the stimulus for learningenables students to understand the relevance of underlyingscientific knowledge and principles in clinical practice.

However, when PBL is introduced into a curriculum, severalother issues for curriculum design and implementation need tobe tackled. PBL is generally introduced in the context of adefined core curriculum and integration of basic and clinicalsciences. It has implications for staffing and learning resourcesand demands a different approach to timetabling, workload,and assessment. PBL is often used to deliver core material innon-clinical parts of the curriculum. Paper based PBL scenariosform the basis of the core curriculum and ensure that allstudents are exposed to the same problems. Recently, modifiedPBL techniques have been introduced into clinical education,with “real” patients being used as the stimulus for learning.Despite the essential ad hoc nature of learning clinicalmedicine, a “key cases” approach can enable PBL to be used todeliver the core clinical curriculum.

What happens in a PBL tutorial?PBL tutorials are conducted in several ways. In this article, theexamples are modelled on the Maastricht “seven jump” process,but its format of seven steps may be shortened.

A typical PBL tutorial consists of a group of students(usually eight to 10) and a tutor, who facilitates the session. Thelength of time (number of sessions) that a group stays togetherwith each other and with individual tutors varies betweeninstitutions. A group needs to be together long enough to allowgood group dynamics to develop but may need to be changedoccasionally if personality clashes or other dysfunctionalbehaviour emerges.

Students elect a chair for each PBL scenario and a “scribe”to record the discussion. The roles are rotated for each scenario.Suitable flip charts or a whiteboard should be used forrecording the proceedings. At the start of the session,

Generic skills and attitudesx Teamworkx Chairing a groupx Listeningx Recordingx Cooperationx Respect for

colleagues’ views

x Critical evaluation ofliterature

x Self directed learningand use of resources

x Presentation skills

The group learning process: acquiring desirable learning skills

Scribe Tutor Chair Group member

• Record points made by group

• Help group order their thoughts

• Participate in discussion

• Record resources used by group

• Encourage all group members to participate

• Assist chair with group dynamics and keeping to time

• Check scribe keeps an accurate record

• Prevent side- tracking

• Ensure group achieves appropriate learning objectives

• Check understanding

• Assess performance

• Lead the group through the process

• Encourage all members to participate

• Maintain group dynamics

• Keep to time

• Ensure group keeps to task in hand

• Ensure scribe can keep up and is making an accurate record

• Follow the steps of the process in sequence

• Participate in discussion

• Listen to and respect contributions of others

• Ask open questions

• Research all the learning objectives

• Share information with others

All participants have role to play

Roles of participants in a PBL tutorial

8

depending on the trigger material, either the student chairreads out the scenario or all students study the material. If thetrigger is a real patient in a ward, clinic, or surgery then astudent may be asked to take a clinical history or identify anabnormal physical sign before the group moves to a tutorialroom. For each module, students may be given a handbookcontaining the problem scenarios, and suggested learningresources or learning materials may be handed out atappropriate times as the tutorials progress.

The role of the tutor is to facilitate the proceedings (helpingthe chair to maintain group dynamics and moving the groupthrough the task) and to ensure that the group achievesappropriate learning objectives in line with those set by thecurriculum design team. The tutor may need to take a moreactive role in step 7 of the process to ensure that all the studentshave done the appropriate work and to help the chair tosuggest a suitable format for group members to use to presentthe results of their private study. The tutor should encouragestudents to check their understanding of the material. He or shecan do this by encouraging the students to ask open questionsand ask each other to explain topics in their own words or bythe use of drawings and diagrams.

PBL in curriculum designPBL may be used either as the mainstay of an entire curriculumor for the delivery of individual courses. In practice, PBL isusually part of an integrated curriculum using a systems basedapproach, with non-clinical material delivered in the context ofclinical practice. A module or short course can be designed toinclude mixed teaching methods (including PBL) to achieve thelearning outcomes in knowledge, skills, and attitudes. A smallnumber of lectures may be desirable to introduce topics orprovide an overview of difficult subject material in conjunctionwith the PBL scenarios. Sufficient time should be allowed eachweek for students to do the self directed learning required forPBL.

Writing PBL scenariosPBL is successful only if the scenarios are of high quality. Inmost undergraduate PBL curriculums the faculty identifieslearning objectives in advance. The scenario should leadstudents to a particular area of study to achieve those learningobjectives (see box on page 11).

Examples of trigger material for PBL scenariosx Paper based clinical scenariosx Experimental or clinical laboratory datax Photographsx Video clipsx Newspaper articlesx All or part of an article from a scientific journalx A real or simulated patientx A family tree showing an inherited disorder

PBL tutorial processStep 1—Identify and clarify unfamiliar terms presented in the scenario;

scribe lists those that remain unexplained after discussionStep 2—Define the problem or problems to be discussed; students may

have different views on the issues, but all should be considered;scribe records a list of agreed problems

Step 3—“Brainstorming” session to discuss the problem(s), suggestingpossible explanations on basis of prior knowledge; students drawon each other’s knowledge and identify areas of incompleteknowledge; scribe records all discussion

Step 4—Review steps 2 and 3 and arrange explanations into tentativesolutions; scribe organises the explanations and restructures ifnecessary

Step 5—Formulate learning objectives; group reaches consensus onthe learning objectives; tutor ensures learning objectives arefocused, achievable, comprehensive, and appropriate

Step 6—Private study (all students gather information related to eachlearning objective)

Step 7—Group shares results of private study (students identify theirlearning resources and share their results); tutor checks learningand may assess the group

How to create effective PBL scenarios*x Learning objectives likely to be defined by the students after

studying the scenario should be consistent with the faculty learningobjectives

x Problems should be appropriate to the stage of the curriculum andthe level of the students’ understanding

x Scenarios should have sufficient intrinsic interest for the students orrelevance to future practice

x Basic science should be presented in the context of a clinicalscenario to encourage integration of knowledge

x Scenarios should contain cues to stimulate discussion andencourage students to seek explanations for the issues presented

x The problem should be sufficiently open, so that discussion is notcurtailed too early in the process

x Scenarios should promote participation by the students in seekinginformation from various learning resources

*Adapted from Dolmans et al. Med Teacher 1997;19:185-9

Define learning outcomes for module

Clinical skills Lectures PBLCommunication

skills Practicals

How will individual outcomes be achieved?

How many PBL sessions in the module?

Refine scenarios and tutor notes

Pilot with staff

Write learningobjectives for each PBL

Write PBLscenarios

Write tutornotes

Pilot with group of students

Design timetable for module;write module handbook for students

Implement module

Evaluate

Designing and implementing a curriculum module using PBL supported byother teaching methods

Problem based learning

9

Staff developmentIntroducing PBL into a course makes new demands on tutors,requiring them to function as facilitators for small grouplearning rather than acting as providers of information. Staffdevelopment is essential and should focus on enabling the PBLtutors to acquire skills in facilitation and in management ofgroup dynamics (including dysfunctional groups).

Tutors should be also given information about theinstitution’s educational strategy and curriculum programme sothat they can help students to understand the learningobjectives of individual modules in the context of thecurriculum as a whole. Methods of assessment and evaluationshould be described, and time should be available to discussanxieties.

Staff may feel uncertain about facilitating a PBL tutorial fora subject in which they do not themselves specialise. Subjectspecialists may, however, be poor PBL facilitators as they aremore likely to interrupt the process and revert to lecturing.None the less, students value expertise, and the best tutors aresubject specialists who understand the curriculum and haveexcellent facilitation skills. However, enthusiastic non-specialisttutors who are trained in facilitation, know the curriculum, andhave adequate tutor notes, are good PBL tutors.

Assessment of PBLStudent learning is influenced greatly by the assessmentmethods used. If assessment methods rely solely on factualrecall then PBL is unlikely to succeed in the curriculum. Allassessment schedules should follow the basic principles oftesting the student in relation to the curriculum outcomes andshould use an appropriate range of assessment methods.

Assessment of students’ activities in their PBL groups isadvisable. Tutors should give feedback or use formative orsummative assessment procedures as dictated by the facultyassessment schedule. It is also helpful to consider assessment ofthe group as a whole. The group should be encouraged toreflect on its PBL performance including its adherence to theprocess, communication skills, respect for others, and individualcontributions. Peer pressure in the group reduces the likelihoodof students failing to keep up with workload, and the award of agroup mark—added to each individual’s assessment schedule—encourages students to achieve the generic goals associatedwith PBL.

ConclusionPBL is an effective way of delivering medical education in acoherent, integrated programme and offers several advantagesover traditional teaching methods. It is based on principles ofadult learning theory, including motivating the students,encouraging them to set their own learning goals, and givingthem a role in decisions that affect their own learning.

Predictably, however, PBL does not offer a universalpanacea for teaching and learning in medicine, and it hasseveral well recognised disadvantages. Traditional knowledgebased assessments of curriculum outcomes have shown little orno difference in students graduating from PBL or traditionalcurriculums. Importantly, though, students from PBLcurriculums seem to have better knowledge retention. PBL alsogenerates a more stimulating and challenging educationalenvironment, and the beneficial effects from the genericattributes acquired through PBL should not be underestimated.

Advantages and disadvantages of PBLAdvantages of PBLStudent centred PBL—It fosters

active learning, improvedunderstanding, and retentionand development of lifelonglearning skills

Generic competencies—PBL allowsstudents to develop generic skillsand attitudes desirable in theirfuture practice

Integration—PBL facilitates anintegrated core curriculum

Motivation—PBL is fun for studentsand tutors, and the processrequires all students to beengaged in the learning process

“Deep” learning—PBL fosters deeplearning (students interact withlearning materials, relateconcepts to everyday activities,and improve theirunderstanding)

Constructivist approach—Studentsactivate prior knowledge andbuild on existing conceptualknowledge frameworks

Disadvantages of PBLTutors who can’t “teach”—Tutors

enjoy passing on theirown knowledge andunderstanding so mayfind PBL facilitationdifficult and frustrating

Human resources—More staffhave to take part in thetutoring process

Other resources—Large numbersof students need access tothe same library andcomputer resourcessimultaneously

Role models—Students may bedeprived access to aparticular inspirationalteacher who in a traditionalcurriculum would deliverlectures to a large group

Information overload—Studentsmay be unsure how muchself directed study to do andwhat information is relevantand useful

Further readingx Davis MH, Harden RM. AMEE medical education guide number

15: problem-based learning: a practical guide. Med Teacher1999;21:130-40.

x Norman GR, Schmidt HG. Effectiveness of problem-based learningcurricula: theory, practice and paper darts. Med Educ 2000;34:721-8.

x Albanese M. Problem based learning: why curricula are likely toshow little effect on knowledge and clinical skills. Med Educ2000;34:729-38.

Christ and St John with Angels by Peter Paul Rubens is from the collection ofthe Earl of Pembroke/BAL.The Mad Hatter’s Tea Party is by John Tenniel.

A dysfunctional group: a dominant character may makeit difficult for other students to be heard

ABC of Learning and Teaching in Medicine

10

PBL scenarios: the importance of linking to faculty learning objectivesPBL scenario 1A 35 year old part time nurse, presented to her generalpractitioner, Dr Smith, with a six month history of weight loss(12.7 kg). When questioned, she said she was eating well but haddiarrhoea. She also felt exhausted and had developed insomnia.On further questioning she admitted to feeling increasingly hotand shaky and to having muscle weakness in her legs, particularlywhen climbing stairs. She was normally well and had not seen thedoctor since her last pregnancy eight years ago.

A blood test showed the following results:Free thyroxine 49.7 pmol/l (normal range 11 to 24.5)Total thyroxine 225 nmol/l (normal range 60 to 150)Thyroid stimulating hormone < 0.01 mU/l (0.4 to 4.0)

Dr Smith referred her to an endocrinologist at the local hospitalwhere initial investigations confirmed a diagnosis of Graves’disease. She was treated with carbimazole and propranolol forthe first month of treatment followed by carbimazole alone.After discussing the therapeutic options, she opted to haveiodine-131 treatment.

Faculty learning objectivesx Describe the clinical features of thyrotoxicosis and diagnostic

signs of Graves' diseasex Interpret basic thyroid function tests in the light of the pituitary

thyroid axis and feedback mechanismsx List the types of treatment for thyrotoxicosis including their

indications, mode of action, and potential side effectsNotesThis scenario is part of a core endocrinology and metabolism module forthird year undergraduate medical students. The faculty learning objectivesrelate to the scenario; the problem is relevant to the level of study andintegrates basic science with clinical medicine. The combination of basicscience, clinical medicine, and therapeutics should lead to extensivediscussion and broadly based self directed learning

PBL scenario 2Mr JB, a 58 year old car mechanic with a history of chronicobstructive pulmonary disease, was at work when he complainedof pain in his chest. The pain steadily got worse and he describedan aching in his jaw and left arm. One hour after the pain startedhe collapsed and his colleagues called an ambulance. When hearrived at the local accident and emergency department, Mr JBwas pale, sweaty, and in severe pain.

Examination showed:Blood pressure 80/60 mm HgHeart rate 64 beats/minElectrocardiography showed anterolateral myocardialinfarction

He was treated with diamorphine, metoclopramide, and aspirin.As the accident and emergency staff were preparing to give himstreptokinase, he had a cardiac arrest. Electrocardiographyshowed asystolic cardiac arrest. Despite all efforts, resuscitationfailed.

Faculty learning objectivesx List the risk factors for myocardial infarctionx Describe a rehabilitation programme for patients who have

had a myocardial infarctionNotesThis scenario is part of a core module in the cardiorespiratory system forfirst year undergraduate medical students. The scenario is complex forstudents with limited clinical experience. The faculty learning objectivesrelate to public health and epidemiological aspects of ischaemic heartdisease. For increased impact, the faculty illustrated the case with adramatic scenario. Students would be unlikely to arrive at the sameobjectives, probably concentrating on clinical aspects of acute myocardialinfarction and its management

Problem based learning

11

4 EvaluationJill Morrison

Evaluation is an essential part of the educational process. Thefocus of evaluation is on local quality improvement and isanalogous to clinical audit. Medical schools require evaluationas part of their quality assurance procedures, but the value ofevaluation is much greater than the provision of simple auditinformation. It provides evidence of how well students’ learningobjectives are being achieved and whether teaching standardsare being maintained. Importantly, it also enables thecurriculum to evolve. A medical curriculum should constantlydevelop in response to the needs of students, institutions, andsociety. Evaluation can check that the curriculum is evolving inthe desired way. It should be viewed positively as contributing tothe academic development of an institution and its members.

Evaluation versus researchEvaluation and educational research are similar activities butwith important differences. Research is usually aimed atproducing generalisable results that can be published in peerreviewed literature, and it requires ethical and other safeguards.Evaluation is generally carried out for local use and does notusually require ethics committee approval. Evaluation has to becarefully considered by curriculum committees, however, toensure that it is being carried out ethically. Finally, evaluation isa continuous process, whereas research may not becomecontinuous if the answer to the question is found.

What should be evaluatedEvaluation may cover the process and/or outcome of anyaspect of education, including the delivery and content ofteaching. Questions about delivery may relate toorganisation—for example, administrative arrangements,physical environment, and teaching methods. Information mayalso be sought about the aptitude of the teacher(s) involved. Thecontent may be evaluated for its level (it should not be too easyor too difficult), its relevance to curriculum objectives, andintegration with previous learning.

Outcome measures may show the impact of the curriculumon the knowledge, skills, attitudes, and behaviour of students.Kirkpatrick described four levels on which to focus evaluation;these have recently been adapted for use in health educationevaluation by Barr and colleagues. Some indication of theseattributes may be obtained by specific methods of inquiry—forexample, by analysing data from student assessments.

Evaluation in curriculum planningEvaluation should be designed at the start of developing acurriculum, not added as an afterthought. When an educationalneed has been identified, the first stage is to define the learningoutcomes for the curriculum. The goals of the evaluationshould be clearly articulated and linked to the outcomes.

Clarifying the goals of evaluation will help to specify theevidence needed to determine success or failure of the training..A protocol should then be prepared so that individualresponsibilities are clearly outlined.

Purpose of evaluationx To ensure teaching is meeting students’ learning needsx To identify areas where teaching can be improvedx To inform the allocation of faculty resourcesx To provide feedback and encouragement for teachersx To support applications for promotion by teachersx To identify and articulate what is valued by medical schoolsx To facilitate development of the curriculum

Kirkpatrick’s four levels on which to focus evaluation*x Level 1—Learner’s reactionsx Level 2a—Modification of attitudes and perceptionsx Level 2b—Acquisition of knowledge and skillsx Level 3—Change in behaviourx Level 4a—Change in organisational practicex Level 4b—Benefits to patients or clients

*Adapted by Barr et al (see “Further reading” box)

The full impact of the curriculum may not be known untilsome time after the student has graduated

Questions to ask when planning an evaluationx What are the goals of the evaluation?x From whom and in what form will data be collected?x Who will collect and analyse data?x What type of analysis, interpretation, and decision rules will be used

and by whom?x Who will see the results of the evaluation?

Teaching or learning activity

Reflection and analysis

Collection andassessment

of evaluation dataPlanning andpreparation

Evaluation cycle. From Wilkes et al (see “Further reading” box)

12

Designing evaluationAn ideal evaluation method would be reliable, valid, acceptable,and inexpensive. Unfortunately, ideal methods for evaluatingteaching in medical schools are scarce.

Establishing the reliability and validity of instruments andmethods of evaluation can take many years and be costly.Testing and retesting of instruments to establish theirpsychometric properties without any additional benefit forstudents or teachers is unlikely to be popular with them. Thereis a need for robust “off the shelf” instruments that can be usedto evaluate curriculums reliably. The process of evaluation itselfmay produce a positive educational impact if it emphasisesthose elements that are considered valuable and important bymedical schools.

Participation by studentsSeveral issues should be considered before designing anevaluation that collects information from students.

Competence—Students can be a reliable and valid source ofinformation. They are uniquely aware of what they can consume,and they observe teaching daily. They are also an inexpensiveresource. Daily contact, however, does not mean that students areskilled in evaluation. Evaluation by students should be limited toareas in which they are competent to judge.

Ownership—Students who are not committed to anevaluation may provide poor information. They need to feelownership for an evaluation by participating in its development.The importance of obtaining the information and the type ofinformation needed must be explicit. Usually the results of anevaluation will affect only subsequent cohorts of students, socurrent students must be convinced of the value of providingdata.

Sampling—Students need to feel that their time is respected.If they are asked to fill out endless forms they will resent thewaste of their time. If they become bored by tedious repetition,the reliability of the data will deteriorate. One solution is to usedifferent sampling strategies for evaluating different elements ofa curriculum. If reliable information can be obtained from 100students, why collect data from 300?

Anonymity is commonly advocated as a guard against biaswhen information is collected from students. However, thosewho support asking students to sign evaluation forms say thatthis helps to create a climate of responsible peer review. Ifstudents are identifiable from the information they provide, thismust not affect their progress. Data should be collectedcentrally and students’ names removed so that they cannot beidentified by teachers whom they have criticised.

Feedback—Students need to know that their opinions arevalued, so they should be told of the results of the evaluationand given details of the resulting action.

Methods of evaluationEvaluation may involve subjective and objective measures andqualitative and quantitative approaches. The resources devotedto evaluation should reflect its importance, but excessive datacollection should be avoided. A good system should be easy toadminister and use information that is readily available.

Interviews—Individual interviews with students are useful ifthe information is sensitive—for example, when a teacher hasreceived poor ratings from students, and the reasons are notclear. A group interview can provide detailed views fromstudents or teachers. A teaching session can end with reflectionby the group.

Characteristics of an ideal evaluationx Reliabilityx Validityx Acceptability—to evaluator and to person being evaluatedx Inexpensiveness

To reduce possible bias in evaluation,collect views from more than one groupof people—for example, students,teachers, other clinicians, and patients

Areas of competence of students to evaluate teaching andcurriculumx Design: whether the curriculum enables students to reach their

learning objectives; whether it fits well with other parts of thecurriculum

x Delivery: attributes of teacher and methods usedx Administrative arrangements

Participation by teachers in evaluation

Self evaluation• Academic staff increasingly evaluate their own teaching practice• Self evaluation is useful if the objective is to provide motivation to change beahviour• To help define what they are doing, teachers may find it useful to use videotapes made during teaching, logbooks, and personal portfolios

Peer evaluation• Direct observation of teachers by their peers can provide an informed, valuable, and diagnostic evaluation• Mutual classroom exchange visits between trusted colleagues can be valuable to both the teacher and the observer

Issues relating to students’ participation in evaluation may also apply toteachers, but self evaluation and peer evaluation are also relevant

Evaluation

13

Surveys—Questionnaires are useful for obtaininginformation from large numbers of students or teachers aboutthe teaching process. Electronic methods for administeringquestionnaires may improve response rates. The quality of thedata, however, is only as good as the questions asked, and thedata may not provide the reasons for a poorly rated session.

Information from student assessment—Data from assessment areuseful for finding out if students have achieved the learningoutcomes of a curriculum. A downward trend in examinationresults over several cohorts of students may indicate adeficiency in the curriculum. Caution is needed wheninterpreting this source of information, as students’ examinationperformance depends as much on their application, ability, andmotivation as on the teaching.

Completing the evaluation cycleThe main purpose of evaluation is to inform curriculumdevelopment. No curriculum is perfect in design and delivery. Ifthe results of an evaluation show that no further development isneeded, doubt is cast on the methods of evaluation or theinterpretation of the results.

This does not mean that curriculums should be in aconstant state of change, but that the results of evaluation tocorrect deficiencies are acted on, that methods continue toimprove, and that content is updated. Then the process starts allover again.

Questionnaire surveys are the mostcommon evaluation tool

Key pointsEvaluation should:x Enable strategic development of a curriculumx Be a positive process that contributes to the academic development

of a medical school

The goals of an evaluation should:x Be clearly articulatedx Be linked to the outcomes of the teaching

When carrying out an evaluation:x More than one source and type of information should be soughtx The results should be fed back to participants and details of the

resulting action given

Learners need:x To be involved in developing an evaluationx To feel their time is respectedx To know their opinions are valued and acted on

Evaluators must:x Act on the results of the evaluation to correct deficiencies, improve

methods, and update contentx Repeat the process

Further readingx Robson C. Small scale evaluation. London: Sage, 2000.x Cohen L, Manion L. Research methods in education. 4th ed. London:

Routledge, 1994.x Snell L, Tallett S, Haist S, Hays R, Norcini J, Prince K, et al. A review

of the evaluation of clinical teaching: new perspectives andchallenges. Med Educ 2000;34:862-70.

x Barr H, Freeth D, Hammick M, Koppel, Reeves S. Evaluations ofinterprofessional education: a United Kingdom review of health and socialcare. London: CAIPE/BERA, 2000.

x Wilkes M, Bligh J. Evaluating educational interventions. BMJ1999;318:1269-72.

Examples of methods of evaluation

Worked exampleBackgroundClinical teaching staff think that students are becoming weaker atexamining cranial nerves. The examination scores for that part of theobjective structured clinical examination (OSCE) carried out at theend of year three show a decline over several years. Three focusgroups are held with students in year four, and several clinicalteachers are interviewed. The results suggest that the decline is due tofewer appropriate patients presenting at outpatient sessions wherecranial nerve examination is taught and to a lack of opportunities forpractising examination skills.

InterventionA teaching session is designed for delivery in the clinical skills centre.After that, students should be able to do a systematic examination ofcranial nerves. They should also recognise normal signs and knowwhich common abnormalities to look for. Sessions are timetabled forpractising skills learnt during the teaching session.

EvaluationA questionnaire is developed for completion by a third of students. Itseeks their views on the teaching process, including the teaching skillsof the tutor, physical aspects of the teaching environment,appropriateness of the teaching material, and opportunities forpractising examination skills. Outcome measures include comparisonof examination scores for students in the previous cohort with thoseparticipating in the teaching session, plus a questionnaire for allclinical supervisors for neurology in the following year to get theirviews about students’ examination skills. A tenth of students with arange of scores in the relevant part of the OSCE are interviewed tofind out the reasons for their varied scores. The evaluation results aredisseminated widely to staff and students.

ABC of Learning and Teaching in Medicine

14

Qualitative

Analysis of taskssuccessfullycompleted in anOSCE station

ObjectiveSubjective

Focus groupsInterviews

QuantitativeTrends inexaminationscores

OSCE = objective structured clinical examination

Surveys

5 Teaching large groupsPeter Cantillon

Lecturing or large group teaching is one of the oldest forms ofteaching. Whatever their reputation, lectures are an efficientmeans of transferring knowledge and concepts to large groups.They can be used to stimulate interest, explain concepts,provide core knowledge, and direct student learning.

However, they should not be regarded as an effective way ofteaching skills, changing attitudes, or encouraging higher orderthinking. Large group formats tend to encourage passivelearning. Students receive information but have littleopportunity to process or critically appraise the new knowledgeoffered.

How can lectures be used to maximise learning and provideopportunities for student interaction? This article will supplysome of the answers and should help you to deliver better,more interactive lectures.

Getting your bearingsIt is important to find out as much as possible about the contextof the lecture—that is, where it fits into the course of which it ispart.

An understanding of the context will allow you to prepare alecture that is both appropriate and designed to move studentson from where they are.

Helping students to learn in lecturesAn important question for any lecturer to consider whenplanning a teaching session is, “how can I help my students tolearn during my lecture?” There are several different techniquesyou can use to aid student learning in a large group setting.

Planning your lectureIt is important to distinguish between the knowledge andconcepts that are essential (need to know) and those which,though interesting, are not part of the core message (nice toknow).

The aims of the lecture should be clearly defined (“what do Ihope to achieve with this lecture?”). These will help to definethe teaching methods and the structure. If, for example, thepurpose of the lecture is to introduce new knowledge andconcepts, then a classic lecture structure might be mostappropriate.

On the other hand, if the purpose is to make the studentsaware of different approaches to a particular clinical problem, aproblem oriented design in which alternative approaches arepresented and discussed might be a more appropriate format.

What you need to know before planning a lecturex How your lecture fits into the students’ course or curriculumx The students’ knowledge of your subject—try to get a copy of the

lecture and tutorial list for the coursex How the course (and your lecture) will be assessedx The teaching methods that the students are accustomed to

The successful teacher is no longer on aheight, pumping knowledge at highpressure into passive receptacles . . . he isa senior student anxious to help hisjuniors.

William Osler (1849-1919)

A lecturer holds forth . . .

• Outline purpose of lecture• Describe main themes that will be covered

Optional student activity designed toreinforce learning first key point

• Outline and explain first key point• Illustrate with examples• Repeat first key point

• Outline and explain second key point• Illustrate with examples• Repeat second key point

• Summarise• Repeat main themes and conclude

Optional student activity designed toreinforce learning second key point

Example of a lecture plan with a classic structure

Helping your students to learnx Use concrete examples to illustrate abstract principlesx Give handouts of the lecture slides, with space to write notesx Give handouts with partially completed diagrams and lists for the

students to complete during or after the lecturex Allow for pauses in the delivery to give students time to write notesx Check for understanding by asking questions or by running a mini

quiz

15

Choosing teaching mediaWhen you have selected the content of the lecture and placed itinto a working structure, the next consideration is how todeliver the message. Which teaching media should be used (forexample, slides, overheads, handouts, quizzes)? The mostappropriate media will differ depending on the venue, class size,and topic.

Getting startedIn the first moments of a lecture it is important that thestudents are given some sense of place and direction. Thus abrief summary of the previous lecture and an indication of themajor themes and learning objectives for the current sessionprovide both you and the students with a relatively easy start. Ifyou are working with a new group it may be useful to indicatethe ground rules for the session—for example, “switch offmobile phones,” or “ask questions at any time.”

Encouraging students to interactStudents learn well by “doing.” Yet there is an understandabletendency for students to regard lectures as an opportunity to sitback, be entertained, and “soak up” the learning. However, youcan use various methods to encourage students to take a moreactive part in the learning process.

Students’ attention (and recall) is best at the beginning andend of a lecture. Recall can be improved by changing theformat of your lecture part way through. It is also importantwhen planning a lecture to think about activities and exercisesthat will break up the presentation.

Ask questionsIt is useful to ask questions of the group at various stages in thelecture, to check comprehension and promote discussion. Manylecturers are intimidated by the silence following a question andfall into the trap of answering it themselves. Wait for theanswers to come. It takes time for students to move fromlistening to thinking mode. A simple tip is to count slowly to 10in your head—a question is almost certain to arrive.

Get students to ask you questionsAn alternative to getting students to answer questions is to askthem to direct questions at you. A good way of overcomingstudents’ normal fear of embarrassment is to ask them toprepare questions in groups of two or three. Questions can thenbe invited from groups at random. When asked a question, youshould repeat it out loud to ensure that the whole group isaware of what was asked. Seeking answers to the question fromother students, before adding your own views, can increase thelevel of interaction further.

Handoutsx Handouts can encourage better learning if they allow students

more time to listen and thinkx Handouts should provide a scaffold on which students can build

their understanding of a topicx Handouts should provide a summary of the major themes while

avoiding an exhaustive explanation of eachx Handouts can be used to direct further learning, by including

exercises and questions with suggested reading lists

“Tell me, and I forget. Show me, and I remember.Involve me, and I understand”

Chinese proverb

Statement of problem

Optional student activity based onsolution 1

Offer solution 1

Discuss strengths and weaknessesof solution 1

Offer solution 2

Discuss strengths and weaknessesof solution 2

Summary and concluding remarks

Optional student activity based onsolution 2

Example of a lecture plan with a problem oriented structure

Time

Stud

ents

' rec

all o

f fac

ts

Start of lecture End of lectureStudentactivity

Worse

Better Lecture without student activityLecture with student activity

Graph showing effect of students’ interaction on their ability to recall whatthey have heard in a lecture. Adapted from Bligh, 2000 (see “Recommendedreading” box)

Choosing the medium for delivering the lecturex Which teaching media are available at the teaching venue?x Which teaching media are you familiar with? (It is not always

appropriate to experiment with new media)x Which medium will best illustrate the concepts and themes that you

want to teach the students?x Which medium would encourage students to learn through

interaction during your lecture?

ABC of Learning and Teaching in Medicine

16

BrainstormingBrainstorming is a technique for activating the students’knowledge or current understanding of an issue or theme. Thelecturer invites answers to a question or problem from theaudience and writes them, without comment, on a board oroverhead. After a short period, usually about two or threeminutes, the lecturer reviews the list of “answers” with the class.The answers can be used to provide material for the next partof the lecture or to give students an idea of where they arebefore they move on. By writing answers in a way that can beseen by everyone in the audience, you allow the students tolearn from each other.

Buzz groupsBuzz groups also encourage interaction. They consist of groupsof two to five students working for a few minutes on a question,problem, or exercise set by the lecturer. Buzz group activity is auseful means of getting students to process and use newinformation to solve problems. At the end of the buzz groupsession, the teacher can either continue with the lecture orcheck the results of the exercise by asking one or two groups topresent their views. Remember that in an amphitheatre lecturehall, students can sit on their own desks to interact with thestudents behind them.

Mini-assessmentsMini-assessments and exercises are used in lectures to helpstudents to recognise gaps in their learning and to encouragethem to use new material in practice. Brief assessments can alsoallow the lecturer to measure how well the messages are beingunderstood. Students could be asked, for example, to completea brief, multiple choice questionnaire or a “one-minute” paper.The timing of quizzes and exercises will depend on what isrequired. An assessment of prior learning would be best at thestart of a lecture, whereas an estimate of learning from thecurrent session might be best carried out towards the end of thelecture.

How to end your lectureAt the end of a lecture it is important to summarise the keypoints and direct students toward further learning. You maypresent the key points on a slide or overhead. Alternatively, youmay go through the main headings on a handout. Students areencouraged to learn more about a subject if they are set tasks orexercises that will require them to look further than the lecturenotes for answers and ideas. The end of a lecture is also acommon time for questions. Students may find the use of aone-minute paper a useful tool to help them to identifyconcepts and impressions that need clarification.

Evaluating your lecturePractice does make perfect, but the process of developing as alecturer is greatly helped if some effort is made to evaluateperformance. Evaluation involves answering questions such as“how did I do?” or “what did the students learn?”

A lecture can be evaluated in different ways. If the studentsare to be used as a source of feedback, the following methodsare useful:x Ask a sample of the students if you can read their lecturenotes—this exercise gives some insight into what students havelearned and understoodx Ask for verbal feedback from individual studentsx Ask the students to complete a one-minute paper

"One-minute" paper worksheet

Name:

Date:

Directions: Take a moment to think about the lecture you have just attended, and then answer the following questions.

1. What was the most important thing you learned in today's lecture?

Lecture title:

2. What question remains uppermost in your mind at the end of today's lecture?

3. What was the "muddiest point" in today's lecture?

Example of a one-minute paper

Please rate the lecture on the following items

Stronglyagree

Clear

Interesting

Easy to takenotes from

Well organised

Relevant tothe course

Slightlyagree

Slightlydisagree

Stronglydisagree

Example of an evaluation form focusing on the lecture. Adapted fromBrown et al, 2001 (see “Recommended reading” box)

Teaching large groups

17

x Ask the students to complete an evaluation questionnaire.If you want to evaluate your teaching style and delivery,

peers can be a useful source of feedback:x Ask a colleague to observe part or all of a lecture and providefeedback afterwards. It is important to inform the observer whataspects of the lecturing process you want evaluated—forexample, clarity, logical flow, effectiveness of the media usedx Videotape the lecture for private viewing, and arrange a jointviewing with a colleague later.

Lectures are still a common teaching method in bothundergraduate and postgraduate medical education. Theircontinued popularity is due to the fact that they represent aneffective and efficient means of teaching new concepts andknowledge. This article has emphasised the importance of goodlecture planning and of the inclusion of student interaction toensure effective learning.

Please rate the lecturer on the following items

Stronglyagree

Wasenthusiastic

Was clearlyaudible

Seemedconfident

Gave clearexplanations

Encouragedparticipation

Slightlyagree

Slightlydisagree

Stronglydisagree

Example of an evaluation form focusing on the lecturer rather than thelecture. Adapted from Brown et al, 2001 (see “Recommended reading” box)

Recommended readingx Newble DI, Cannon R. A handbook for medical teachers. 4th ed.

Dordrecht, Netherlands: Kluwer Academic, 2001.x Gibbs G, Habeshaw T. Preparing to teach. Bristol: Technical and

Educational Services, 1989.x Bligh DA. What’s the use of lectures? San Francisco: Jossey-Bass, 2000.x Brown G, Manogue M. AMEE medical education guide No 22:

refreshing lecturing: a guide for lecturers. Medical Teacher2001;23:231-44.

18

6 Teaching small groupsDavid Jaques

Group discussion plays a valuable role in the all-roundeducation of students, whether in problem based learning andteam projects or in the more traditional academic scenario ofthe tutorial or seminar. When it works well, discussion can allowstudents to negotiate meanings, express themselves in thelanguage of the subject, and establish closer contact withacademic staff than more formal methods permit. Discussioncan also develop the more instrumental skills of listening,presenting ideas, persuading, and working as part of a team. Butperhaps most importantly, discussion in small groups can orshould give students the chance to monitor their own learningand thus gain a degree of self direction and independence intheir studies.

All these worthy aims require active participation and theready expression of ideas. Yet it frequently doesn’t work out thisway. Indeed many tutors too readily fall back on their reservepositions of authority, expert, and prime talker. Many of theproblems associated with leading small groups effectively arelikely to be exacerbated with larger groups. So how can weavoid the common traps?

If you are leading a group discussion you will need toconsider both the configuration of the group and your ownbehaviour. Groups often communicate poorly because thephysical conditions make it difficult to communicate. Forexample, in a group of 10 students seated round a rectangulartable, at least four students on either side of the table have no eyecontact with each other, thus reducing participation. If you askand answer questions all the time, even less interaction is likely.

If a group sits in a circle without a table, communication islikely to be easier. When the discussion has started, it is yourresponsibility as discussion leader to listen to and respond tothe whole group. Listening becomes a problem when thestudents regard you as an expert or you engage with one or twoof the more vocal students rather than the whole group.

More structure, less interventionBeing a democratic discussion leader involves making the rightsort of nudges and interventions. The role can be made a lotless demanding by using more structure and less intervention inthe group process. The rest of this article shows how clear andpurposeful group structures can help to bypass many of theproblems outlined above, by delegating responsibility for groupinteraction (and therefore for learning) to the students.

Group structures and processesYou can minimise your internal involvement in the groupprocess by organising or structuring groups into smaller units,especially when the group process is likely to be problematical.This is particularly so when you wish to mobilise a sense ofcoherence and full participation among a largish group ofstudents. A sequence of tasks might then be set. For example:x Students work individually for five minutes drawing up a listx They share their ideas in pairs for 10 minutesx In groups of four to six, students write up categories on alarge sheet of paperx This is followed by 25 minutes of open discussion among thegroups.

“By separating teaching from learning, we have teacherswho do not listen and students who do not talk”Based on Palmer P (The Courage to Teach. Jossey Bass,1998)

Problems associated with leading effective small groupsx The teacher gives a lecture rather than conducting a dialoguex The teacher talks too muchx Students cannot be encouraged to talk except with difficulty; they

will not talk to each other, but will only respond to questions fromthe tutor

x Students do not prepare for the sessionsx One student dominates or blocks the discussionx The students want to be given the solutions to problems rather than

discuss them

Your own behaviour can have anenormous effect on how the groupfunctions

Techniques for effective facilitation in group discussionx Ensure that group members have an agreed set of ground

rules—for example, not talking at the same time as another groupmember

x Ensure that the students are clear about the tasks to be carried outx When you present a question don’t answer it yourself or try to

reformulate it—count to 10 silently before speaking againx When you have something you could say (which could be most of

the time), count to 10 againx Look round the group both when you are speaking and when a

student is speaking. That way the students will quickly recognisethat they are addressing the group rather than just you. It will allowyou to pick up cues from those who want to speak but are either abit slow or inhibited

Step 1Consider what you want the students to learn or achieve - in other words, what the

learning outcomes should be (for example, students will be able to identify andcompetently use three different general strategies for solving patients' problems)

Step 2Choose a suitable set of group tasks to deliver the desired outcomes. For example:

• The group is given a problem to solve• The students have to monitor the problem solving strategies that they are to use

• They then share their findings and compare them with research evidence• They draw up a classification of the findings

Step 3Decide how to organise the small group. Your tasks

are to prepare any materials, explain and check agreement on thetasks, monitor the development of the tasks, and control time boundaries

Planning the structure of a small discussion group

19

Your role in this kind of situation may be to move roundchecking that everyone understands and accepts the task and isdoing it in an appropriate way and to encourage completion asthe end point approaches. You could leave the room for a whileand let the groups work without supervision.

The following group structures require some assertiveleadership to set up but allow you to take a back seat as theprocess itself takes over the direction of events.

Group roundEach person has a brief time—say, 20 seconds or one minute—to say something in turn round the group. The direction roundthe group can be decided by the first contributor, or memberscan speak in a random order. More interest and energy isusually generated, however, if the first person chooses whoshould go second, the second who should go third, and so on.

Buzz groupsWith larger groups a break is often needed:x To provide a stimulating change in the locus of attentionx For you to gain some idea of what the students knowx For the students to check their own understanding.

During a discussion students could be asked to turn to theirneighbour to discuss for a few minutes any difficulties inunderstanding, to answer a prepared question, or to speculateon what they think will happen next in the proceedings. Thiswill bring a sense of participation and some lively feedback.Buzz groups enable students to express difficulties they wouldhave been unwilling to reveal to the whole class. (A variation isto allocate three or five minutes each way to the pairs—eachphase is for one-way communication.)

Snowball groupsSnowball groups (or pyramids) are an extension of buzz groups.Pairs join up to form fours, then fours to eights. These groupsof eight report back to the whole group. This developingpattern of group interaction can ensure comprehensiveparticipation, especially when it starts with individuals writingdown their ideas before sharing them. To avoid studentsbecoming bored with repeated discussion of the same points, itis a good idea to use increasingly sophisticated tasks as thegroups gets larger.

FishbowlsThe usual fishbowl configuration has an inner group discussingan issue or topic while the outer group listens, looking forthemes, patterns, or soundness of argument or uses a groupbehaviour checklist to give feedback to the group on itsfunctioning. The roles may then be reversed.

Crossover groupsStudents are divided into subgroups that are subsequently splitup to form new groups in such a way as to maximise thecrossing over of information. A colour or number coding in thefirst groupings enables a simple relocation—from, for example,three groups of four students to four groups of three, with eachgroup in the second configuration having one from each of thefirst.

Circular questioningIn circular questioning each member of the group asks aquestion in turn. In its simplest version, one group memberformulates a question relevant to the theme or problem andputs it to the person opposite, who has a specified time (say, one

To encourage group interaction considerbreaking a larger group into smallergroups of five or six students; organisemembership on a heterogeneous orrandom basis to prevent cliques forming

Group rounds are particularly useful atthe beginning of any meeting so thateveryone is involved from the start and,depending on what the group is asked tospeak about, as a way of checking onlearning issues

Buzz groups, with pairs for one-way, five-minute communication

Fishbowl structure—inside group discusses, outside grouplistens in

Crossover groups—redistribution of 12 students (each allocated one of fourcolours) for second period of session

ABC of Learning and Teaching in Medicine

20

or two minutes) to answer it. Follow up questions can be asked iftime permits. The questioning and answering continuesclockwise round the group until everyone has contributed, atwhich time a review of questions and answers can take place.This could also include answers that others would like to havegiven. Alternatively, you or the students could prepare thequestions on cards. You could also mix the best of the students’questions with some of your own.

Horseshoe groupsThis method allows you to alternate between the lecture anddiscussion formats, a common practice in workshops. Groupsare arranged around tables, with each group in a horseshoeformation with the open end facing the front. You can thus talkformally from the board for a time before switching topresenting a group task. Subsequent reporting from each groupcan induce boredom. To avoid this danger, the tutor cancirculate written reports for comment; get groups to intervieweach other publicly or get one member of each group tocirculate; ask groups to produce and display posters; ask thereporters from each group to form an inner group in afishbowl formation; or use the crossover method to movestudents around.

ConclusionThis article has emphasised the choices available to you inworking with groups. Some of these involve more skilled andsensitive handling of group process from within the group;others require imaginative management in the setting of tasksand the organising of purposeful activities for subgroups. Wellorganised and purposeful group discussion can create a firmfoundation for qualities such as openness, networking, andproactive communication—important ingredients in the processof personal and organisational change. The value of effectivegroup management in professional development and lifelonglearning cannot be underestimated.

Recommended readingx Brookfield S, Preskill S. Discussion as a way of teaching—tools and

techniques for university teachers. Buckingham: Open University Press,1999.

x Forster F, Hounsell D, Thompson S. Tutoring and demonstrating—ahandbook. Sheffield: Universities’ and Colleges’ Staff DevelopmentAgency, 1995.

x Habeshaw T, Habeshaw S, Gibbs G. 53 interesting things to do in yourseminars and tutorials. Bristol: Technical and Educational Services,1992.

x Jaques D. Learning in groups. 3rd ed. London: Kogan Page, 2000.x Tiberius R. Small group teaching: a trouble-shooting guide. London:

Kogan Page, 1999.

Circularquestioning

Horseshoe groups

The group structures described require an explicit taskand topic, and they are possible only if the furniture ismovable. Tutors could also consider experimenting withfurniture to see if other group structures work. Thephysical configuration is a strong determinant of social(and hence learning) processes, as is the sequence ofactivities

Teaching small groups

21

7 One to one teaching and feedbackJill Gordon

Although it is not clear whether Dr Sylvius (above) wasdescribing his teaching method in relation to a group ofstudents or to a succession of individual students, heunderstood the essential features of clinical education. Heunderstood, for example, the need for active learning in anauthentic clinical setting.

Dr Sylvius also understood another important feature ofone to one teaching—close behavioural observation (of eachother, teacher and learner). No other setting provides the sameopportunity for this. Dr Sylvius led his students “by [the] hand.”He cared about his role as a teacher. In the closely observed oneto one relationship your unguarded statements, your reactionsunder pressure, and your opinions about other people and theworld at large are all magnified.

Just as you cannot hide from the learner, so the learner’sknowledge, skills, and attitudes will become apparent to you.Provided that you have created a trusting relationship, you candiscuss his or her personal and professional attitudes and valuesin a way that is seldom possible in a larger group. This isperhaps one of the key benefits of one to one teaching.

Another feature of one to one teaching is the opportunity toadjust what you teach to the learner’s needs—“customise” yourteaching. In 1978 Ausubel and colleagues suggested that thesecret of education is to find out what the learner already knowsand teach accordingly. In a lecture, tutorial, or seminar youcannot hope to diagnose and respond to every individual’slearning needs, but a one to one relationship provides anopportunity to match the learning experience to the learner.

One to one teaching is perhaps one of the most powerfulways of “influencing students.” You can create opportunities foractive learning in authentic clinical settings while modellingdesirable personal and professional attributes.

Stott and Davis in 1979 promoted the idea that one to oneprimary care consultations offer exceptional but oftenunrealised potential. The principles used in primary careconsultations can be applied to one to one teaching, and thesecret is forethought and planning.

“My method (is) to lead my students by hand to thepractice of medicine, taking them every day to seepatients in the public hospital, that they may hear thepatients’ symptoms and see their physical findings.Then I question the students as to what they have notedin the patients and about their thoughts and perceptionsregarding the cause of the illness and the principles oftreatment”

Dr Franciscus de la Boe Sylvius, 17th century professorof medicine at the University of Leyden, Netherlands

Wards, operating theatres, generalpractice, and community clinics provide acontext for active learning

As a teacher, you are an important rolemodel whether you wish it or not

What’s different about one to one teaching?

Lecture Seminar PBLgroup

Clinicaltutorial

One to oneclinicalattachment

Efficiency* High Medium Low Low Very lowActivelearning

Low(usually)

Variable High Mediumto high

Very high

Mutualfeedback

Low Medium High Mediumto high

Very high

Modellingbehaviourin real lifesetting Low Low Medium High Very high

PBL = problem based learning.*Based on student numbers.

Plan ahead—ask yourself some important questionsx What is the main purpose of the one to one attachment?x Do you know why it is part of the learning programme?x What are the learner’s needs?x How will you gauge how effectively you have met the learner’s

needs?x How would you like this learner to describe the experience to a peer?Exceptional potential of one to one teaching

x It tackles current learning needsx It promotes autonomy and self directed learningx It links prior knowledge with new clinical experiencesx It enables opportunistic teaching

Dr Franciscus de laBoe Sylvius

22

Provide an orientationMost of us recall clinical teachers whose social skills amountedto a brief glance and a grunt. Times have changed, or shouldhave. Find out and remember the learner’s name—a simple butimportant courtesy. Outline the special opportunities andbenefits that the attachment can provide. Ask the learner toprepare a learning plan and then compare the learner’s plan toyour own expectations. Once the plan has been agreed, don’tshelve it—refer to it during the attachment and modify asnecessary.

Agree on the ground rulesGround rules are both practical (punctuality, dress, access topatient records) and philosophical (respect for patients andcolleagues, confidentiality, consent, openness to different pointsof view). If these have been spelled out on day 1, you won’t becaught out later. Make sure that the learner knows how muchtime you will be able to spend in observing, teaching, and givingfeedback and what you expect in return.

Ask helpful questionsOpen ended questions are generally better than closedquestions at the beginning of the exchange. A small number ofclosed questions later in the conversation help you to“diagnose” just how much the learner knows and understands.Avoid questions that require nothing but recall. Try toformulate questions that assume an appropriate amount ofknowledge, but build in higher order thinking and/or higherorder skills. You might ask the learner, for example, to explainto you (as if you were the patient) the mechanisms behind acondition such as asthma or hypertension. This simulatesclinical interface with a patient—testing recall, understanding,and communications skills all at once.

Give feedbackLearners value feedback highly, and valid feedback is based onobservation. Deal with observable behaviours and be practical,timely, and concrete. The one to one relationship enables you togive feedback with sensitivity and in private. Begin by asking thelearner to tell you what he or she feels confident of having donewell and what he or she would like to improve. Follow up withyour own observations of what was done well (be specific), andthen outline one or two points that could help the student toimprove.

Encourage reflectionJust as many learning opportunities are wasted if they are notaccompanied by feedback from an observer, so too are theywasted if the learner cannot reflect honestly on his or herperformance. One to one teaching is ideally suited toencouraging reflective practice, because you can model the waya reflective practitioner behaves. Two key skills are (a)“unpacking” your clinical reasoning and decision makingprocesses and (b) describing and discussing the ethical valuesand beliefs that guide you in patient care.

Use other one to one teachersSenior medical students, junior doctors, registrars, nurses, andallied health professionals are all potential teachers. When

Skilful teaching is not unlike skilfulhistory taking

“If musicians learned to play their instruments asphysicians learn to interview patients, the procedurewould consist of presenting in lectures or maybe in ademonstration or two the theory and mechanisms ofthe music-producing ability of the instrument andtelling him to produce a melody. The instructor ofcourse, would not be present to observe or listen to thestudent’s efforts, but would be satisfied with thestudent’s subsequent verbal report of what came out ofthe instrument.”

George Engel, after visiting 70 medical schools inNorth America

Monitor progressx Identify deficienciesx Ask the learner, half way through the attachment, to do a self

assessment of how things are going. If both you and the learner canidentify deficiencies within a safe learning environment, you canwork together to tackle them well before the attachment ends

x If you have serious concerns, you have an obligation to make themknown to the learner and to the medical school or trainingauthority

x It is not appropriate to diagnose serious problems and hand thelearner on to the next stage of training in the hope that theproblems will somehow be correct themselves

Find out and remember the learner’s name—a simple but important courtesy

One to one teaching and feedback

23

junior colleagues interact with a learner, you can encouragethem with positive feedback on their teaching.

Every patient interview and every physical examinationplaces the learner in a privileged relationship with a patient. Weall have patients whom we especially admire—particularlypeople who have coped bravely with a chronic illness or amajor disability, a disaster such as war, or other misfortunes.Such patients activate an emotional response in the learner,imprinting an enriched memory of the patient and the patient’sillness.

Reap the rewardsThe role of the teacher is frequently undervalued, and yetteaching is potentially rewarding and enjoyable. It is also one ofthe defining features of a profession. Without teaching toensure the transmission of knowledge, medicine becomes justanother “job.”

One to one teaching inevitably exposes you to evaluation bythe learner. If the learner trusts you, he or she will be able to tellyou what has worked well, and what could be improved.Respond to feedback by reflecting rather than by explaining,excusing, or offering counter arguments to defend yourparticular style. Openness to feedback is another professionalattribute that is best modelled one to one.

Learners for whom you have been a role model and mentorare likely to repay you many times over. Sometimes you havethe opportunity to observe their personal and professionaldevelopment long after the one to one attachment has finished.Dr Sylvius knew how to do it, and Cicero understood itsrejuvenating qualities.

Points to rememberDox Welcomex Set shared achievable goalsx Put yourself in the learner’s shoesx Ask interesting questionsx Monitor progress and give feedbackx Encourage

Don’tx Appear unpreparedx Be vague about your expectationsx Confine the learner to passive rolesx Be “nit-picking”x Leave feedback to the final assessmentx Humiliate

Further readingx Parsell G, Bligh J. Recent perspectives on clinical teaching. Med Educ

2001;35:409-14.x Paulman PM, Susman JL, Abboud CA, eds. Precepting medical

students in the office. London: Johns Hopkins University Press, 2000.x Westberg J, Jason H. Collaborative clinical education: the foundation of

effective health care. New York: Springer, 1993.x Whitehouse C, Roland M, Campion P, eds. Teaching medicine in the

community: a guide for undergraduate education. New York: OxfordUniversity Press, 1997.

x Stott N, Davis R. The exceptional potential in each primary careconsultation. J R Coll Gen Pract 1979;29:201-5.

x Ausubel D, Novak J, Hanesian H. Educational psychology: a cognitiveview. New York: Rinehart and Winston, 1978.

The illustration on p 544 is reproduced with permission from JakeWyman/Photonica.

The illustration of the young Cicero reading (by Vicenzo Foppa) isreproduced with permission from the Bridgeman Art Library.

“Cultivate the society of the young, remain interested and never stoplearning” (Cicero)

Promote active learningx Time is limited in most clinical settings, and it can be tempting to

revert to a passive observational teaching modelx Think about strategies to promote active learningx Brief students to observe specific features of a consultation or

procedurex Ask patients for permission for the learner to carry out all or part

of the physical examination or a procedure while you observex If space is available, allow students to interview patients in a

separate room or cubicle before presenting them to youx If possible videotape consultations for a debriefing session at a

more convenient timex Arrange for the learner to see the same patient over time, or in

another context, such as a home visit

ABC of Learning and Teaching in Medicine

24

8 Learning and teaching in the clinical environmentJohn Spencer

Clinical teaching—that is, teaching and learning focused on, andusually directly involving, patients and their problems—lies atthe heart of medical education. At undergraduate level, medicalschools strive to give students as much clinical exposure aspossible; they are also increasingly giving students contact withpatients earlier in the course. For postgraduates, “on the job”clinical teaching is the core of their professional development.How can a clinical teacher optimise the teaching and learningopportunities that arise in daily practice?

Strengths, problems, and challengesLearning in the clinical environment has many strengths. It isfocused on real problems in the context of professionalpractice. Learners are motivated by its relevance and throughactive participation. Professional thinking, behaviour, andattitudes are “modelled” by teachers. It is the only setting inwhich the skills of history taking, physical examination, clinicalreasoning, decision making, empathy, and professionalism canbe taught and learnt as an integrated whole. Despite thesepotential strengths, clinical teaching has been much criticisedfor its variability, lack of intellectual challenge, and haphazardnature. In other words, clinical teaching is an educationallysound approach, all too frequently undermined by problems ofimplementation.

The importance of planningMany principles of good teaching, however, can (and should) beincorporated into clinical teaching. One of the most importantis the need for planning. Far from compromising spontaneity,planning provides structure and context for both teacher andstudents, as well as a framework for reflection and evaluation.Preparation is recognised by students as evidence of a goodclinical teacher.

How doctors teachAlmost all doctors are involved in clinical teaching at somepoint in their careers, and most undertake the jobconscientiously and enthusiastically.

However, few receive any formal training in teaching skills,and in the past there has been an assumption that if a personsimply knows a lot about their subject, they will be able to teachit. In reality, of course, although subject expertise is important, it

Common problems with clinical teachingx Lack of clear objectives and expectationsx Focus on factual recall rather than on development of problem

solving skills and attitudesx Teaching pitched at the wrong level (usually too high)x Passive observation rather than active participation of learnersx Inadequate supervision and provision of feedbackx Little opportunity for reflection and discussionx “Teaching by humiliation”x Informed consent not sought from patientsx Lack of respect for privacy and dignity of patientsx Lack of congruence or continuity with the rest of the curriculum

Clinical teaching in general practice

Questions to ask yourself when planning a clinical teaching session

Challenges of clinical teachingx Time pressuresx Competing demands—clinical (especially when needs of patients

and students conflict); administrative; researchx Often opportunistic—makes planning more difficultx Increasing numbers of studentsx Fewer patients (shorter hospital stays; patients too ill or frail; more

patients refusing consent)x Often under-resourcedx Clinical environment not “teaching friendly” (for example, hospital

ward)x Rewards and recognition for teachers poor

25

is not sufficient. Effective clinical teachers use several distinct, ifoverlapping, forms of knowledge.

How students learnUnderstanding the learning process will help clinical teachersto be more effective. Several theories are relevant (see firstarticle in the series, 25 January). All start with the premise thatlearning is an active process (and, by inference, that theteacher’s role is to act as facilitator). Cognitive theories arguethat learning involves processing information through interplaybetween existing knowledge and new knowledge. An importantinfluencing factor is what the learner knows already. The qualityof the resulting new knowledge depends not only on“activating” this prior knowledge but also on the degree ofelaboration that takes place. The more elaborate the resultingknowledge, the more easily it will be retrieved, particularly whenlearning takes place in the context in which the knowledge willbe used.

Experiential learningExperiential learning theory holds that learning is often mosteffective when based on experience. Several models have beendescribed, the common feature being a cyclical process linkingconcrete experience with abstract conceptualisation throughreflection and planning. Reflection is standing back andthinking about experience (What did it mean? How does itrelate to previous experience? How did I feel?). Planninginvolves anticipating the application of new theories and skills(What will I do next time?). The experiential learning cycle,which can be entered at any stage, provides a useful frameworkfor planning teaching sessions.

QuestionsQuestions may fulfil many purposes, such as to clarifyunderstanding, to promote curiosity, and to emphasise keypoints. They can be classified as “closed,” “open,” and“clarifying” (or “probing”) questions.

Closed questions invoke relatively low order thinking, oftensimple recall. Indeed, a closed question may elicit no responseat all (for example, because the learner is worried about beingwrong), and the teacher may end up answering their ownquestion.

How to use cognitive learning theory in clinical teachingHelp students to identify what they already knowx “Activate” prior knowledge through brainstorming and briefing

Help students elaborate their knowledgex Provide a bridge between existing and new information—for

example, use of clinical examples, comparisons, analogiesx Debrief the students afterwardsx Promote discussion and reflectionx Provide relevant but variable contexts for the learning

Example of clinical teaching session based on experientiallearning cycleSetting—Six third year medical students doing introductory clinical

skills course based in general practiceTopic—History taking and physical examination of patients with

musculoskeletal problems (with specific focus on rheumatoidarthritis); three patients with good stories and signs recruited fromthe community

The sessionPlanning—Brainstorm for relevant symptoms and signs: this activates

prior knowledge and orientates and provides framework andstructure for the task

Experience—Students interview patients in pairs and do focusedphysical examination under supervision: this provides opportunitiesto implement and practise skills

Reflection—Case presentations and discussion: feedback and discussionprovides opportunities for elaboration of knowledge

Theory—Didactic input from teacher (basic clinical information aboutrheumatoid arthritis): this links practice with theory

Planning— “What have I learned?” and “How will I approach such apatient next time?” Such questions prepare students for the nextencounter and enable evaluation of the session

Effective teaching depends crucially on the teacher’scommunication skills. Two important areas ofcommunication for effective teaching are questioning andgiving explanations. Both are underpinned by attentivelistening (including sensitivity to learners’ verbal andnon-verbal cues). It is important to allow learners toarticulate areas in which they are having difficulties orwhich they wish to know more about

Casebased

"teachingscripts"

Knowledgeaboutthe

learners

Knowledgeabout thesubject

Knowledge aboutlearning and

teaching

Learningcontext

Knowledgeabout

the patient

Various domains of knowledge contribute to the idiosyncratic teachingstrategies (“teaching scripts”) that tutors use in clinical settings

Reflection

Theory

Experience

Planning

Experiential learning cycle: the role of the teacher is to help students tomove round, and complete, the cycle

ABC of Learning and Teaching in Medicine

26

In theory, open questions are more likely to promotedeeper thinking, but if they are too broad they may be equallyineffective. The purpose of clarifying and probing questions isself evident.

ExplanationTeaching usually involves a lot of explanation, ranging from the(all too common) short lecture to “thinking aloud.” The latter isa powerful way of “modelling” professional thinking, giving thenovice insight into experts’ clinical reasoning and decisionmaking (not easily articulated in a didactic way). There are closeanalogies between teacher-student and doctor-patientcommunication, and the principles for giving clear explanationsapply. If in doubt, pitch things at a low level and work upwards.As the late Sydney Jacobson, a journalist, said, “Neverunderestimate the person’s intelligence, but don’t overestimatetheir knowledge.” Not only does a good teacher avoid answeringquestions, but he or she also questions answers.

Exploiting teaching opportunitiesMost clinical teaching takes place in the context of busypractice, with time at a premium. Many studies have shown thata disproportionate amount of time in teaching sessions may bespent on regurgitation of facts, with relatively little on checking,probing, and developing understanding. Models for using timemore effectively and efficiently and integrating teaching intoday to day routines have been described. One such, the“one-minute preceptor,” comprises a series of steps, each ofwhich involves an easily performed task, which when combinedform an integrated teaching strategy.

Teaching on the wardsDespite a long and worthy tradition, the hospital ward is not anideal teaching venue. None the less, with preparation andforethought, learning opportunities can be maximised withminimal disruption to staff, patients, and their relatives.

Approaches include teaching on ward rounds (eitherdedicated teaching rounds or during “business” rounds);students seeing patients on their own (or in pairs—students canlearn a lot from each other) then reporting back, with orwithout a follow up visit to the bedside for further discussion;and shadowing, when learning will inevitably be moreopportunistic.

Key issues are careful selection of patients; ensuring wardstaff know what’s happening; briefing patients as well asstudents; using a side room (rather than the bedside) fordiscussions about patients; and ensuring that all relevantinformation (such as records and x ray films) is available.

Teaching in the clinicAlthough teaching during consultations is organisationallyappealing and minimally disruptive, it is limited in what it canachieve if students remain passive observers.

With relatively little impact on the running of a clinic,students can participate more actively. For example, they can be

How to use questionsx Restrict use of closed questions to establishing facts or baseline

knowledge (What? When? How many?)x Use open or clarifying/probing questions in all other circumstances

(What are the options? What if?)x Allow adequate time for students to give a response—don’t speak

too soonx Follow a poor answer with another questionx Resist the temptation to answer learners’ questions—use counter

questions insteadx Statements make good questions—for example, “students

sometimes find this difficult to understand” instead of “Do youunderstand?” (which may be intimidating )

x Be non-confrontational—you don’t need to be threatening to bechallenging

Questions can be sequenced to draw out contributions orbe built on to promote thinking at higher cognitive levelsand to develop new understanding

How to give effective explanationsx Check understanding before you start, as you proceed, and at the

end—non-verbal cues may tell you all you need to know aboutsomeone’s grasp of the topic

x Give information in “bite size” chunksx Put things in a broader context when appropriatex Summarise periodically (“so far, we’ve covered . . .”) and at the end;

asking learners to summarise is a powerful way of checking theirunderstanding

x Reiterate the take home messages; again, asking students will giveyou feedback on what has been learnt (but be prepared for somesurprises)

Teaching during consultations has been much criticisedfor not actively involving learners

Patientencounter(history,

examination, etc)

Teach generalprinciples

("When thathappens, do

this..." )

Get acommitment

("What do youthink is

going on?" )

Help learneridentify and

give guidanceabout omissions

and errors("Although your

suggestionof Y was apossibility,

in a situationlike this, Z ismore likely,because..." )

Probe forunderlyingreasoning("What ledyou to that

conclusion?" )

Reinforce what was done well("Your diagnosis of X was wellsupported by the history..." )

“One-minute preceptor” model

Learning and teaching in the clinical environment

27

asked to make specific observations, write down thoughts aboutdifferential diagnosis or further tests, or note any questions—fordiscussion between patients. A more active approach is “hotseating.” Here, the student leads the consultation, or part of it.His or her findings can be checked with the patient, anddiscussion and feedback can take place during or after theencounter. Students, although daunted, find this rewarding. Athird model is when a student sees a patient alone in a separateroom, and is then joined by the tutor. The student then presentstheir findings, and discussion follows. A limitation is that theteacher does not see the student in action. It also inevitablyslows the clinic down, although not as much as might beexpected. In an ideal world it would always be sensible to blockout time in a clinic to accommodate teaching.

The patient’s roleSir William Osler’s dictum that “it is a safe rule to have noteaching without a patient for a text, and the best teaching isthat taught by the patient himself” is well known. Theimportance of learning from the patient has been repeatedlyemphasised. For example, generations of students have beenexhorted to “listen to the patient—he is telling you thediagnosis.” Traditionally, however, a patient’s role has beenessentially passive, the patient acting as interesting teachingmaterial, often no more than a medium through which theteacher teaches. As well as being potentially disrespectful, this isa wasted opportunity. Not only can patients tell their stories andshow physical signs, but they can also give deeper and broaderinsights into their problems. Finally, they can give feedback toboth learners and teacher. Through their interactions withpatients, clinical teachers—knowingly or unknowingly—have apowerful influence on learners as role models.

Drs Gabrielle Greveson and Gail Young gave helpful feedback on earlydrafts.

Working effectively and ethically with patientsx Think carefully about which parts of the teaching session require

direct patient contact—is it necessary to have a discussion at thebedside?

x Always obtain consent from patients before the students arrivex Ensure that students respect the confidentiality of all information

relating to the patient, verbal or writtenx Brief the patient before the session—purpose of the teaching

session, level of students’ experience, how the patient is expected toparticipate

x If appropriate, involve the patient in the teaching as much aspossible

x Ask the patient for feedback—about communication and clinicalskills, attitudes, and bedside manner

x Debrief the patient after the session—they may have questions, orsensitive issues may have been raised

Suggested readingx Cox K. Planning bedside teaching. (Parts 1 to 8.) Med J Australia

1993;158:280-2, 355-7, 417-8, 493-5, 571-2, 607-8, 789-90, and159:64-5.

x Parsell G, Bligh J. Recent perspectives on clinical teaching. Med Educ2001;35:409-14.

x Hargreaves DH, Southworth GW, Stanley P, Ward SJ. On-the-joblearning for physicians. London: Royal Society of Medicine, 1997.

Seating arrangements for teaching in clinic or surgery

ABC of Learning and Teaching in Medicine

28

Patient

Student

"Sitting in" as observer

Teacher

Patient

Student

Three way consultation

Teacher

Patient

Teacher

"Hot seating"

Student

9 Written assessmentLambert W T Schuwirth, Cees P M van der Vleuten

Some misconceptions about written assessment may still exist,despite being disproved repeatedly by many scientific studies.Probably the most important misconception is the belief thatthe format of the question determines what the questionactually tests. Multiple choice questions, for example, are oftenbelieved to be unsuitable for testing the ability to solve medicalproblems. The reasoning behind this assumption is that all astudent has to do in a multiple choice question is recognise thecorrect answer, whereas in an open ended question he or shehas to generate the answer spontaneously. Research hasrepeatedly shown, however, that the question’s format is oflimited importance and that it is the content of the questionthat determines almost totally what the question tests.

This does not imply that question formats are alwaysinterchangeable—some knowledge cannot be tested withmultiple choice questions, and some knowledge is best nottested with open ended questions.

Five criteria can be used to evaluate the advantages anddisadvantages of question types: reliability, validity, educationalimpact, cost effectiveness, and acceptability. Reliability pertainsto the accuracy with which a score on a test is determined.Validity refers to whether the question actually tests what it ispurported to test.

Educational impact is important because students tend tofocus strongly on what they believe will be in the examinations.Therefore they will prepare strategically depending on thequestion types used. Whether different preparation leads todifferent types of knowledge is not fully clear, however. Whenteachers are forced to use a particular question type, they willtend to ask about the themes that can be easily assessed withthat question type, and they will neglect the topics for which thequestion type is less well suited. Therefore, it is wise to vary thequestion types in different examinations.

Cost effectiveness and acceptability are important as thecosts of different examinations have to be taken into account,and even the best designed examination will not survive if it isnot accepted by teachers and students.

“True or false” questionsThe main advantage of “true or false” questions is theirconciseness. A question can be answered quickly by the student,so the test can cover a broad domain. Such questions, however,have two major disadvantages. Firstly, they are quite difficult toconstruct flawlessly—the statements have to be defensibly trueor absolutely false. Teachers must be taught thoroughly how toconstruct these question types. Secondly, when a studentanswers a “false” question correctly, we can conclude only thatthe student knew the statement was false, not that he or sheknew the correct fact.

“Single, best option” multiple choicequestionsMultiple choice questions are well known, and there is extensiveexperience worldwide in constructing them. Their mainadvantage is the high reliability per hour oftesting—mainly

Choosing the most appropriate type ofwritten examination for a certain purposeis often difficult. This article discussessome general issues of written assessmentthen gives an overview of the mostcommonly used types, together with theirmajor advantages and disadvantages

Reliabilityx A score that a student obtains on a test should indicate the score

that this student would obtain in any other given (equally difficult)test in the same field (“parallel test”)

x A test represents at best a sample—selected from a range of possiblequestions. So if a student passes a particular test one has to be surethat he or she would not have failed a parallel test, and vice versa

x Two factors influence reliability negatively:

Sample error—The number of items may be too small to provide areproducible result

Sample too narrow—If the questions focus only on a certainelement, the scores cannot generalise to the whole discipline

Validityx The validity of a test is the extent to which it measures what it

purports to measurex Most competencies cannot be observed directly (body length, for

example, can be observed directly; intelligence has to be derivedfrom observations). Therefore, in examinations it is important tocollect evidence to ensure validity:

One simple piece of evidence could be, for example, that expertsscore higher than students on the testAlternative approaches include (a) an analysis of the distribution ofcourse topics within test elements (a so called blueprint) and(b) an assessment of the soundness of individual test items.

x Good validation of tests should use several different pieces ofevidence

True or false questions are most suitablewhen the purpose of the question is totest whether students are able to evaluatethe correctness of an assumption; in othercases they are best avoided

Multiple choice questions can be used inany form of testing, except whenspontaneous generation of the answer isessential, such as in creativity,hypothesising, and writing skills

29

because they are quick to answer—so a broad domain can becovered. They are often easier to construct than true or falsequestions and are more versatile. If constructed well, multiplechoice questions can test more than simple facts. Unfortunatelythough, they are often used to test only facts, as teachers oftenthink this is all they are fit for.

Multiple true or false questionsThese questions enable the teacher to ask a question to whichthere is more than one correct answer. Although they takesomewhat longer to answer than the previous two types, theirreliability per hour of testing time is not much lower.

Construction, however, is not easy. It is important to havesufficient distracters (incorrect options) and to find a goodbalance between the number of correct options and distracters.In addition, it is essential to construct the question so thatcorrect options are defensibly correct and distracters aredefensibly incorrect. A further disadvantage is the rathercomplicated scoring procedure for these questions.

“Short answer” open ended questionsOpen ended questions are more flexible—in that they can testissues that require, for example, creativity, spontaneity—but theyhave lower reliability. Because answering open ended questionsis much more time consuming than answering multiple choicequestions, they are less suitable for broad sampling. They arealso expensive to produce and to score. When writing openended questions it is important to describe clearly how detailedthe answer should be—without giving away the answer. A goodopen ended question should include a detailed answer key forthe person marking the paper. Short answer, open endedquestions are not suitable for assessing factual knowledge; usemultiple choice questions instead.

Short answer, open ended questions should be aimed at theaspects of competence that cannot be tested in any other way.

EssaysEssays are ideal for assessing how well students can summarise,hypothesise, find relations, and apply known procedures to newsituations. They can also provide an insight into differentaspects of writing ability and the ability to process information.Unfortunately, answering them is time consuming, so theirreliability is limited.

When constructing essay questions, it is essential to definethe criteria on which the answers will be judged. A commonpitfall is to “over-structure” these criteria in the pursuit ofobjectivity, and this often leads to trivialising the questions.Some structure and criteria are necessary, but too detailed astructure provides little gain in reliability and a considerableloss of validity. Essays involve high costs, so they should be usedsparsely and only in cases where short answer, open endedquestions or multiple choice questions are not appropriate.

“Key feature” questionsIn such a question, a description of a realistic case is followed bya small number of questions that require only essentialdecisions; these questions may be either multiple choice oropen ended, depending on the content of the question. Keyfeature questions seem to measure problem solving ability

Teachers need to be taught well how towrite good multiple choice questions

Open ended questions are perhaps themost widely accepted question type. Theirformat is commonly believed to beintrinsically superior to a multiple choiceformat. Much evidence shows, however,that this assumed superiority is limited

“Key feature” questions aim to measureproblem solving ability validly withoutlosing too much reliability

Which of the following drugsbelong to the ACE inhibitor group?

(a) atenolol

(b) pindolol

(c) amiloride

(d) furosemide (frusemide)

(e) enalapril

(f) clopamide

(g) epoprostenol

(h) metoprolol

(i) propranolol

(j) triamterene

(k) captopril

(l) verapamil

(m) digoxin

Example of a multiple, true or false question

ABC of Learning and Teaching in Medicine

30

validly and have good reliability. In addition, most peopleinvolved consider them to be a suitable approach, which makesthem more acceptable.

However, the key feature approach is rather new andtherefore less well known than the other approaches. Also,construction of the questions is time consuming; inexperiencedteachers may need up to three hours to produce a single keyfeature case with questions. Experienced writers, though, mayproduce up to four an hour. Nevertheless, these questions areexpensive to produce, and large numbers of cases are normallyneeded to prevent students from memorising cases. Key featurequestions are best used for testing the application of knowledgeand problem solving in “high stakes” examinations.

Extended matching questionsThe key elements of extended matching questions are a list ofoptions, a “lead-in” question, and some case descriptions orvignettes. Students should understand that an option may becorrect for more than one vignette, and some options may notapply to any of the vignettes. The idea is to minimise therecognition effect that occurs in standard multiple choicequestions because of the many possible combinations betweenvignettes and options. Also, by using cases instead of facts, theitems can be used to test application of knowledge or problemsolving ability. They are easier to construct than key featurequestions, as many cases can be derived from one set of options.Their reliability has been shown to be good. Scoring of theanswers is easy and could be done with a computer.

The format of extended matching questions is still relativelyunknown, so teachers need training and practice before theycan write these questions. There is a risk of anunder-representation of certain themes simply because they donot fit the format. Extended matching questions are best usedwhen large numbers of similar sorts of decisions (for example,relating to diagnosis or ordering of laboratory tests) needtesting for different situations.

ConclusionChoosing the best question type for a particular examination isnot simple. A careful balancing of costs and benefits is required.A well designed assessment programme will use different typesof question appropriate for the content being tested.

Example of a key feature questionCaseYou are a general practitioner. Yesterday you made a house call on MrDowning. From your history taking and physical examination youdiagnosed nephrolithiasis. You gave an intramuscular injection of100 mg diclofenac, and you left him some diclofenac suppositories.You advised him to take one when in pain but not more than two aday. Today he rings you at 9 am. He still has pain attacks, whichrespond well to the diclofenac, but since 5 am he has also had acontinuous pain in his right side and a fever (38.9°C).

Which of the following is the best next step?(a) Ask him to wait another day to see how the disease progresses(b) Prescribe broad spectrum antibiotics(c) Refer him to hospital for an intravenous pyelogram(d) Refer him urgently to a urologist

Example of an extended matching question(a) Campylobacter jejuni, (b) Candida albicans, (c) Giardia lamblia,(d) Rotavirus, (e) Salmonella typhi, (f) Yersinia enterocolitica,(g) Pseudomonas aeruginosa, (h) Escherichia coli, (i) Helicobacter pylori,(j) Clostridium perfringens, (k) Mycobacterium tuberculosis, (l) Shigellaflexneri, (m) Vibrio cholerae, (n) Clostridium difficile, (o) Proteus mirabilis,(p) Tropheryma whippeliiFor each of the following cases, select (from the list above) themicro-organism most likely to be responsible:x A 48 year old man with a chronic complaint of dyspepsia suddenly

develops severe abdominal pain. On physical examination there isgeneral tenderness to palpation with rigidity and reboundtenderness. Abdominal radiography shows free air under thediaphragm

x A 45 year old woman is treated with antibiotics for recurringrespiratory tract infections. She develops a severe abdominal painwith haemorrhagic diarrhoea. Endoscopically apseudomembranous colitis is seen

Using only one type of questionthroughout the whole curriculum is not avalid approach

Further readingx Case SM, Swanson DB. Extended-matching items: a practical

alternative to free response questions. Teach Learn Med1993;5:107-15.

x Frederiksen N. The real test bias: influences of testing on teachingand learning. Am Psychol 1984;39:193-202.

x Bordage G. An alternative approach to PMPs: the “key-features”concept. In: Hart IR, Harden R, eds. Further developments in assessingclinical competence; proceedings of the second Ottawa conference.Montreal: Can-Heal Publications, 1987:59-75.

x Swanson DB, Norcini JJ, Grosso LJ. Assessment of clinicalcompetence: written and computer-based simulations. Assessmentand Evaluation in Higher Education 1987;12:220-46.

x Ward WC. A comparison of free-response and multiple-choiceforms of verbal aptitude tests. Applied Psychological Measurement1982;6(1):1-11.

x Schuwirth LWT. An approach to the assessment of medical problemsolving: computerised case-based testing. Maastricht: DatawysePublications, 1998. (Thesis from Department of EducationalDevelopment and Research, Maastricht University.)

Written assessment

31

10 Skill based assessmentSydney Smee

Skill based assessments are designed to measure theknowledge, skills, and judgment required for competency in agiven domain. Assessment of clinical skills has formed a keypart of medical education for hundreds of years. However, thebasic requirements for reliability and validity have not alwaysbeen achieved in traditional “long case” and “short case”assessments. Skill based assessments have to contend with casespecificity, which is the variance in performance that occursover different cases or problems. In other words, casespecificity means that performance with one patient relatedproblem does not reliably predict performance withsubsequent problems.

For a reliable measure of clinical skills, performance has tobe sampled across a range of patient problems. This is the basicprinciple underlying the development of objective structuredclinical examinations (OSCEs). Several other structured clinicalexaminations have been developed in recent years, includingmodified OSCEs—such as the Royal College of Physicians’Practical Assessment of Clinical Examination Skills (PACES)and the objective structured long case (OSLER). This articlefocuses mainly on OSCEs to illustrate the principles of skillbased assessment.

OSCEsThe objective structured clinical examination (OSCE) wasintroduced over 30 years ago as a reliable approach to assessingbasic clinical skills. It is a flexible test format based on a circuitof patient based “stations.”

At each station, trainees interact with a patient or a“standardised patient” to demonstrate specified skills.Standardised patients are lay people trained to present patientproblems realistically. The validity of interactions with realpatients, however, may be higher than that with standardisedpatients, but standardised patients are particularly valuablewhen communication skills are being tested.

OSCE stations may be short (for eample, five minutes) orlong (15-30 minutes). There may be as few as eight stations ormore than 20. Scoring is done with a task specific checklist or acombination of checklist and rating scale. The scoring of thestudents or trainees may be done by observers (for example,faculty members) or patients and standardised patients.

DesignThe design of an OSCE is usually the result of a compromisebetween the assessment objectives and logistical constraints;however, the content should always be linked to the curriculum,as this link is essential for validity.

Using many short stations should generate scores that aresufficiently reliable for making pass-fail decisions within areasonable testing time. (Whether any OSCE is sufficientlyreliable for grading decisions is debatable.) Fewer but longerstations maximise learning relative to the selected patientproblems, especially when students or trainees receive feedbackon their performance. The number of students, time factors,and the availability of appropriate space must also beconsidered.

Written tests can assess knowledge acquisition and reasoning ability, but theycannot so easily measure skills

Patient-doctor interaction for assessing clinicalperformance

Is this a practice opportunity or an end-of-course asessment?

What skills should trainees have acquired(such as history taking, physical examination,

patient education, diagnosis)?

What presenting complaints have been taught(such as abdominal pain,

shortness of breath, fatigue)?

Questions to answer when designing an OSCE

32

PlanningPlanning is critical. Patients and standardised patients can berecruited only after stations are written. Checklists must bereviewed before being printed, and their format must becompatible with the marking method, ideally computerised.OSCEs generate a lot of data—for 120 students in a 20 stationOSCE there will be 2400 mark sheets!)

Stations are the backbone of an OSCE, and yet the singlemost common problem is that station materials are incompleteand subject to last minute changes. The result is increased costand wasted time.

If OSCE scores are being used for making pass-faildecisions, then it is also necessary to set strandards. Severalmethods for setting standards have been used, with the Angoffmethod described below being the most commonly used.

Plans should allow sufficient time to process and analyse thescores carefully.

CostsOSCE costs vary greatly because the number of stationsdetermines the number of standardised patients, examiners,and staff required. Whether or not faculty members volunteerto write cases, set standards, and examine is also a significantfactor.

Developing the stationsOSCE stations have three components.

StemA standardised format for the “stem” (task) is helpful—forexample, providing the patient’s name, age, presentingcomplaint, and the setting (such as clinic, emergency, or ward)for all stations. The stem must clearly state the task—forexample, “in the next eight minutes, conduct a relevant physicalexamination.”

ChecklistThe checklist items are the actions that should be taken inresponse to the information in the stem. These items should bereviewed and edited to ensure that (a) they are appropriate forthe level of training being assessed, (b) they are task based, and(c) they are observable (so the observer can score them).

The length of the checklist depends on the clinical task, thetime allowed, and who is scoring. A checklist for a five minutestation that is testing history taking may have up to 25 items if afaculty observer is doing the scoring. If a patient or standardisedpatient is doing the scoring, then fewer items should be used.Use of detailed items will guide scorers: for example, “examinesthe abdomen” is a general item that might better be separatedinto a series of items such as “inspects the abdomen,”“auscultates the abdomen,” “lightly palpates all four quadrants,”and so on.

A score must be assigned to every item. Items may be scored1 or 0, or relative weights may be assigned, with more criticalitems being worth more. Weights may not change the overallpass-fail rate of an OSCE, but they may improve the validity of achecklist and can affect which trainees pass or fail.

Training informationFor standardised patients, directions should use patient basedlanguage, specify the patient’s perception of the problem (forexample, serious, not serious), provide only relevantinformation, and specify pertinent negatives. Responses to allchecklist items should be included. The patient’s behaviour andaffect should be described in terms of body language, verbal

Tasks to do aheadx Create blueprintx Set timeline (how long do we need?)x Get authors for a case-writing workshopx Review and finalise casesx Arrange workshop on setting standardsx Recruit standardised patients; recruit faculty members as examinersx Train standardised patientsx Print marking sheets, make signsx List all supplies for set-up of OSCE stationsx Remind everyone of datex Make sure students have all the informationx Plans for the examination day: diagram of station layout; directions

for examiners, standardised patients, and staff; possible registrationtable for students; timing and signals (for example, stopwatch andwhistles); procedures for ending the examination

x Anything else?

The fixed costs of running an OSCEremain much the same regardless of thenumber of examination candidates.Administering an OSCE twice in one dayonly slightly increases the fixed costs,although the examiners’ time is animportant cost

StemJohn Smith, aged 37, arrived in the emergency department

complaining of acute abdominal pain that began 16 hours previously.

In the next eight minutes, conduct a relevant physical examination

Training informationHistory of painThe pain started 16 hours ago, etc

SymptomsThe pain is in the right lower quadrant, at "at least 9", and is constant.His abdomen is tense, even when palpated lightly. With deeperpalpation there is guarding in the RLQ, and McBurney's point isacutely tender.Obturator (raising right knee against resistance) and psoas signs(extension of right leg at hip–kicking backwards) are positive.

ChecklistExaminer to fill in box for each item that trainee successfully completes

❑ Drapes patient appropriately❑ Inspects abdomen❑ Auscultates abdomen❑ Percusses abdomen❑ Lightly palpates each quadrant❑ Deeply palpates each quadrant❑ Checks for peritoneal irritation Etc

Marks2111222

Components of OSCE station

Skill based assessment

33

tone, and pace. Symptoms to be simulated need to bedescribed.

LimitationsSkill based assessments are based on tasks that approximateperformance in the area of interest. The assumption is that thecloser the tasks are to “real world” tasks, the more valid theassessment.

Three aspects of an OSCE limit how closely the stationsapproximate clinical practice. Firstly, time limited stations oftenrequire trainees to perform isolated aspects of the clinicalencounter. This deconstructs the doctor-patient encounter andmay be unacceptable for formative assessments. The trade-off isthat limiting the time allows for more stations, which canprovide performance snapshots that allow for reliable,summative decision making.

Secondly, OSCEs rely on task specific checklists, whichassume that the doctor-patient interaction can be described as alist of actions. As a result, checklists tend to emphasisethoroughness, and this may become a less relevant criterion asthe clinical experience of candidates increases. Thirdly, there arelimits to what can be simulated, and this constrains the natureof the patient problems that can be sampled. Again, thisbecomes more of an issue as candidates’ level of training andclinical experience increases.

Other approaches to skill basedassessmentTraditional approachesThe oral examination (also known as the “viva”) and the “longcase” have long been used for assessing clinical competence.The oral examination is traditionally an unstructured face toface session with the examiners. This allows them to explore thetrainee’s understanding of topics deemed relevant to clinicalpractice. The long case is patient based, but the interaction withthe patient is usually not observed. Instead, trainees summarisethe patient problem for the examiners and respond toexaminers’ questions about findings, diagnosis or management,and other topics deemed relevant by examiners. The strength ofthe long case is the validity that comes from the complexities ofa complete encounter with a real patient. However, the difficultyand relevance of these assessments varies greatly as the contentis limited to one or two patient problems (selected from theavailable patients), and decisions are made according tounknown criteria, as examiners make holistic judgments. Forthis reason traditional unstructured orals and long cases havelargely been discontinued in North America.

Alternative formatsAlternative formats tackle the problems associated withtraditional orals and long cases by (a) having examiners observethe candidate’s complete interaction with the patient, (b)training examiners to a structured assessment process, and/or(c) increasing the number of patient problems. For a short caseassessment, for example, one or two examiners may direct atrainee through a series of five or six encounters with realpatients. They observe, ask questions, and make a judgmentbased on the candidate’s performance with all the patients.Similarly, a structured oral examination is still a face to facesession with examiners, but guidelines for the topics to becovered are provided. Alternatively, a series of patient scenariosand agreed questions may be used so that the content anddifficulty of the assessment is standardised across the trainees.Each of these adaptations is aimed at improving reliability, but

Limitations of OSCEsx Stations often require trainees to perform isolated aspects of the

clinical encounter, which “deconstructs” the doctor-patientencounter

x OSCEs rely on task specific checklists, which tend to emphasisethoroughness. But with increasing experience, thoroughnessbecomes less relevant

x The limitations on what can be simulated constrain the type ofpatient problems that can be used

None of these limitations is prohibitive,but they should be considered whenselecting an OSCE as an assessment tooland when making inferences from OSCEscores

An alternative way to assess skills is to observe candidates’ interaction withpatients

ABC of Learning and Teaching in Medicine

34

the most important improvement comes from greatlyincreasing the number of patient problems, which may wellcause an impractical increased testing time.

Reliability and validityThe reliability of a test describes the degree to which the testconsistently measures what it is supposed to measure. The morereliable a test, the more likely it is that a similar result will beobtained if the test is readministered. Reliability is sensitive tothe length of the test, the station or item discrimination, and theheterogeneity of the cohort of candidates. Standardisedpatients’ portrayals, patients’ behaviour, examiners’ behaviour,and administrative variables also affect reliability.

The validity of a test is a measure of the degree to which thetest actually measures what it is supposed to measure. Validity isa property of test scores and justifies their interpretation for aspecific purpose. The most basic evidence of validity comesfrom documenting the links between the content of theassessment and the curriculum’s objectives and from thequalifications of those who develop the assessment.

Setting standardsChecklists generate scores; judges set standards. The validity ofa standard depends on the judges’ qualifications and thereasonableness of the procedure they use to set it. Whenpass-fail decisions are being made, a skill based assessmentshould be “criterion referenced” (that is, trainees should beassessed relative to performance standards rather than to eachother or to a reference group). An Angoff approach iscommonly used to set the standard for an OSCE.

Skill based assessments do not replace knowledge basedtests, but they do assess aspects of competence that knowledgebased tests cannot assess. Although the use of OSCEs for skillbased assessment is increasingly widespread, modifying moretraditional formats may be appropriate when they arecombined with other forms of assessment or are used to screentrainees. The success of any skill based assessment depends onfinding a suitable balance between validity and reliability andbetween the ideal and the practical.

Factors leading to lower reliabilityx Too few stations or too little testing timex Checklists or items that don’t discriminate (that is, are too easy or

too hard)x Unreliable patients or inconsistent portrayals by standardised

patientsx Examiners who score idiosyncraticallyx Administrative problems (such as disorganised staff or noisy rooms)

Questions to ensure validityx Are the patient problems relevant and important to the

curriculum?x Will the stations assess skills that have been taught?x Have content experts (generalists and specialists) reviewed the

stations?

Further readingx Gorter S, Rethans JJ, Scherpbier A, van der Heijde D, Houben H,

van der Linden S, et al. Developing case-specific checklists forstandardized-patient-based assessments in internal medicine: areview of the literature. Acad Med 2000;75:1130-7.

x Hodges B, Regehr G, McNaughton N, Tiberius RG, Hanson M.OSCE checklists do not capture increasing levels of expertise. AcadMed 1999;74:1129-34.

x Kaufman DM, Mann KV, Muijtjens AMM, van der Vleuten CPM. Acomparison of standard-setting procedures for an OSCE inundergraduate medical education. Acad Med 2001;75:267-71.

x Newble DI, Dawson B, Dauphinee WD, Page G, Macdonald M,Swanson DB, et al. Guidelines for assessing clinical competence.Teach Learn Med 1994;6:213-20.

x Norcini JJ. The death of the long case? BMJ 2002;324:408-9.x Reznick RK, Smee SM, Baumber JS, Cohen R, Rothman AI,

Blackmore DE, et al. Guidelines for estimating the real cost of anobjective structured clinical examination. Acad Med 1993;67:513-7.

The second picture and the picture showing an oral examination are fromMicrosoft Clipart.

Judges (n≥12) must first imagine the minimally competent or borderline trainee

Judges discuss rationale for each other's judgments

For each item in each checklist, judges record what they believethe chances are that their imaginary trainee will be successful

For each item in each checklist,judges revise (or not) their initial predictions

Item score = average of revised judgementsStation pass mark = average of the item scores

OSCE pass mark = average of the station pass marks

A modified Angoff procedure for an OSCE

Skill based assessment

35

11 Work based assessmentJohn J Norcini

In 1990 psychologist George Miller proposed a framework forassessing clinical competence. At the lowest level of the pyramidis knowledge (knows), followed by competence (knows how),performance (shows how), and action (does). In this framework,Miller distinguished between “action” and the lower levels.“Action” focuses on what occurs in practice rather than whathappens in an artificial testing situation. Work based methods ofassessment target this highest level of the pyramid and collectinformation about doctors’ performance in their normalpractice. Other common methods of assessment, such asmultiple choice questions, simulation tests, and objectivestructured clinical examinations (OSCEs) target the lower levelsof the pyramid. Underlying this distinction is the sensible butstill unproved assumption that assessments of actual practiceare a much better reflection of routine performance thanassessments done under test conditions.

MethodsAlthough the focus of this article is on practising doctors, workbased assessment methods apply to medical students andtrainees as well. These methods can be classified in many ways,but this article classifies them in two dimensions. The firstdimension describes the basis for making judgments about thequality of performance. The second dimension is concernedwith how data are collected.

Basis for judgmentOutcomesIn judgments about the outcomes of their patients, the qualityof a cardiologist, for example, might be judged by the mortalityof his or her patients within 30 days of acute myocardialinfarction. Historically, outcomes have been limited to mortalityand morbidity, but in recent years the number of clinical endpoints has been expanded. Patients’ satisfaction, functionalstatus, cost effectiveness, and intermediate outcomes—forexample, HbA1c and lipid concentrations for diabeticpatients—have gained acceptance.

Patients’ outcomes are the best measures of the quality ofdoctors for the public, the patients, and doctors themselves. Forthe public, outcomes assessment is a measure of accountabilitythat provides reassurance that the doctor is performing well inpractice. For individual patients, it supplies a basis for decidingwhich doctor to see. For doctors, it offers reassurance that theirassessment is tailored to their unique practice and based on realwork performance.

Despite the fact that an assessment of outcomes is highlydesirable, at least four substantial problems remain. These areattribution, complexity, case mix, and numbers.

Firstly, for a good judgment to be made about a doctor’sperformance, the patients’ outcomes must be attributable solelyto that doctor’s actions. This is not realistic when care isdelivered within systems and teams. Secondly, patients with thesame condition will vary in complexity depending on theseverity of their illness, the existence of comorbid conditions,and their ability to comply with the doctor’s recommendations.Although statistical adjustments may tackle these problems,they are not completely effective. So differences in complexitydirectly influence outcomes and make it difficult to compare

This article explains what is meant bywork based assessment and presents aclassification scheme for current methods

Three aspects of doctors’ performancecan be assessed—patients’ outcomes,process of care, and volume of practice

Does

Shows how

Knows how

Knows

Miller’s pyramid for assessing clinical competence

Clinical records

Administrative data

Diaries

Observation

Basis for judgment

Methods of collecting data

• Outcomes of care• Process of care• Practice volume

Classification for work based assessment methods

Care is delivered in teams, so judging a doctor’s performance throughoutcomes is not realistic

36

doctors or set standards for their performance. Thirdly,unevenness exists in the case mix of different doctors, againmaking it difficult to compare performance or to set standards.Finally, to estimate well a doctor’s routine performance, asizeable number of patients are needed. This limits outcomesassessment to the most frequently occurring conditions.

Process of careIn judgments about the process of care that doctors provide, ageneral practitioner, for example, might be assessed on the basisof how many of his or her patients aged over 50 years havebeen screened for colorectal cancer. General process measuresinclude screening, preventive services, diagnosis, management,prescribing, education of patients, and counselling. In addition,condition specific processes might also serve as the basis formaking judgments about doctors—for example, whetherdiabetic patients have their HbA1c monitored regularly andreceive routine foot examinations.

Measures of process of care have substantial advantagesover outcomes. Firstly, the process of care is more directly in thecontrol of the doctor, so problems of attribution are greatlyreduced. Secondly, the measures are less influenced by thecomplexity of patients’ problems—for example, doctorscontinue to monitor HbA1c regardless of the severity of thediabetes. Thirdly, some of the process measures, such asimmunisation, should be offered to all patients of a particulartype, reducing the problems of case mix.

The major disadvantage of process measures is that simplydoing the right thing does not ensure the best outcomes forpatients. That a physician regularly monitors HbA1c, forexample, does not guarantee that he or she will make thenecessary changes in management. Furthermore, althoughprocess measures are less susceptible to the difficulties ofattribution, complexity, and case mix, these factors still have anadverse influence.

VolumeJudgments about the number of times that doctors haveengaged in a particular activity might include, for example, thenumber of times a surgeon performed a certain surgicalprocedure. The premise for this type of assessment is the largebody of research showing that quality of care is associated withhigher volume.

Data collectionClinical practice recordsOne of the best sources of information about outcomes,process, and volume is the clinical practice record. The externalaudit of these records is a valid and credible source of data. Twomajor problems exist, however, with clinical practice records.

Firstly, judgment can be made only on what is recorded—this may not be an accurate assessment of what was actuallydone in practice.

Secondly, abstracting records is expensive and timeconsuming and is made cumbersome by the fact that they areoften incomplete or illegible.

Widespread adoption of the electronic medical record maybe the ultimate solution, although this is some years away.Meanwhile, some groups rely on doctors to abstract their ownrecords and submit them for evaluation. Coupled with anexternal audit of a sample of the participating doctors, this is acredible and feasible alternative.

For a sound assessment of an individualdoctor’s process of care, a sizeablenumber of patients need to be included

Advantages of volume based assessment over assessment ofoutcomes and processx Problems of attribution are reduced substantiallyx Complexity is eliminatedx Case mix is not relevant

However, such assessment alone offers no assurance that the activity wasconducted properly

Unevenness in case mix can reduce usefulness of using patients’ outcomesas a measure of doctors’ competence

Judgments on process of care might include foot examinations for diabeticpatients

Traditional medical records may give way to widespread use of electronicrecords, making data collection easier and quicker

Work based assessment

37

Administrative databasesIn some healthcare systems large computerised databases areoften developed as part of the process of administering andreimbursing for health care. Data from these sources areaccessible, inexpensive, and widely available. They can be usedin the evaluation of some aspects of practice performance—such as cost effectiveness—and of medical errors. However, thelack of clinical information and the fact that the data are oftencollected for invoicing purposes makes them unsuitable as theonly source of information.

DiariesDoctors, especially trainees, may use diaries or logs to recordthe procedures they perform. Depending on the purpose of thediary, entries can be accompanied by a description of thedoctor’s role, the name of an observer, an indication of whetherit was done properly, and a list of complications. This is areasonable way to collect data on volume and an acceptablealternative to the abstraction of clinical practice records untilmedical records are kept electronically.

ObservationData can be collected in many ways through practiceobservation, but to be consistent with Miller’s definition of workbased assessment, the observations need to be routine or covertto avoid an artificial test situation. They can be made in anynumber of ways and by any number of different observers. Themost common forms of observation based assessment areratings by supervisors, peers, and patients. Other examples ofobservation include visits by standardised patients (lay peopletrained to present patient problems realistically) to doctors intheir surgeries and audiotapes or videotapes of consultationssuch as those used by the General Medical Council.

PortfoliosDoctors typically collect from various sources the practice datathey think pertinent to their evaluation. A doctor’s portfoliomight contain data on outcomes, process, or volume, collectedthrough clinical record audit, diaries, or assessments by patientsand peers. It is important to specify what to include inportfolios as doctors will naturally present their best work, andthe evaluation of it will not be useful for continuing qualityimprovement or quality assurance. In addition, if there is adesire to compare doctors or to provide them with feedbackabout their relative performance, then all portfolios mustcontain the same data collected in a similar fashion. Otherwise,there is no basis for legitimate comparison or benchmarking.

Databases for clinical audit are becomingmore available and may provide moreuseful information relating to clinicalpractice

Peer evaluation rating formsBelow are the aspects of competence assessed with the peer ratingform developed by Ramsey and colleagues.* The form, given to10 peers, provides reliable estimates of two overall dimensions ofperformance: cognitive and clinical skills, and professionalism.

Cognitive and clinical skillsx Medical knowledgex Ambulatory carex Management of complex problemsx Management of hospital inpatientsx Problem solvingx Overall clinical competence

Professionalismx Respectx Integrityx Psychosocial aspects

of illnessx Compassionx Responsibility

*Ramsey PG et al. Use of peer ratings to evaluate physician performance. JAMA1993;269:1655-60

Patient rating form*The form is given to 25 patients and gives a reliable estimate of adoctor’s communication skills. Scores are on a five point scale—poorto excellent—and are related to validity measures. The patients mustbe balanced in terms of age, sex, and health status. Typical questionsare:x Tells you everything?x Greets you warmly?x Treats you as if you’re on the same level?x Lets you tell your story?x Shows interest in you as a person?x Warns you what is coming during the physical examination?x Discusses options?x Explains what you need to know?x Uses words you can understand?

*Webster GD. Final report of the patient satisfaction questionnaire study. AmericanBoard of Internal Medicine, 1989

Processof care

Portfolio

Patientoutcomes

Practicevolume

Clinicalrecords

Administrationdatabase

Diary Observation

How portfolios are compiled

The photograph of a surgical team is from Philippe Plailly/Eurelios/SPL;the photographs illustrating case mix are from Photofusion, by David Tothill(left) and Pete Addis (right); the photograph of the foot is from Ray Clarke(FRPS) and Mervyn Goff (FRPS/SPL); and the medical records photographis from Michael Donne/SPL.

Further readingx McKinley RK, Fraser RC, Baker R. Model for directly assessing and

improving competence and performance in revalidation ofclinicians. BMJ 2001;322:712-5.

x Miller GE. The assessment of clinical skills/competence/performance. Acad Med 1990:S63-7.

x Norcini JJ. Recertification in the United States. BMJ1999;319:1183-5.

x Cunningham JPW, Hanna E, Turnbull J, Kaigas T, Norman GR.Defensible assessment of the competency of the practicingphysician. Acad Med 1997;72:9-12.

ABC of Learning and Teaching in Medicine

38

12 Educational environmentLinda Hutchinson

A student might find a particular question threatening andintimidating in one context yet stimulating and challenging in adifferent context. What makes one learning context unpleasantand another pleasant?

Learning depends on several factors, but a crucial step is theengagement of the learner. This is affected by their motivationand perception of relevance. These, in turn, can be affected bylearners’ previous experiences and preferred learning styles andby the context and environment in which the learning is takingplace. In adult learning theories, teaching is as much aboutsetting the context or climate for learning as it is aboutimparting knowledge or sharing expertise.

MotivationMotivation can be intrinsic (from the student) and extrinsic(from external factors). Assessments are usually a strongextrinsic motivator for learners. Individual learners’ intrinsicmotivation can be affected by previous experiences, by theirdesire to achieve, and the relevance of the learning to theirfuture.

A teacher’s role in motivation should not beunderestimated. Enthusiasm for the subject, interest in thestudents’ experiences, and clear direction (among other things)all help to keep students’ attention and improve assimilation ofinformation and understanding.

Even with good intrinsic motivation, however, externalfactors can demotivate and disillusion. Distractions, unhelpfulattitudes of teachers, and physical discomfort will promptlearners to disengage. Maslow described a model to illustratethe building blocks of motivation. Each layer needs to be inplace before the pinnacle of “self actualisation” is reached.

Physiological needsAlthough the need to be fed, watered, and comfortable seemstrite, many teachers will have experienced, for example, thedifficulties of running sessions in cold or overheated rooms, inlong sessions without refreshments, in noisy rooms, in facilitieswith uncomfortable seating.

Physical factors can make it difficult for learners andteachers to relax and pay attention. Ensuring adequate breaksand being mindful of the physical environment are part of theteacher’s role.

SafetyA teacher should aim to provide an environment in whichlearners feel safe to experiment, voice their concerns, identifytheir lack of knowledge, and stretch their limits. Safety can becompromised, for example, through humiliation, harassment,and threat of forced disclosure of personal details.

Teachers can create an atmosphere of respect by endorsingthe learners’ level of knowledge and gaps in knowledge asessential triggers to learning rather than reasons for ridicule.

Remembering names and involving the learners in settingground rules are other examples of building mutual trust.Feedback on performance, a vital part of teaching, should bedone constructively and with respect for the learner.

Case history: safe environmentDr Holden claims to use interactive teaching techniques. Sheintroduces the topic then points to a student in the audience and says,“tell me five causes of cyanosis.” Each student will get asked questionsin the session, and most spend their time worrying about when it istheir turn to be “exposed.”

After a course on teaching skills, she runs the same session. After theintroduction, she tells the students to turn to their neighbour andtogether come up with some possible causes of cyanosis. After acouple of minutes of this “buzz group” activity, she asks for onesuggestion from each group until no new suggestions are made. Thesuggestions are then discussed with reference to the aims of thesession.

Educational environment

Learning

• Motivation• Perceived relevance• Perception of task

Course/curriculum

• Curriculum style• Teaching quality• Signposting and clarity of process, outcomes, assessment• Support mechanisms

Student

• Previous experience• Learning style

Individual teachers,supervisors, facilitators

• Teaching style/techniques• Enthusiasm• Maximising physical environment• Role modelling

Many factors influence learning

Self actualisation

Self esteem

Belonging

Safety

Physiological needs

Maslow’s hierarchy of needs for motivating learning. Adapted from MaslowA H. Motivation and Personality, New York: Harper and Row, 1954

39

BelongingMany factors help to give a student a sense of belonging in agroup or team—for example, being a respected member, havingone’s voice heard and attended to, being given a useful role, andhaving colleagues with similar backgrounds, experiences, andgoals.

Learners are motivated through inclusion and consultation.Their input to a course’s objectives and structure should besought, valued, and acted on. On clinical placements, staffshould help to prevent medical students from feeling ignored,marginalised, or “in the way.” Students should instead be valuedas assets to a clinical unit or team.

Self esteemSeveral of the points mentioned above feed directly into selfesteem through making the learner feel valued. Praise, words ofappreciation, and constructive rather than destructive criticismare important. It can take many positive moments to build selfesteem, but just one unkind and thoughtless comment todestroy it.

Doctors are well used to their role in the doctor-patientrelationship. Some find it hard to translate the same skills andattitudes to the teacher-student relationship. Their ownexperience of education or their own distractions, timepressures, and other stresses may be factors.

Self actualisationIf a teacher has attended to the above motivational factors, thenthey have sought to provide the ideal environment in which alearner can flourish.

An ethos that encourages intrinsic motivation withoutanxiety is conducive to a “deep” learning approach. However,there may be some who remain unable to respond to theeducation on offer. Teachers may need to consider whether thecourse (or that particular piece of study) is suitable for thatstudent.

RelevanceThe relevance of learning is closely linked to motivation:relevance for immediate needs, for future work, of getting acertificate or degree regardless of content. Learning forlearning’s sake is back in vogue in higher education after amove towards vocational or industrial preparation.

Certain courses in medical degrees have been notoriouslypoorly received by students. Faculty members need to explainto students why these courses are necessary and how they linkto future practice. Allowing them to see for themselves, throughearly clinical exposure and experience, is likely to be helpful.Similarly, learning the basic medical sciences in the context ofclinical situations is the basis for problem based learning.

A challenging problem is the trainee who is in a postbecause he or she needs to do it for certification, although it isof no perceived value to the trainee’s future career direction. Abalance needs to be negotiated between respect for theindividual’s needs and the expectation of a level of professionalconduct.

Teacher as role modelThe teacher or facilitator is one of the most powerful variablesin the educational environment. The teacher’s actions, attitudes(as evidenced by tone of voice, comments made), enthusiasm,and interest in the subject will affect learners indirectly. Thecapacity for subliminal messages is enormous. Inappropriate

Case history: sense of belongingFive medical students arrive at a distant hospital for a four weekattachment. They are met by a staff member, shown around the unit,canteen, library, and accommodation. They are given name badges inthe style of the existing staff, and after a couple of days are givenspecific roles on the unit. These roles develop over the weeks. There islittle set teaching time, but the students feel free to ask any staffmember for more details at quieter times.

After the attachment, they meet up with friends who were placedelsewhere. Their experience was different. No one was expecting themwhen they arrived, and ward and clinic staff were unhelpful. There wasa set teaching programme and an enthusiastic teacher, but thestudents were relieved that the hospital was near a town centre withgood shops and nightlife.

Case history: self esteemA senior house officer (SHO) is making slow but steady progress. Hisconfidence is growing and the level of supervision he requires islessening. On one occasion, however, his case management is lessthan ideal, although the patient is not harmed or inconvenienced. Theconsultant feels exasperated and tells the SHO that everyone iscarrying him and it still isn’t working. The SHO subsequently revertsto seeking advice and permission for all decision making.

Students’ perception of the relevance ofwhat they are being taught is a vitalmotivator for learning

If a teacher is asked to do a one-off session with learnersthey don’t know, he or she should prepare—both beforeand at the start of the session—by determining what thelearners know, want to know, and expect to learn. Thisinvolves and shows respect for the learners andencourages them to invest in the session

Case history: role modelsDr Jones is a well known “character” in the hospital. Medical studentssitting in his clinic hear him talk disparagingly about nurses, patients,and the new fangled political correctness about getting informedconsent that wastes doctors’ time. Most students are appalled; but afew find him engaging—they view him as a “real” doctor, unlike theethics or communication skills lecturers.

ABC of Learning and Teaching in Medicine

40

behaviour or expression by a staff member will be noticed; atworst the learners will want to emulate that behaviour, at bestthey will have been given tacit permission to do so.

Maximising educational environmentClassroom, tutorials, seminars, lecturesRoom temperature, comfort of seating, background noise, andvisual distractions are all factors of the environment that canaffect concentration and motivation. Some are within theteacher’s control, others not.

Respect for the learners and their needs, praise,encouragement of participation can all lead to a positivelearning experience. Lack of threat to personal integrity andself esteem is essential, although challenges can be rewardingand enjoyable.

Small group teaching facilitates individual feedback, but theseating arrangement used will have an important effect onstudent participation. If, for example, students sit in traditionalclassroom rows, those on the edges will feel excluded. A circularformat encourages interaction. It allows the teacher to sitalongside a talkative person, thus keeping them out of eyecontact and reducing their input. A quiet student can be placedopposite to encourage participation through non-verbal means.Students can also work in unfacilitated groups on a topic,enabling them to work in teams and share the learning tasks.

Clinical settingsIn real life settings, the dual role of teacher and clinician can becomplicated. The students will be closely observing theclinician, picking up hidden messages about clinical practice.They need to feel that there is no danger that they willunnecessarily distress or harm patients or their families. Theyalso need to feel safe from humiliation. Making them feelwelcomed and of value when they arrive at a new placement orpost will aid their learning throughout.

Course and curriculum designThe designers of short and long courses should consider therelevance of the learning environment to the potential learners.Student representation on curriculum committees is one meansof ensuring a more student centred course.

The aims, objectives, and assessments should be signpostedwell in advance of a course and should be demonstrably fair.The teaching methods should build on learners’ experience,creating a collaborative environment. Disseminating thefindings of course evaluations, followed by staff training, helpsto identify and correct undesirable behaviour among facultymembers. Evaluations should also include a means forreviewing the course’s aims and objectives with the students.

In longer courses, student support systems and informalactivities that build collective identity must be considered.Students who are having difficulties need to be identified earlyand given additional support.

It is easy to “learn” attitudes—including poor attitudes.Attitudes are learnt through observation of those inrelative power or seniority. Teachers must therefore beaware of providing good role modelling in the presenceof students

Checklist to ensure good physical environmentx Is the room the right size?x Is the temperature comfortable?x Are there distractions (noise, visual distractions inside or outside)?x Is the seating adequate, and how should it be arranged?x Does the audiovisual equipment work?

Checklist for teaching in clinical settingsx Have patients and families given consent for students to be present?x Do the staff know that teaching is planned and understand what

their roles will be?x Is there adequate space for all participants?x How much time is available for teaching?x How may the students be made to feel useful (for example,

“pre-clerking” and presenting)?

Further readingx Newble D, Cannon R. A handbook for medical teachers. 3rd ed.

London: Kluwer Academic, 1994.x Eraut M. Developing professional knowledge and competence. London:

Falmer, 1994.x Welsh I, Swann C. Partners in learning: a guide to support and

assessment in nurse education. Abingdon: Radcliffe, 2002.x Norman GR, Schmidt HG. The psychological basis of

problem-based learning: a review of the evidence. Acad Med1992;67:557-65.

x Dent JA, Harden RM. A practical guide for medical teachers. London:Churchill Livingstone, 2001.

Traditional teaching canleave some studentsexcluded (that is, outsidethe “triangle ofinfluence”)

Educational environment

41

Teacher

13 Web based learningJudy McKimm, Carol Jollie, Peter Cantillon

Many of us use the internet or the “web” (world wide web) asa source of information. In medical education, the web isincreasingly used both as a learning tool to support formalprogrammes and as a means of delivering online learningprogrammes. What can educators do to ensure that thepotential of the web is used effectively to support both theirown learning and that of their students?

The technologyMuch of the literature on web based learning shows that one ofthe main barriers to the effective use of teaching materials is thetechnology (for example, poor access, slow downloading) ratherthan the design of the learning materials themselves. Some ofthese issues are discussed later in the article, but it is vital thatteachers take on expert help with technical issues in theplanning, design, and delivery of web based learningprogrammes. Through programming and the use of “plug-ins”(programs that can be downloaded from the internet),designers can produce interactive course materials containingonline activities (such as self assessments), animations, andsimulations. These can improve learning and are often moreenjoyable and meaningful for learners.

Distance learningTwo of the main developments in web based learning have beenthe adaptation of communication technology to supportlearning and the changes in distance learning strategiesnecessary for delivering online courses. Both aspects should beconsidered when designing or delivering web based learningprogrammes. Lessons can be learned by considering howdistance education evolved.

Distance and open learning began with correspondencecourses. The Open University in Britain is one of the bestknown examples of how university level education becameaccessible, through effective distance learning, to people whohad neither the traditional qualifications nor the time to enterfull time higher education.

The secret of the Open University’s success lies in clearlyidentifying students’ needs; providing effective, local support;and combining conventionally taught components with the useof up to date multimedia resources, including books, courseguides, videotapes, audiotapes, television, e-conferencing, anddiscussion groups.

What is web based learning?Web based learning is often called online learning or e-learningbecause it includes online course content. Discussion forums viaemail, videoconferencing, and live lectures (videostreaming) areall possible through the web. Web based courses may alsoprovide static pages such as printed course materials.

One of the values of using the web to access coursematerials is that web pages may contain hyperlinks to otherparts of the web, thus enabling access to a vast amount of webbased information.

A “virtual” learning environment (VLE) or managed learningenvironment (MLE) is an all in one teaching and learningsoftware package. A VLE typically combines functions such as

GlossaryE-conferencing—Use of online presentations and discussion forums (in

real time or stored as downloadable files on a website) to avoid theneed for participants to travel

E-learning—Learning through electronic means, such as via the web(see world wide web), an intranet, or other multimedia materials

HTML (hypertext markup language)—The language used to create webpages. HTML files can also contain links to other types of filesincluding wordprocessed files, spreadsheets, presentation slides,and other web pages

Hyperlinks—Links in web pages that enable the user to access anotherweb page (either on the same or a different site) with just onemouse click

Internet—A global network of computers divided into subsets (forexample, the web or email systems). Computers are linked to theinternet via host computers, which link to other computers via dialup (for example, via a modem) and network connections

Internet service provider (ISP)—Home users usually access the internetthrough an internet service provider (such as AOL), whichmaintains a network of PCs permanently connected to the internet

Intranet—A network of computers that share information, usuallywithin an organisation. Access normally requires a password and islimited to a defined range of users

Managed learning environment (MLE)—Usually has an integratedfunction, providing administrative tools, such as student records,and linking with other management information systems (MLS)

Search engines (such as Lycos, Google)—Can be used to help to findinformation

Videostreaming—The process by which video images are able to bestored and downloaded on the web. These might be in real time(such as a conference) or used asynchronously

Virtual learning environment (VLE)—A set of electronic teaching andlearning tools. Principal components include systems that can mapa curriculum, track student activity, and provide online studentsupport and electronic communication

World wide web (web)—Use of the internet to present various types ofinformation. Websites or home pages may be accessed with the aidof a browser program (such as Netscape Communicator orMicrosoft Explorer). All such programmes use HTML

For additional information see www.learnthenet.com/english/section/intbas.html

42

discussion boards, chat rooms, online assessment, tracking ofstudents’ use of the web, and course administration. VLEs act asany other learning environment in that they distributeinformation to learners. VLEs can, for example, enable learnersto collaborate on projects and share information. However, thefocus of web based courses must always be on thelearner—technology is not the issue, nor necessarily the answer.

Models of web based learningSeveral approaches can be used to develop and deliver webbased learning. These can be viewed as a continuum. At oneend is “pure” distance learning (in which course material,assessment, and support is all delivered online, with no face toface contact between students and teachers). At the other endis an organisational intranet, which replicates printed coursematerials online to support what is essentially a traditional faceto face course. However, websites that are just repositories ofknowledge, without links to learning, communication, andassessment activities, are not learner centred and cannot beconsidered true web based learning courses.

In reality, most web based learning courses are a mixtureof static and interactive materials, and most ensure that someindividual face to face teaching for students is a key feature ofthe programme.

The individual learnerThe first step in designing a web based course is to identify thelearners’ needs and whether the learners are to be consideredas part of a group or as individual learners. The web can be auseful tool for bringing isolated learners together in “virtual”groups—for example, through a discussion forum. There areseveral online resources on how to design web based learningprogrammes (for example, at www.ltsn.ac.uk).

Incorporating web based learninginto conventional programmesWeb based learning in an institution is often integrated withconventional, face to face teaching. This is normally done via anintranet, which is usually “password protected” and accessibleonly to registered users. Thus it is possible to protect theintellectual property of online material and to supportconfidential exchange of communication between students.

Medicine has many examples of online learning, in both thebasic sciences and clinical teaching. As students are usually inlarge groups for basic science teaching, web based learning can

“Newer technologies such as computersand video conferencing are notnecessarily better (or worse) for teachingor learning than older technologies . . .they are just different . . . The choice oftechnology should be driven by the needsof the learners and the context in whichwe are working, not by its novelty.”

Bates AW. Technology, open learning anddistance education. London: Routledge,1995

Features of a typical web based coursex Course information, notice board, timetablex Curriculum mapx Teaching materials such as slides, handouts, articlesx Communication via email and discussion boardsx Formative and summative assessmentsx Student management tools (records, statistics, student tracking)x Links to useful internal and external websites—for example, library,

online databases, and journals

Learning and teaching support network (www.ltsn.ac.uk)

x The learning and teaching support network (consisting of 24subject centres) was set up to help staff in higher education todeliver programmes in their own subjects more effectively and toimprove communication between teachers in differentorganisations and government agencies

x One subject centre covers medicine, dentistry, and veterinaryscience (LTSN 01)

x Although not strictly web based learning, LTSN 01 uses acombination of activities delivered via the web (such as an emaildiscussion forum, copies of useful articles, research and fundingprogrammes, and job vacancies) to support staff working inmedical, dental, and veterinary education

Design of curriculum (taught course,directed learning, self learning)

Pedagogy(delivery of teaching)

Assessment(of student learning)

Evaluation(of learning process)

Feedback and modificationto curriculum

The learning cycle: useful to bear in mind when planning a web basedcourse

Questions to ask before starting a web based learning projectx What is the educational purpose of the web based learning project?x What added value will online learning bring to the course or to the

students?x What resources and expertise on web based learning exist in the

institution?x Are colleagues and the institution aware of the planned course?

(You need to avoid duplication of effort and be sure that theinstitution’s computer system can support the course)

x Has the project taken account of existing teaching resources andongoing maintenance costs after initial development?

x Have you allowed enough time to develop or redevelop materials?x Have the particular design and student support requirements of

web based learning courses been taken into account? If not, thee-learning starter guides on the LTSN website are a good resource(www.ltsn.ac.uk/genericcentre)

Web based learning

43

be used to provide learning materials to complementconventional programmes and to enable self assessment—forexample, access to anatomical sites and image banks for theteaching of pathology courses. Web based learning can beuseful to support clinical teaching when learners aregeographically dispersed—for example, to learn clinical skillsthrough video demonstrations.

AssessmentWith all types of learning, including web based learning, it isuseful for students to receive constructive, timely, and relevantfeedback on their progress. Online assessment is sometimesconstrained by the medium in which it is operating. Computermarked assessments alone are not appropriate for marking orgiving feedback on assignments such as essays or projects thatrequire more than the mere reproduction of knowledge.

When planning online assessment it is important todetermine what is to be assessed. If knowledge reproduction isbeing tested, objective questions (such as multiple choice or“true or false” questions) with instant or model answers canprovide excellent feedback. Assessment of higher cognitivefunctions, such as analysis and synthesis, will require morecomplex tests. Automated marking may be difficult for suchassessments, and the teacher is likely to have to do a substantialamount of work before he can add his or her comments to thestudent’s record. Further guidance on how to design web basedassessments for online courses can be found at www.ltsn.ac.ukand www.ltss.bris.ac.uk

For and against web based learningWhen designing web based programmes (as with any learningprogramme), the learners’ needs and experience must be takeninto account. Appropriate technology and reasonable computerskills are needed to get the best out of web based or onlinelearning. Programmes and web pages can be designed toaccommodate different technical specifications and versions ofsoftware. It is frustrating for learners, however, if they are tryingto work on the internet with slow access or cannot downloadimages and videos they need. On the other hand, web basedprogrammes may, for example, encourage more independentand active learning and are often an efficient means ofdelivering course materials.

Effective web teaching and learningCourse designers need to remember that younger students aremore likely to be familiar with using the internet than olderlearners, who may feel less comfortable with a web basedcourse. To get the best out of their learning experience, learnersneed basic computer skills, support, and guidance.

Teachers must design their courses to encourage effectiveweb based learning rather than aimless “surfing.” Programmedesign should therefore filter out poor information as well assignpost key information sources.

Many clinicians are beginning to use electronic patientrecords. This change means that doctors are becoming moreadept at using computers and online resources to support theirdaily work and continuing professional development.Electronic media can facilitate access to evidence based

With web based learning, the material can be linked tolibraries (for example, for ordering books or journals),online databases, and electronic journals. These functionsare particularly useful for research and clinical activities

Advantages and disadvantages of online assessmentAdvantagesx Students can receive quick feedback on their performancex Useful for self assessments—for example, multiple choice questionsx A convenient way for students to submit assessment from remote

sitesx Computer marking is an efficient use of staff time

Disadvantagesx Most online assessment is limited to objective questionsx Security can be an issuex Difficult to authenticate students’ workx Computer marked assessments tend to be knowledge based and

measure surface learning

Advantages and disadvantages of web based learningAdvantagesx Ability to link resources in many different formatsx Can be an efficient way of delivering course materialsx Resources can be made available from any location and at any timex Potential for widening access—for example, to part time, mature, or

work based studentsx Can encourage more independent and active learningx Can provide a useful source of supplementary materials to

conventional programmes

Disadvantagesx Access to appropriate computer equipment can be a problem for

studentsx Learners find it frustrating if they cannot access graphics, images,

and video clips because of poor equipmentx The necessary infrastructure must be available and affordablex Information can vary in quality and accuracy, so guidance and

signposting is neededx Students can feel isolated

ABC of Learning and Teaching in Medicine

44

resources such as the Cochrane Library. These web basedclinical support sites are excellent resources for postgraduate“on the job” learning.

Teachers should be encouraged, through training andsupport, to use the web and other informationtechnologysystems in their teaching. They need examples andawareness of good practice, and standards should be set inrelation to how teachers present information and manage thelearning environment.

ConclusionWeb based learning offers huge opportunities for learning andaccess to a vast amount of knowledge and information. The roleof teachers is to ensure that the learning environment providedtakes account of learners’ needs and ensures that they areeffectively prepared and supported. Online learning hasadvantages, but web based learning should not always be viewedas the method of choice because barriers (such as inadequateequipment) can easily detract from student learning. Thetechnology must therefore be applied appropriately and notused simply because it is available and new or because studentsand teachers have particular expectations of this means ofcourse delivery.

Further readingx Cook J. The role of virtual learning environments in UK medical

education. LTSN Bioscience Bulletin 2002;5. http://bio.ltsn.ac.ukx Forsyth I. Teaching and learning materials and the internet. 3rd ed.

London: Kogan Page, 2001.x Jolliffe A, Ritter J, Stevens D. The online learning handbook: developing

and using web based learning. London: Kogan Page, 2001.x World Federation for Medical Education. www.sund.ku.dk/WFME/

WFME%20Guidelines/guidelines99.html (paper on usinginformation technology in education, including web based learning)

Web based learning

45

14 Creating teaching materialsRichard Farrow

The nature and qualities of the teaching materials that you usecan have a substantial effect on the educational experience ofyour students. Teaching materials can often distract learnersrather than help them to learn. Common avoidable problemsinclude overcrowded or illegible slides, irrelevant or badlyprepared handouts, and incompatible multimedia equipment. Itis important therefore to know how to create effective teachingmaterials.

Ground rulesFive basic principles apply to preparing teaching materials,irrespective of the type of material you choose: links,intelligibility, general style, highlighting, and targeting (LIGHT).You may sometimes decide to ignore one or more of theseprinciples, but if you do, think carefully about what you aretrying to achieve.

LinksYour teaching materials should have obvious and direct links toyour talk, discussion, or presentation. Handouts are the mainoffenders in this category, and it is not unusual for handouts tohave little in common with the talk. It is quite acceptable for theteaching materials to give some additional information, but thisshould not be excessive.

IntelligibilityThe teaching material should be easy to understand and learnfrom. How this is achieved will depend on the medium usedand the venue of the talk or presentation. Use simple languageand avoid overlong sentences or statements. Diagrams can helpto clarify a complex message. If you are using slides or overheadtransparencies, the size of the print needs to be large enough tobe read from the back of the auditorium. The font selectedshould be sans serif (for example, Arial).

General styleYou should aim to use a consistent style throughout yourteaching materials, particularly if you are giving a series of talks.Although it is tempting to use a variety of novel styles,consistency will allow learners to concentrate on the meaningand relevance of what you are trying to communicate.

HighlightingHighlighted information helps to emphasise important issuesor pivotal points in a developing argument. Methods ofhighlighting include changing the colour of text or underliningwords or phrases. This also applies to videotapes andaudiotapes, where changing your tone of voice can be used toemphasise key points.

TargetingIt is important that both the type of educational event (forexample, presentation, seminar, discussion) and the teachingmaterials that supplement it are targeted at what your studentsneed to learn. Targeting therefore requires an awareness ofwhat knowledge and skills your students already have. This can

Preparing overhead transparenciesDox Try to use typed rather than handwritten scriptx Use a type size that is big enough to be read by the whole

audience—for example, at least 20 pointsx Make sure that the colour of your text works—for example, dark

print on a pale backgroundx Limit each transparency to one idea or concept

Don’tx Use small printx Use overhead transparencies packed with tables and figuresx Use light colours

Uniformity in the teaching materials willhelp learners to focus on content ratherthan style

It is easy to overdo highlighting by emphasising virtually every point that

you make. This reduces the usefulness of the technique and hides the

really pivotal shifts in a morass of highlighted text.

Target your talk at learners’ needs—don’tjust pull out the slides or overheads froma previous talk

46

be difficult to judge, but it is worth spending time finding outabout your expected audience. It becomes easier if you aredoing a series of talks with the same group as you can getfeedback from the learners to help you plan more effectively.

Types of teaching materialsBlack, green, or white boardsThese are ideal for brainstorming sessions and small groupwork. If you are doing the writing, try not to talk at the sametime as it is difficult for your learners to hear you if you haveyour back to them. Remember the LIGHT principles, and try toput concepts, not an essay, on the board. Make sure thateveryone has finished copying information before you rub theboard clean. Using different colours can add emphasis andhighlight your important messages.

Lecture notesEnsure that any handouts are produced to a high quality.Photocopies of handwritten notes (and frequently photocopiedelderly pages) look scrappy and tend not to be valued. Givehandouts to the learners at the beginning of the talk as copyingdown information is not a good use of their limited “face toface” time. Use headings and diagrams to make the handoutsintelligible.

Overhead projectorThe technical equipment for displaying overheadtransparencies is widely available and reliable. It is a goodbackup resource, and for critical presentations it is comfortingto know that, if all else fails, you have transparencies in yourbag. Presentations using an overhead projector have theadvantage that they allow you to face your audience whilepointing out features on the transparency.

Correct preparation following the LIGHT principles is vital.Ensure that the transparencies will fit the projector—most willdisplay A4 size, but some are smaller, so check in advance. Theabsolute minimum height for text on transparencies is 5 mm,although using larger text and fewer words usually produces amore effective educational tool. A good rule of thumb is to usea type size of at least 20 points. Several simple transparenciesare usually better than one complicated one.

It is fairly straightforward to design your transparency on acomputer then print it using a colour printer. Avoid usingyellow, orange, and red, as these colours are difficult to see.Insead, use dark text on a light background. You can write anddraw directly on to the transparencies with felt tipped pens. Usepermanent markers to avoid smudging, and place a sheet ofruled paper underneath so that the writing is evenly spaced.You can also use a photocopier to copy print on to atransparency, but remember that you may need to enlarge it tomake the text readable.

If you are likely to use a transparency again it is worthstoring it carefully in dust free covers. One commonly usedmethod is to store transparencies in clear plastic sleeves thatcan be filed in a ring binder. When showing transparencies, donot overuse the technique of covering the transparency andrevealing a little at a time—many learners find this irritating.

35 mm slidesThe need for 35 mm slides has decreased substantially with theadvent of computer programs such as Microsoft’s PowerPoint.However, multimedia projectors and computers are expensiveand not available in all locations, whereas most educationalinstitutions have a slide projector. Making your own slides canbe difficult, so get help from the local illustration department or

Types and uses of teaching materialsBoards, flip charts—Small groups, problem based learning tutorials,

workshopsLecture notes—Small and large groups; help to improve interactivityOverhead projector—Small and large groups, workshops, and interactive

sessions35 mm slides and PowerPoint—Generally large groups and lecture

formatsVideos—Good for clinical teaching in larger groups (use film of

patients); also for teaching communication skills and practicalskills (students can keep films for self appraisal)

Life and plastic models—Anatomy teaching in small groups or for selfdirected learning

Computer assisted learning packages—Small groups with a tutor; largegroups in computer laboratories; self directed learning

Skills centres and simulators—Small groups learning clinical skills

Leave spaces in the handout for your learners to recordthe results of interactive parts of your talk—this ensuresthat the handout the learners take away has more valuethan the one they were given. Also, leave spaces forexercises to be completed later, thus linking self directedlearning with face to face learning

Paper copies of transparencies and slidescan make useful handouts—your learnerscan then add clarifying statements ordiagrams to their own copy of thepresentation

Number your slides so that if aprojectionist is loading them or thecarousel is dropped they can be quicklyreordered

Creating teaching materials

47

a commercial company. Ensure that the text is large enough tosee when projected and that the slides are marked so that theyare loaded in the projector correctly. Dual projection is rarelydone well and rarely necessary unless you are using visualimages (for example, x ray films, clinical photographs) withaccompanying text. If you use dual projection make sure thateach of the slides is labelled for the correct projector.

Computer generated slidesThe ability to make computer generated slides (for example,PowerPoint) has transformed the way that many people createteaching materials and has greatly reduced the use of 35 mmslides. Try not to get seduced by the technology, however, andremember that it is just another educational tool. Having triedall of the colours and slide layouts available, many experiencedlecturers now prefer simple formats that are easy to read and inwhich the medium does not get in the way of the message.

However, the computer package has many usefultools—diagrams and “clip art” can help to conceptualise difficultproblems. Video clips can be inserted into a presentation, butbe certain that they are there to illustrate a point and not simplyto show off your own technological skills. Use advanced formatsfor PowerPoint presentations only if you are well practised andcomfortable with the medium.

Ensure that the computer you are planning to use iscompatible with the multimedia projector. Similarly, if you havestored your presentation on a CD or floppy disk (or any one ofthe other portable storage formats), make sure that this issupported at the venue. The latest version of the presentationsoftware can give you access to many features that may notwork on the computer provided at the teaching venue, so a wiseprecaution is to save your presentation as an older version ofthe software.

Ground rules for slide preparation (35 mm or PowerPoint)x Use a clear font that is easily readablex Use a type size of 20 points or greaterx Use a light text on a dark background for slides (in contrast with

OHP transparencies)x Use short sentences and small tablesx Restrict the overall number of words on each slide to about 40 or

fewerx Avoid patterned backgrounds—they are extremely distractingx Limit the number of colours on your slides to a maximum of threex Use highlighting to emphasise items in listsx Use animation and sound effects sparingly

Further readingx Cannon R, Newble D. A handbook for teachers in universities and

colleges. 4th ed. London: Kogan Page, 1999.x Newble DI, Cannon R. A handbook for medical teachers. 4th ed.

Dordrecht, Netherlands: Kluwer Academic, 2001.x Kemp JE, Dayton DK. Planning and producing instructional media. 5th

ed. New York: Harper and Row, 1985.x Hartley J. Designing instructional text. 3rd ed. London: Kogan Page,

1994.

ABC of Learning and Teaching in Medicine

48

49

maps 7outcome evaluation 14outcome measures 12planning 12–13

data collection 37–8databases, administrative 38decision making processes 23descriptive models of curriculum 5, 6–7diaries 38discussion leading 19distance learning 42

educational environment 39–41educational theory 1–4

conversion into practice 3–4essays 30ethical values/beliefs 31evaluation 12–14

curriculum planning 12–13cycle 12, 14instruments 13learners 13, 14, 24lectures 17–18questions 29

evidence based resources 44–5examiner training in structured assessment 34experiential learning theory 26explanations 26, 27

facilitation 10, 19facilitators 40–1failure 2feedback 11, 13

to learners 23lectures 17–18openness to 24web based learning 44

fishbowls 20, 21

green boards 47group discussion 19group management 21group round 20group teaching

large groups 15–18leading 18, 20–1processes 19–21small groups 8, 19–21, 41, 47structures 19–21

highlighting of teaching materials 46horseshoe groups 21“hot seating” 28

adult learning theory 1andragogy 1Angoff procedure, modified 35anonymity in evaluation 13anxiety, re-interpretation 2artistry 3assessment

brief 17formative 10structured 34student 14summative 10web based learning 44

attention of students 16attitudes 8, 9, 12, 41

Bandura, Albert 2behaviour of students 12behavioural objectives 6behavioural observation 22black boards 47brainstorming 17, 47buzz groups 17, 20, 39

Candy, Philip 1care, process of 37case studies, ethics issues 3clinical competence 37–8

assessment 34, 36clinical environment 25–8clinical interface with patient 23clinical material 8clinical practice records 37–8clinical reasoning 23clinical skills assessment 32–5clip art 48Cochrane Library 45cognitive learning theories 26communication

effective teaching 26proactive 21skill rating 38skill testing 23small group teaching 19teacher-student 27

competence 2, 13constructivism 2consultations, students in 28cost effectiveness of question

types 29course design 41crossover groups 20, 21curriculum

design 5–7, 8, 9, 41

Index

Index

50

individual’s needs 40information processing 26intelligibility of teaching materials 46intent, statements of 6, 7interactive lectures 3interview evaluation 13

key cases approach 8knowledge

acquisition 8clinical teaching 26cognitive learning theory 26lecture structure 15outcomes 9, 12retention 10

knowledge based tests 35Knowles, Malcolm 1

learners see studentslearning

active 24belonging 40cycle 43effective 18experiences 2, 3–4gaps 17goal development 3–4ground rules 23matching experience to learner 22physiological needs 39plans 23process 26relevance 39, 40safety 39stimulus 8tasks 1

learning and teaching support network 43learning approach, deep 40lectures/lecturing 15–18

notes 47lifelong learning 21long case 32, 34

Maastricht “seven jump” process 8managed learning environment 43Maslow’s hierarchy of needs 39mastery, zones of 2Miller, George 36mini-assessments 17motivation for learning 39–40

nervousness, re-interpretation 2networking 21

objective structured clinical examinations (OSCEs) 32–4, 36checklist 33, 34stations 32, 33, 34

objective structured long case (OSLER) 32objectives model of curriculum 5–6“one-minute” paper 17“one-minute preceptor” model 27one to one teaching 22–4online assessment 44openness 21oral examinations 34organisational change 21

Osler, William 28outcomes

evaluation 12, 14judgements of patients 36–7learning 9, 14measures 12significant and enduring 6

outcomes based education 6overhead projector 47ownership 13

participation, active in groups 19patients

candidate interactions 34electronic records 44learner relationship 24outcome assessment 36–7rating doctor communication skills 38role in teaching 28selection for teaching on wards 27

peer evaluation rating 38personal change 21personal integrity 41portfolios 38postgraduates, clinical teaching 25PowerPoint (Microsoft) 47, 48Practical Assessment of Clinical Examination Skills

(PACES) 32prescriptive models of curriculum 5–6primary care consultations 22problem based learning 8–11

lectures 15, 16process evaluation 12professional conduct 25, 40professional development 21projection, dual 48projector, multimedia 48

questionnaire evaluation 14questions/questioning

circular 20–1clarifying 26, 27closed 23, 26, 27evaluation 29experiential learning 26–7extended matching 31key feature 30–1in lectures 16, 17multiple choice 29–30, 36, 44multiple true or false 30objective 44one to one learning 22open ended 23, 26, 27, 30safe environment 39short answer open ended 30students 16true or false 29, 44written assessment 29

recall 16, 23records 37–8

electronic 44reflection/reflective practice 2–3, 13, 23

experiential learning 26reliability

clinical skills assessment 34, 35question types 29

Index

51

research 12role models 40–1

sampling evaluation 13Schön, Donald 2self actualisation 39, 40self directed learning 1–2self efficacy 2self esteem 40simulation tests 36situational analysis 6, 7situational models of curriculum 7skill based assessment 32–5

see also objective structured clinical examinations (OSCEs)skills

generic 8outcomes 9, 12

slides, 35 mm/computer generated 47–8snowball groups 20standard setting for clinical skills 35statements of intent 6, 7students 1, 2

active participation 19attention 16communication with teachers 27consultation 28, 40engagement 39evaluation 13, 14, 24feedback 23inclusion 40individual 43interaction 18learning process 26lectures 15matching learning experience to 22name 23needs 22, 46patient relationship 24progress monitoring 23questions 16recall 16respect for 41teacher relationship 40

success 2surveys, evaluation 14Sylvius, Franciscus de la Boe 22

teacherscommunication with students 27motivation 39–40one to one 23–4relationship with students 40role models 40–1

teachingcustomising 22effective 4integrated strategies 27large groups 15–18methods 41openness to feedback 24rewards 24small groups 19–21

teaching media/materials 16creating 46–8types 47–8

teaching opportunities, exploitation 27teaching practice principles 3transparencies 47triangle of influence 41triggers 8, 9tutors in small group teaching 19Tyler, Ralph 5

understanding 23, 27

validityclinical skills assessment 34, 35long case 34question types 29

virtual learning environment 43viva 34

wards, teaching on 27web based learning 42–3white boards 47wisdom 3work based assessment 36–8written assessment 29–31

zones of mastery 2