Audience Response Systems: Transforming your Lecture

Hall into a Platform for InteractivityAlice Chuang, MD

University of North Carolina – Chapel Hill

Francis S. Nuthalapaty, MDUniversity of South Carolina - Greenville

2009 APGO Faculty Development Seminar

Objectives

• Demonstrate ARS in action*

• Discuss the rational for using Audience

Response Systems (ARS) in medical education

• Outline ARS question types and best practices

• Outline practical concerns regarding use of ARS

• Describe available ARS systems

Audience Response Systems

• Definition: Electronic tool which allows lecturer to poll learners with immediate feedback.

• Synonyms:– Classroom response system– Student response system– Personal response system

Instructions for Clickers

• To vote: Press 1/A, 2/B etc…• Vote Status Light:

– Green if your vote was recorded– Orange if not

• To change your vote – press another key• Clickers work like an RF remote• Receiver is plugged into USB port

Are you here today?

Yes N

o

0%

100%1. Yes2. No

I am a…

11%

22%

0%

11%

56% 1. Clerkship director

2. Program director

3. Chair

4. Clerkship coordinator

5. Other

What is your role in education?

Tea

chin

g stu

dents

in th

e...

Tea

chin

g stu

dents

in th

e...

Adm

inis

tratio

n (Dea

n, C...

Oth

er

0%

44%

11%

44%1. Teaching students in the pre-clinical curriculum

2. Teaching students in the clerkship

3. Administration (Dean, Clerkship Director)

4. Other

How often do you currently use ARS in your teaching?

70%

30%0%0%

Never used ARS before

Rarely (<25% of my lectures)

Often (>50% of my lectures)

Very Often (>75% of my lectures)

1. Never used ARS before

2. Rarely (<25% of my lectures)

3. Often (>50% of my lectures)

4. Very Often (>75% of my lectures)

Roles of ARS

• Restarting the attention clock– Average human attention span: 20

minutes– Recall of information decreases after

15-20 minutes– At maximum 20-25% of material

presented in standard lecture is retained

0

10

20

30

40

50

60

70

80

90

0 15 30 45 60

Time (min)

Att

enti

on p

aid

to lectu

rer

(%)

Adult Learner(i.e. tiredoverworkedmedicalstudent)

Has renewed sense of interest in topic at hand

Is finding lecture material somewhat

mundane, starts previewing following lecture for interesting

slides

Listens intently, hoping to catch key

summary points

Settling into lecture, previewing

lecture slides

Roles for ARS

• To increase interactions– To assess learner’s attitudes– To start or focus discussions– To vote after a debate or panel discussion

• To assess preparation and ensure accountability– Questions about reading or homework

Roles for ARS

• To engage learner throughout a lecture– To assess learner’s understanding and

misunderstanding of topics during a lecture– To find out more about student attitudes on a topic– To assess need for more explanation or detail

• For quizzes or tests– paying attention– taking good notes– preparing for class or labs– keeping up with homework– actively thinking

Prevalence of ARS

• Used ubiquitously in many disciplines…mathematics, philosophy, psychology, economics

• Used in many age groups…K-12, elementary, undergraduate, graduate

History

• Pradhan et al• RCT of ARS v. standard lecture format on

the topic of contraception• 17 ob/gyn residents at UMDNJ• Compared pretest/posttest (6 weeks

later) scores– ARS: 21% improvement– Standard lecture: 2% improvement– P = 0.18

History

• Nayak and Erinjeri• 27 medical students who were instructed to

give a presentation at the end of course– From learners viewpoint

• Increased self-confidence with understanding material and verbally answering questions

• No differences in end of course exam scores

– From presenters viewpoint• Made them more comfortable that they were conveying

information effectively• Prepared them for future presentations

Pros…based on the literature

• Instructor benefits– Immediate feedback on learners understanding,

identify gaps– Improved retention of knowledge

• Technical issues– Decreased error in collecting data because of

electronic transmission– Anonymous

• Can increase attendance– in one study only if participation was worth at least

10% of grade– Can reduce attrition through the duration of a

course (4% vs 8-12%)– Particularly if class was midday

Pros…based on the literature

• Increased learner satisfaction by improving interactivity and by engaging learner– Student gave instructors and “learning

environment” more positive ratings• Allows learner to compare own understanding

with others• Increases active participation

– Learner must commit to answer, thus increasing attention

– Affects active participation in other ways (asking more questions)

– Less sleeping during class– Superior to show of hands technique with increased

participation

Cons

• Need initial investment in software and hardware– Program to create questions– “clickers” or other data entry modality

• Training and buy-in from faculty• Willingness to participate by students

– Some negative responses

• Using ARS decreases time for actual delivery of material during the course of lecture

• Can be distracting if overused in a single lecture (2-5/50 minute lecture)

Question Types

• Factual Questions• Conceptual Questions• One-best-answer Questions• Opinion Questions• Data Gathering Questions• Questions Asking for Predictions• Feedback on Teaching

Question Types

Factual Questions

These questions might be used to see if students did the reading, remember

important points from prior classes, or have memorized key facts.

Vanderbilt Center for Teaching

Question Types

Conceptual Questions

Multiple-choice questions that demonstrate whether or not students understand important concepts and principles.

Vanderbilt Center for Teaching

Question Types

One-best-answer Questions

Questions which include multiple answer choices, more than one of which could be argued as correct. Students are asked to

select the one best answer from these choices. Also referred to as "partially

correct multiple-choice questions" (Burnstein & Lederman, 2006)

Vanderbilt Center for Teaching

Question Types

Opinion Questions

Evaluative and opinion questions which may not have correct answers. Asking

these questions can engage students and provoke rich discussions, particularly in

response to ethical, legal, or moral issues.

Vanderbilt Center for Teaching

Question Types

Data Gathering Questions

Used to quickly gather demographic, opinion, or other data from a class full of

students. This is often useful in social science classes that utilize social or

economic experiments in the classroom.

Vanderbilt Center for Teaching

Question Types

Questions Asking for Predictions

Students can be asked to predict the outcome of an experiment prior to being

shown the experiment. This gives the teacher a sense of the students’

preconceptions and increases the surprise value of the experiment when students see

just how many of their classmates expected different outcomes.

Vanderbilt Center for Teaching

Question Types

Feedback on Teaching

Ask students to provide direct feedback on how well a particular class is going. For example, one might ask students to rate

their level of understanding of the lecture thus far on a scale of 1 to 4.

Vanderbilt Center for Teaching

Question Best Practices

• Apply Bloom’s Taxonomy to test concepts rather than recall

• Test understanding of jargon separately from understanding of concepts to be sure source of confusion can be clearly identified

• Include “I don’t know” to ensure data truly reflects understanding rather than some sample of student guessing

Bloom’s Taxonomy

Bloom’s Taxonomy6. Evaluation: think critically about and defend a position

– Judge, Justify, Defend, Criticize, Evaluate7. Synthesis: transform, combine ideas to create

something new– Develop, Create, Propose, Design, Invent

8. Analysis: break down concepts into parts– Compare, Contrast, Distinguish

9. Application: apply comprehension to unfamiliar situations– Apply, Use, Compute, Solve, Predict

10.Comprehension: demonstrate understanding of ideas, concepts– Restate, Explain, Summarize, Interpret, Describe

11.Factual Knowledge: remember and recall factual information– Define, List, State, Name, Cite

Question Best Practices

• Use variety of types of questions• Ask about linking a general concept to a specific

example• Ask about linking a familiar idea to a new concept• Ask about a concept yet to be introduced to

increase student curiosity• Can ask question twice

– First for individuals– Second after student discussion to promote exchange

of ideas

How would you rate this workshop so far?

Fas

cinat

ing

Som

ewhat

inte

rest

ing

Borin

g

A w

aste

of t

ime

25%

0%0%

75%1. Fascinating2. Somewhat

interesting3. Boring4. A waste of time

Question Writing Exercise

35 yo G0 BMI 40 recently married with long h/o oligoovulation. EMB shows CAH. Propose next step:

Obse

rvat

ion, r

epea

t bx

in..

Rep

eat e

mb o

r D a

nd C n

ow

Ultr

asoun

d

Med

ical

mgm

t: p

roges

ti...

Gyn

Onc

consu

lt/pa

th re

...

hys

tere

ctom

y

0%

14%

0%

14%

71%

0%

1. Observation, repeat bx in 3 mos

2. Repeat emb or D and C now

3. Ultrasound4. Medical mgmt:

progestin Rx and eval for PCOS

5. Gyn Onc consult/path review

6. hysterectomy

To use for future conference planning, Palm Beach is a great location for a conference because:

0%

31%

54%

15% 1. Accessible location

2. Pleasant weather

3. Luxurious accomodations

4. Various recreational activities

UNC Experience

• Questions should be similar difficulty as those on examinations

• Students liked ARS especially if questions provided hints to examination materials

• Participation was about 60-70%, not the 100% we anticipated

Best Practices

• Planning– Be sure to have tech support– Decide how ARS will or will not affect student

grades– Define clear objectives for use of ARS in your

classroom and write questions with these objectives in mind

– First year will take a little more time and effort– Practice when first starting to use ARS

Best Practices

• Student communication– If used for grading, be sure to have rules

about use (i.e. no use of other’s clickers)• Consider throwing out some grades for

inadvertent clicking• Consider making it worth 5-10% of total grade (i.e.

low stakes, but not too low)

– Plan for time to debrief after each question

ARS Options

• Different technologies– IR transmitters– RF transmitters– Web-based– Smartphone

• Presentation software integration• LMS integration

ARS Options

Turning Point

iClickerInterwrite

Acknowledgements

• Special thanks to:

Jake AcheyInstructional Technology TechnicianUNC School of MedicineMedical Students Teaching Lab

References• 1. BARBER M, NJUS D. Clicker evolution: seeking intelligent

design. CBE Life Sci Educ 2007;6:1-8.• 2. CALDWELL JE. Clickers in the large classroom: current

research and best-practice tips. CBE Life Sci Educ 2007;6:9-20.• 3. COLLINS J. Audience response systems: technology to

engage learners. J Am Coll Radiol 2008;5:993-1000.• 4. COLLINS LJ. Livening up the classroom: using audience

response systems to promote active learning. Med Ref Serv Q 2007;26:81-8.

• 5. NAYAK L, ERINJERI JP. Audience response systems in medical student education benefit learners and presenters. Acad Radiol 2008;15:383-9.

• 6. PRADHAN A, SPARANO D, ANANTH CV. The influence of an audience response system on knowledge retention: an application to resident education. Am J Obstet Gynecol 2005;193:1827-30.

• 7. TURPIN DL. Enhance learning with an audience response system. Am J Orthod Dentofacial Orthop 2003;124:607

Session content, including narrated MS Powerpoint slides

available at:

http://www.obgynknowledgebank.net