Embed Size (px)
Citation preview
Reutilization of multiple-choice questions
PEDRO HERSKOVICDepartment of Pediatrics, East Campus, Luis Calvo Mackenna Hospital, School of Medicine,University of Chile
SUMM AR Y Multiple-choice questions (MCQs) (one-best-answer
type, with ® ve alternatives) used for assessment of ® fth-year
medical students in the Pediatr ics course are thoroughly discussed
and analysed with the students after each test. This study was
undertaken with the aim of establishing whether this kind of
exposure precludes the reutilization of questions in future years.
In all, 197 questions were used more than once from 1994 to
1998. Considering the ® rst and last time they were included in a
test, 50 (25%) increased their difficulty, 39 (20%) diminished
it, and 108 (55%) remained unchanged. The correlation coef-
® cient between the percentage of students that answered correctly
each question the ® rst and last time they were used was 0.51 (p
< 0.001). It is concluded that the kind of exposure students are
given by our course to MCQs, seems not to preclude their future
reutilization.
Introduction
Exposure to students of multiple-choice questions (MCQ),
by publication in a book (Bandaranayake et al., 1990), or by
reutilization with the same group of students (Rees, 1986;
Miller et al., 1993), leads from modest to marked improve-
ment in performance when the students are confronted for
a second time with the same questions.
Since 1994 the MCQs (one-best-answer type, with ® ve
alternatives) that are part of the assessment of ® fth-year
medical students in the Pediatrics course given on campus
at the University of Chile are discussed in depth with the
students, shortly after the tests.The purposes of this are: to
obtain feedback from students, to assure the fairness of the
assessments, and to help them with their learning process
(Herskovic & Herskovic, 1998). The students are not given
copies of the questions to keep.
During the ensuing 5 years we have reused many MCQs
in different years, so we wanted to establish whether the
kind of exposure our questions receive is damaging them as
assessment tools.
Methods
The Pediatrics MCQ pool was reviewed to detect those
questions that had been used more than once, in different
years, from 1994 to 1998. The percentage of students that
answered the item correctly, (pc value) was registered for
the ® rst and last time each particular question was used.
Results
In all, 197 questions were used more than once in different
years. Seventy-two were used twice, 67 three times, 48 four
times and 10 ® ve times.
Table 1 gives the pc values arranged in increments of 20
for the ® rst and last time every question was used. The pc
value of 108 questions (55%) remained unchanged, for 50
questions (25%) decreased, and for 39 items (20%)
increased.
The correlation coefficient between the pc values for
both times each question was considered was 0.51 (p <
0.001).
Discussion
The variations observed in the pc values of these 197 ques-
tions seem irrelevant. The correlation coefficient suggests
some correlation between the pc values the ® rst and last
time each MCQ was used. Some improvement from year to
year might have been expected because, besides using tests
for grading, we use the analysis of questions as a diagnostic
tool to improve the learning process in the forthcoming
years. Nevertheless, 50 items (25%) had a higher level of
difficulty when repeated, and 39 (20%) were easier, with
the remaining 108 (55%) showing no change.
We believe that if the exposure of questions in the post-
test discussions had made the items `public’ among students,
a consistent improvement in their performance would have
been found in this analysis. Whether or not the `damage’ to
questions is time dependent cannot be answered with these
data, because more than half of the items were used only
two or three times.
Correspondence: Dr Pedro Herskovic, 8644 Principe de Gales, La Reina, Santiago,
Chile, 6871619. Tel/fax: + 56 2 2739897; email: [email protected]
Table 1. pc values for the ® rst and last time multiple-choice questions were used.
pc values (%) for the last use of questions
0 ± 20 21± 40 41± 60 61± 80 81± 100
pc values (%)
for the ® rst use
of questions
0± 20 1
21± 40 2 1 3 3 1
41± 60 6 14 6 7
61± 80 1 3 12 26 18
81± 100 1 2 3 20 67
Note: pc value = percentage of students that answered the item correctly.
Medical Teacher, Vol. 21, No. 4, 1999
ISSN 0142-159X print /ISSN 1466-187X online/99/040430± 02 ½ 1999 Taylor & Francis Ltd430
Med
Tea
ch D
ownl
oade
d fr
om in
form
ahea
lthca
re.c
om b
y SU
NY
Sta
te U
nive
rsity
of
New
Yor
k at
Sto
ny B
rook
on
11/0
1/14
For
pers
onal
use
onl
y.
We conclude from post-test analysis that this kind of
exposure to students of MCQs seems not to preclude their
reutilization in future years.
Notes on contributor
PE D RO HER SK OVIC MD is Assistant Professor of Pediatrics at
the Department of Pediatrics, East Campus, Luis Calvo Mack-
enna Hospital, School of Medicine, University of Chile. His
interests are in nutrition and in medical education. From 1994 to
1997 he was in charge of the Pediatrics course for ® fth-year medical
students.
References
BANDARA NAYAK E, R.C., BUZZARD , A.J. & M ARSHALL, V.C. (1990)
Effects of publishing multiple-choice questions on their subsequent
item analysis, Australia and New Zealand Journal of Surgery, 60, pp.
937 941.
HERSKOVIC , P. & HERSKOVIC , J. (1998) Knowledge evaluation that
teaches and satis® es students, in: Proceedings of the AMEE Conference,
Prague 1998, p. 17 (Association for Medical Education in Europe).
M ILLER, B.J., EFFENEY, D.J. & GOUGH, I.R. (1993 ) Do medical
students remember multiple choice questions?, Australia and New
Zealand Journal of Surgery, 63, pp. 897± 900.
REES, P.J. (1986 ) Do medical students learn from multiple choice
examinations?, Medica l Education, 20, pp. 123 ± 125.
Reutilization of MCQs
431
Med
Tea
ch D
ownl
oade
d fr
om in
form
ahea
lthca
re.c
om b
y SU
NY
Sta
te U
nive
rsity
of
New
Yor
k at
Sto
ny B
rook
on
11/0
1/14
For
pers
onal
use
onl
y.
