Embed Size (px)
Citation preview
Does correct answer distribution influence student choices when writing multiple choice
question exams? Hannah Gray* and Dr. Jacqueline Carnegie**
Faculty of Health Sciences*, Faculty of Medicine**, University of OttawaIntroductionMultiple choice testing is a commonly used method of examination in many university classes’ particularly first year courses. Large class sizes associated with first- and second-year undergraduate courses have two important consequences for approaches to student evaluation. Midterm examinations often consist extensively of multiple-choice questions (MCQs) and multiple exam versions are created via MCQ scrambling in order to deter cheating when students write under crowded situations. However, when creating multiple-choice exams one must ensure that all version of the exam give equal opportunity to all students. This project investigates the possible influence of answer key organization on student selection when writing multiple versions of MCQ exams. The question is if a randomly distributed exam with the same lettered answer choice appears in a series of 4-6 consecutive MCQ will there be a disadvantage for students writing that version of the exam? If so, should professors put more time and effort into creating exams where series of over 4 do not occur in order to maintain equal opportunity for students?
MethodologyThroughout this study the incidence of correct answer selection for key questions (all with the correct answer represented by the same letter choice) in two or three versions of midterm exams written by students studying first-year anatomy and physiology (ANP) was examined. In one exam version, random scrambling during online exam preparation has placed those key questions (4-6) in a consecutive series; in the other two exam versions, the same questions are indiscriminately distributed among the other MCQs. In order to compare previously answer keys with how student view multiple choice a short, Likert-style survey was developed and administered to ANP students in which they self-assessed the extent to which they are influenced by correct answer distribution, when initially choosing their answers and when subsequently reviewing their array of answer choices prior to handing in the exam.
ConclusionThe random scrambling of multiple choice questions(MCQ) does not provide unfair advantage for students dependent on which multiple choice exam they choose and gives professors freedom to not have to spend excess time adjusting different versions of a MCQ exam. Data has shown that although students recognize a sequence of answers, often the deciding factor of the percentage of correct answers is due to the difficulty of a question and not the letter answer awarded. Many of the questions when cross-examined with their same question on a different MCQ exam had similar percentages of correct answers showing that the difficulty of the question is responsible for the number of wrong answers more than the placement of the question.
Acknowledgements
I would like to thank Jacqueline Carnegie for the opportunity to work with her for this project. I would also like to thank the Undergraduate Research Opportunity Program for the opportunity to learn about and participate in a research
project. I have enjoyed this experience and look forward to developing my recent found interest in research.
Contact Information:
Hannah Gray Email: [email protected]
33%
40%
27%
Transfering answers to scantron
Transfer to scantron right away
Answer questions in exam book then transfer
Answer question on the scantron and then answer the questions that are more difficult
Is a pattern noticed on scantron
Does a pattern cause a change in answer or second
look
Does the letter matter?
20
15
49
18
24
4044
35
4
11
24
2
74
0
Never Not Usually SometimesOften Always
10
5
10
15
20
25
30
35
40
18
40
27
4 5
How many questions in a series creates attention?
Three (3) Four (4) Five (5) Six (6) Seven (7)
90%
10%
When a Pattern is noticed, what questions are reviewed?
All questions in sequence Just last one or two
Just first one or two
0
10
20
30
40
50
60
70
80
90
100
Exam with Series (MCQ average=72.9%)Exam 2 (MCQ av-erage 72.5%)Exam 3(MQC av-erage= 70.7%)
0
10
20
30
40
50
60
70
80
90
100
Exam with series (MCQ Average=82.7%)Exam 2(MCQ Av-erage=81.1%)Exam 3 (MCQ Av-erage78.6%)
0
10
20
30
40
50
60
70
80
90
Exam with seriesExam without series
This exam existed in two versions with dark blue representing the exam with the four-question series and light blue the random distribution. The correct answer was C throughout the series. Especially striking are the third and fourth questions in the series where the same percent of students chose the correct answer whether that question was in the series or randomly distributed. (n = 135-145)
This was a series of five-questions. The dark blue column is the exam in which the series was found. The other two columns (Exam 2 and 3) are the two other versions of exams. It is worth noting that none of these questions was associated with a lower student success rate when present in a series. (n = 78)
Dark blue represents the exam with A as the correct answer in a series of four questions. In exam versions 2 and 3, these questions with correct answer A were randomly distributed throughout the rest of the MCQs. While there appears to be some variability in the percentage of students (n = 86-88 per exam version) who chose A as the answer, especially when looking at the more challenging questions, these differences were found to be not significant. Hence for this exam, the pattern of correct answers, whether in a series or randomly distributed, had no effect.
The following three bar graphs represent questions asked in the survey given to students to help gain an understanding of whether students notice a pattern caused by same lettered answers and if the letter in question makes a difference to whether students take a second look or if the series cause them to change their answers.
The following graphs are created based on answers received through an online survey sent out to the students of Anatomy and Physiology (ANP) III. The sample size is 55 (n = 55) and all percentages are based off this sample size.
A B C D ??
?
