Exploring student understanding of the engineering design

RESEARCH Open Access

Exploring student understanding of theengineering design process using distractoranalysisStefanie A. Wind1* , Meltem Alemdar2, Jeremy A. Lingle2, Roxanne Moore2 and Abdullah Asilkalkan1

Abstract

Typical approaches to assessing students’ understanding of the engineering design process (EDP) include performanceassessments that are time-consuming to score. It is also possible to use multiple-choice (MC) items to assess the EDP,but researchers and practitioners often view the diagnostic value of this assessment format as limited. However,through the use of distractor analysis, it is possible to glean additional insights into student conceptualizations ofcomplex concepts. Using an EDP assessment based on MC items, this study illustrates the value of distractor analysisfor exploring students’ understanding of the EDP. Specifically, we analyzed 128 seventh grade students’ responses to20 MC items using a distractor analysis technique based on Rasch measurement theory. Our results indicated thatstudents with different levels of achievement have substantively different conceptualizations of the EDP, wherethere were different probabilities for selecting various answer choices among students with low, medium, andhigh relative achievement. We also observed statistically significant differences (p < 0.05) in student achievementon several items when we analyzed the data separately by teacher. For these items, we observed different patterns ofanswer choice probabilities in each classroom. Qualitative results from student interviews corroborated many of theresults from the distractor analyses. Together, these results indicated that distractor analysis is a useful approach toexplore students’ conceptualization of the EDP, and that this technique provides insight into differences instudent achievement across subgroups. We discuss the results in terms of their implications for research andpractice.

IntroductionResearchers have found that engineering education inmiddle school is essential to increase students’ awarenessof engineering as a career path; therefore, researchers haveidentified middle school as a crucial period for engineer-ing education (Tafoya et al. 2005). Further, engineering de-sign is a core component of middle school engineeringcurricula. As a result, effectively measuring middle schoolstudents’ understanding of the engineering design processis essential.Approaches to assessing students’ understanding of

the engineering design process (EDP) are varied in re-search and practice. Many of the methods are qualita-tive in nature. For example, several researchers haveused interviews with both students and teachers to

examine a broad-range of perspectives of the EDP, in-cluding understanding of design practices, as well asvariants of approaches and perspectives of design.Alemdar et al. (2017a, 2017b) applied student inter-views to examine student understandings of the designprocess, perspectives of the collaborative design experi-ence, and recognition of engineering connections toother content areas. Researchers have also incorporatedinterviews into phenomenological studies for the pur-pose of identifying concepts such as successful engin-eering teaching practices (Adams et al. 2003).In other studies, researchers have used the process of

engineering design itself as a data source throughmethods such as protocol analysis and ethnographies.Protocol analyses explore student approaches to abbrevi-ated design tasks or student responses to design briefs.Field studies and full ethnographies include observationsof students as they approach design challenges in the

* Correspondence: [email protected] Studies in Psychology, Research Methodology, and Counseling,The University of Alabama, Box 870231, Tuscaloosa 35487, AL, USAFull list of author information is available at the end of the article

International Journal ofSTEM Education

© The Author(s). 2019 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made.

Wind et al. International Journal of STEM Education (2019) 6:4 https://doi.org/10.1186/s40594-018-0156-x

http://crossmark.crossref.org/dialog/?doi=10.1186/s40594-018-0156-x&domain=pdf

http://orcid.org/0000-0002-1599-375X

mailto:[email protected]

http://creativecommons.org/licenses/by/4.0/

classroom environment. Most often, researchers haveused these methods to identify patterns in behavior ofparticipants as they design, either as individuals or asgroups. Researchers have used think-aloud protocolswith individual students, and they have used analyses ofdialog for group-based activities. Generally, researchersuse these methods to understand the processes of stu-dent design and the student conceptions that underlietheir design decisions (Atman et al. 2014).Researchers also frequently use survey methods for the

purpose of individual-level assessment. Specifically, re-searchers conduct studies with surveys to gather informa-tion about students’ affective responses to engineering,such as design self-efficacy, the relevance and importanceof the EDP, and to identify difficulties that students experi-ence as they approach specific stages or design activities(Atman et al. 2014). Surveys are useful in capturingchanges in student perception over time, for example tomeasure self-efficacy and to determine whether studentself-efficacy changes before and after a design experience(Purzer 2011).Another common data source for assessing EDP

knowledge is student-produced classroom artifacts. Ar-tifacts may include students’ engineering notebooks orjournals, reflective narratives, portfolios, or designproducts such as prototypes. For example, researchershave examined students’ engineering journals to docu-ment improvements in design practices over time(Sobek II 2002). Likewise, researchers have analyzedstudents’ engineering portfolios to examine professionalengineering identify formation (Eliot and Turns 2011).For these artifacts, researchers apply evaluative rubricsto quantify student learning, measuring such aspects ascommunication, engineering practice, design, and ex-periments (Mourtes 1999; Williams 2001).These assessment techniques are valuable in that they

can provide researchers and practitioners with detailedinsight into students’ understanding of engineering design.However, in the classroom, the demands of teaching manystudents (particularly in secondary schools) often preventteachers’ thoughtful analysis of these products. In con-trast, multiple-choice (MC) items can be scored morequickly, and the results can be incorporated more readilyinto teachers’ instructional decisions (Briggs et al. 2006).A common criticism of assessments based on MC items isthat they provide limited diagnostic information regardingstudents’ conceptualization of complex concepts (Klassen2006). However, it is possible to construct MC items suchthat analyses of students’ answer choices provide insightinto students’ conceptualizations of the assessed concept(Haladyna and Rodriguez 2013). In particular, distractoranalysis is an approach to examining response patterns onMC items in which students’ answer choices are examinedalongside measures of achievement. When MC items are

constructed such that the distractors are related to im-portant aspects of student understanding of a particularconcept, this analytic approach provides valuable insightinto the types of correct and alternative conceptions thatare present within a sample of students, as well as insightinto the developmental hierarchy of the misconceptionsincluded within the item distractors. This information canbe used to inform instructional decisions to target specificareas of student understanding without the use of moretime-intensive performance assessment procedures. Re-searchers have discussed and applied distractor analysistechniques in many different content areas, including sci-ence and engineering (discussed further below). However,there has been relatively limited research on the use ofdistractor analysis in engineering education for middleschool students, and limited use of this approach to evalu-ate students’ understanding of the EDP. Furthermore, fewresearchers have used distractor analysis to explore differ-ences in student conceptualizations across subgroups.

PurposeThe primary purpose of this study is to illustrate the diag-nostic value of distractor analysis for exploring middleschool students’ (grades 6–8) understanding of the EDP.As a secondary purpose, we use distractor analysis to ex-plore differences in student achievement across subgroups.Accordingly, we focus on the following research questions:

1. What does distractor analysis of multiple-choiceEDP assessment items reveal about students’conceptions of the EDP?

2. How can distractor analysis of multiple-choice EDPassessment items be used to understand differencesin student performance on individual items acrosslearning environments?

Distractor analysis as a diagnostic toolDespite the limitations that are commonly associated withassessments based on multiple-choice (MC) items, manypractitioners and researchers use assessments with MCitems in educational settings. One reason is that MC itemsare easy to implement and score objectively, thus allowingtest users to assess a wide range of content in a single ad-ministration of an assessment. Furthermore, it is possibleto construct MC items such that they provide test userswith useful diagnostic information about student under-standing (Gierl et al. 2017; Thissen et al. 1989). Gierl et al.(2017) conducted a comprehensive literature review of theuse of MC item distractors in education, and concludedthat, when constructed with diagnostic purposes in mindand analyzed appropriately, MC item distractors provideuseful diagnostic information.In their review, Gierl et al. (2017) described two major

categories of distractor analysis techniques that

Wind et al. International Journal of STEM Education (2019) 6:4 Page 2 of 18

researchers and practitioners can use to extract diagnos-tic information from MC items whose answer choicesare constructed to provide meaningful diagnostic infor-mation, such as alternative conceptions or developmen-tal stages related to a particular concept. Gierl et al.called the first type of distractor analysis traditional dis-tractor analysis. Traditional distractor analysis tech-niques can be conducted using either classical testtheory (CTT) or item response theory (IRT) techniques.In both CTT and IRT traditional distractor analyses, re-searchers and practitioners examine patterns of students’responses at different total scores (CTT) or achievementestimates (IRT) in order to identify the relationship be-tween students’ understanding of the particular conceptincluded in the item, including students’ tendency to se-lect particular incorrect answer choices, and students’overall achievement. Such information can be useful forinforming theory related to students’ development re-lated to a concept or set of concepts as well as for plan-ning or revising instructional activities. Traditionaldistractor analysis techniques also include analyses ofdifferential distractor functioning (DDF). DDF analysesprovide valuable information about the degree to whichindividual distractors function in the same way for stu-dent subgroups given the same achievement level (Terziand Suh 2015; Penfield 2008). Along with differentialitem functioning (DIF) analyses, DDF can be used for avariety of purposes, including refining existing tests, de-veloping new instruments, and providing validity evi-dence for the interpretation and use of test scoreinferences.Second, Gierl et al. described contemporary distractor

analysis as distractor analysis techniques that are basedon relatively more-recent psychometric developments,with the goal of obtaining diagnostic information aboutstudents’ understanding using their responses to MCitems. In particular, some researchers have integratedcognitive diagnostic methods (CDM) with the construc-tion and analysis of MC item distractors. In traditionalmultiple-choice educational tests, the purpose of dis-tractors is just to identify students with limited under-standing or possible misconceptions of the conceptbeing tested. Among these techniques include the useof two-tier items (Treagust 1995). Two-tier items in-clude traditional MC items that are paired with follow-up, open-ended items that ask students to explain whythey selected their answer. Researchers have used thetwo-tiered item approach to explore student miscon-ceptions (Treagust 1995) as well as to develop new dis-tractors for MC items (Hestenes et al. 1992). Anotherexample of a contemporary distractor analysis approachis the use of ordered multiple-choice items (Briggs etal. 2006). When test developers construct ordered MCitems, they use a learning model to develop distractors

that reflect ordinal, increasing levels of understandingof a particular concept. Then, responses to ordered MCitems can be analyzed using IRT models in which par-tial credit is assigned to distractors relative to theirproximity to the correct response, such as the orderedpartition model (OPM; Wilson 1992). The distinguish-ing feature of the OPM is that multiple answer choicescan share the same score. Similar to the OPM, Wang(1998) proposed a distractor model that assigns differentlevels of difficulty to each distractor and examinesmodel-data fit statistics associated with the distractors. As afinal example of contemporary distractor analysis tech-niques, Gierl et al. described de la Torre’s (2009) modifiedversion of the multiple-choice deterministic-input noisy“and” gate model (DINA; Junker and Sijtsma 2001). Essen-tially, researchers use this analytic approach to model stu-dent responses to MC items in order to estimate a profileof “attributes” for each student that represent the particularknowledge, skills, or cognitive processes that the studentused to arrive at a particular answer choice.

Distractor analysis in engineering educationAs the above review demonstrates, research related todistractor analysis techniques is relatively common in re-search on educational measurement in general (Gierl etal. 2017). Several researchers have also examined the useof distractor analysis techniques in the context of engin-eering education. For example, two teams of engineeringeducation researchers proposed frameworks related tothe development and evaluation of curriculum and as-sessment that incorporate distractor analysis as a keycomponent.First, Jorion et al. (2015) presented an analytic frame-

work for evaluating concept inventories. Concept inven-tories are popular assessment formats in a variety ofdomains, including Science, Technology, Engineering,and Mathematics (STEM). These assessments incorpor-ate common or developmentally meaningful misconcep-tions as a method for identifying and addressing thesemisconceptions in instructional settings. In their frame-work, Jorion et al. present distractor analysis as a keycomponent of a body of evidence that researchers andpractitioners can collect to provide support for the inter-pretation and use of assessment instruments. These re-searchers provided empirical illustrations in which theyapplied their analytic framework to three assessments inthe context of engineering. Within Jorion et al.’s frame-work, the major role of distractor analysis is to provideinsight into student misconceptions to inform curriculaand instruction. Importantly, these researchers note thatone cannot meaningfully interpret the results from dis-tractor analyses unless there is sufficient evidence thatthe assessment instrument is targeting the intendeddomain. These researchers discussed distractor


analysis techniques based on CTT that reflect the trad-itional distractor analysis approach, as defined by Gierlet al. (2017). Similarly, Kong et al. (2014) proposed aframework to guide the design of curriculum and assess-ment. Focusing on nanotechnology as a content area, theseresearchers presented distractor analysis as a methodthrough which researchers and practitioners can evaluateand improve assessment instruments made up of MCitems, as well as a tool for identifying areas for improve-ments to engineering curricula. Kong et al. illustrated theirframework using empirical data from a MC assessment ofnanotechnology concepts. Similar to Jorion et al. (2015),these researchers discussed traditional distractor analysistechniques based on CTT.In addition to these frameworks, several researchers

have used distractor analyses in applied research as adiagnostic method for improving instructional practices.For example, Wage et al. (2005) applied traditional dis-tractor analysis techniques based on CTT to assessmentsin an undergraduate signals and systems course. Theseresearchers used distractor analysis to identify the mostcommon misconceptions, as well as misconceptions thatpersisted between a pre-test and post-test. More re-cently, Peuker et al. (2013) applied traditional distractoranalysis techniques based on CTT to an assessment inan undergraduate introductory engineering course. Theirstudy included a comparison of the benefits of con-structed response (CR) and MC-format assessments forinforming instructional practices in engineering educa-tion. These researchers concluded that distractor ana-lysis “can give insight to student misconceptions in thesame way that the traditional CR problems can providethe instructor information about misunderstandings,”and provide an efficient means through which to incorp-orate this information into instruction (p. 9). Along thesame lines, Sengupta et al. (2017) used traditional dis-tractor analysis techniques based on CTT to examine aconcept inventory in an undergraduate environmentalengineering course. These researchers discussed the ben-efits of distractor analysis as a tool for improving thequality of MC items, as well as a method for improvinginstructional practices.In this study, we use a traditional, Rasch-based ap-

plication of distractor analysis to examine students’conceptual understanding of the EDP. We also extendprevious research on distractor analysis in general byillustrating a method for using Rasch-based distractoranalysis to explore differences in patterns of studentresponses to MC items related to student subgroups.

MethodsIn order to explore the research questions for this study,we used data from middle-school students’ responses toa MC EDP assessment that was administered as part of

an experimental engineering curriculum project. Afterwe examined the quantitative data, we also examinedqualitative data from a later study of the same studentparticipants in order to more fully understand the quan-titative results. The quantitative data were our primaryfocus in the analysis, and the qualitative data served asecondary, explanatory role. We collected and analyzedthe quantitative and qualitative data separately, and thenmerged the qualitative results with our quantitative re-sults. Thus, our study can be described as a convergentexplanatory design where the quantitative results wereprimary (Creswell and Plano Clark 2011). We describethe context for data collection, the instruments, and ourdata analysis methods below.

Context for data collectionOur data were collected during the implementation ofan 18-week STEM-Innovation & Design (STEM-ID) en-gineering curriculum for middle school students.1 Cur-riculum developers iteratively designed the coursesbased on feedback from teachers and observations of thecourse in action (Alemdar et al. 2017a, 2017b). Thecourses were designed around the framework of the en-gineering design process (EDP). Essentially, the EDP is aconceptual model for the application of engineeringprinciples to practical design problems. Researchers andpractitioners have used a variety of EDP models as guid-ing frameworks for engineering education that vary interms of specific terms and sequences of activities (Car-berry et al. 2010). Figure 1 includes the specific EDP thatwas used in the curriculum context from which our datawere collected; we provide operational definitions foreach stage in our EDP model in Additional file 1. Theengineering curriculum was designed to utilize the EDPwithin a problem-based learning context, combined withan emphasis on science and mathematics practices, asdefined by the Next Generation Science Standards(NGSS; NGSS Lead States 2013) and the Standards ofMathematical Practice (Illustrative Mathematics 2014).

InstrumentsMultiple choice engineering design process assessmentA team of engineering education experts and psycho-metricians developed a series of MC EDP assessmentsfor each grade level (6th, 7th, and 8th) that corre-sponded to the experimental curriculum. The assess-ments were developed iteratively using a mixed-methods approach (Alemdar et al. 2017a; Wind et al.2017). Each of the EDP assessment items was alignedwith one or more stages of the EDP (see Fig. 1 andAdditional file 1). Furthermore, the items were devel-oped such that the distractors reflected alternativeconceptions related to engineering in general or par-ticular stages in the EDP that could be addressed with


additional instruction (for details about the developmentand students’ interpretations of the distractors, please seeAlemdar et al. 2017a, 2017b and Wind et al. 2017). Resultsfrom previous analyses suggested that the final versions of

the EDP assessments demonstrated acceptable psychomet-ric properties, such that they could be used to evaluate stu-dent achievement following the implementation of theexperimental engineering curriculum (see Wind et al.

Fig. 1 Conceptual model for the engineering design process. This figure is the representation of the engineering design process included in thecurriculum used to instruct the students who participated in this study


2017). We provide additional background informationabout the EDP assessment used in this study, as well as thecontext in which we collected data in Additional file 1.

Student interview protocolAfter we collected student responses to the MC items,we collected supplementary qualitative data to informour interpretation of the quantitative results. Specific-ally, we conducted semi-structured interviews withstudents using an interview protocol. All of the inter-viewers used the same semi-structured interview proto-col with follow-up questions to elicit rich data thatcould be subjected to sequential qualitative analysis(Miles et al. 2014). We designed the interview protocolaround several themes, including students’ understand-ing of the engineering design process, their perceptionsabout the course, and the mathematics and scienceconnections. In this study, we focused on interview re-sults related to students’ understanding of the EDP.Specifically, during the relevant section of the inter-view, we asked students to describe the use of the EDPwhile working on a design challenge. Some of the ques-tions included the following: “Tell me about how youused the Engineering Design Process in your STEM-IDclass. Were you ever confused about what you weresupposed to do for a certain stage in the engineeringdesign process?”

ParticipantsIn the current analysis, we focused on the Fall 2016 ad-ministration of the EDP assessment for seventh gradestudents, which included 20 items. A total of 128 stu-dents participated in this assessment who were enrolledin four classrooms in separate schools (teacher 1: N = 39;teacher 2: N = 25; teacher 3: N = 35; teacher 4: N = 29).These sample sizes are within the recommended rangefor Rasch analyses of low-stakes multiple-choice assess-ments for stable estimates of item or person calibrationswithin ± 1 logit over replication (Linacre 1994; Wrightand Stone 1979). Using the interview protocol describedabove, we collected qualitative data from thirty studentsfrom four middle schools.

Data analysisIn order to explore the use of distractor analysis as amethod for understanding students’ conceptualizationsof the EDP, we used a combination of quantitative andqualitative analytic tools. In light of the purpose of ourstudy, we focused primarily on the results from thequantitative data analysis, and we used qualitative datato inform our interpretations of the quantitative results.

Quantitative data analysisOur quantitative data analysis procedure consisted offour main steps. First, we calculated measures of studentachievement and item difficulty using the dichotomousRasch model (Rasch 1960). Briefly, the Rasch model is astatistical approach that can be used to examine studentresponses to items in educational assessments in orderto obtain measures of student achievement and item dif-ficulty. This approach is based on a transformation ofassessment item responses from an ordinal scale to alog-odds (logit) scale, such that measures of studentachievement and item difficulty can be expressed ona common linear scale with equal units.We selected the Rasch model for our analysis for three

reasons. First, this model has been used in previous ap-plications of distractor analysis techniques in STEMeducation (Herrmann-Abell and DeBoer 2011, 2014;Wind and Gale 2015). Second, the EDP assessment wasdeveloped based on the Rasch model (Wind et al. 2017).Finally, the Rasch model leads to a straightforward inter-pretation of student achievement and item difficulty thatfacilitates the interpretation of results from the dis-tractor analysis procedure. The model can be stated inlog-odds form as follows:

ln Pni x ¼ 1ð Þ=Pni x ¼ 0ð Þ½ � ¼ θn−δi ð1Þ

where ln[Pni(x = 1)/Pni(x = 0)] is the log of the odds that a stu-dent n provides a correct response to item i, rather than anincorrect response; θn is the location of student n on thelogit scale; and δi is the location of item i on the logit scale.Student measures reflect student achievement levels, wherehigher values indicate higher achievement, and lower valuesindicate lower achievement. Likewise, item locations reflectitem difficulty. For items, higher locations suggest more dif-ficult items, and lower locations suggest easier items. Weconducted the Rasch analysis using the Facets™ softwareprogram (Linacre 2015).In step two, we examined students’ responses to the

EDP assessment items using distractor analysis. Wedeigned our distractor analysis procedure based on astudy by Wind and Gale (2015). Specifically, we startedby examining the frequency of students who selectedeach of the four answer choices for all 20 EDP assess-ment items in order to verify that at least one studentselected each of the answer choices for all items. Evi-dence that students used all four of the answer choicessuggested that the distractors are useful for exploringpatterns in student understanding. We also examinedthe average achievement level (θ estimates from Eq. 1)among students who selected each answer choice. If theitem was functioning as expected, the correct answerchoice would be associated with the highest averageachievement level.


To interpret the distractor analysis results, we con-structed graphical displays that illustrate the relation-ship between measures of student achievement and theprobability of selecting each of the four answer choices.Figure 2 is an example of a distractor analysis plot cal-culated using student achievement estimates from theRasch model. The x-axis shows Rasch-model estimatesof student achievement. The lines in the plot show theproportion of students (y-axis) who selected each an-swer choice, where different lines are used for each an-swer choice. The correct answer for the example itemis C. As might be expected, the proportion of studentsselecting answer choice C increased across increasinglevels of student achievement. One can obtain add-itional information about students’ answer choices byexamining the proportion of students who selected thethree incorrect answer choices in relation to studentachievement. Specifically, beginning with the lowest-achieving students (farthest left on the x-axis), studentswith the lowest overall achievement levels were most-often attracted to answer choice B, followed by answerchoice A. Moving to the right, students with slightlyhigher achievement levels were most often attracted toanswer choice A. Answer choice A remained the mostpopular distractor for all other achievement levels.In step three, we identified items on the EDP assess-

ment on which students in the four classrooms had sig-nificantly different performance using an extension ofthe Rasch model that included an additional variable

for classrooms. Specifically, we specified a Many-FacetRasch (MFR) model (Linacre 1989) with a facet forclassrooms as follows:

ln½Pnigðx¼1Þ=Pnigðx¼0Þ� ¼ θn−δi−ηg ð2Þ

where θn and δi are defined as in Eq. 1, and ηg is de-fined as the average achievement level of studentswithin classroom g. We estimated this model usingFacets (Linacre 2015).Using Eq. 2, we examined differences in student

achievement across classrooms related to individualitems. Specifically, we used the Facets computer pro-gram to calculate the difficulty of each EDP assessmentitem separately for each classroom. Then, we used pair-wise analyses (t tests) to determine whether the difficultyof individual items were significantly different across theclassrooms. This test is appropriate for comparing itemestimates between classrooms because the item esti-mates are calculated on an interval-level scale.Finally, in step four, we constructed distractor plots

separately for the classrooms between which we ob-served significant differences in item difficulty. Afterconstructing the graphical distractor analysis plots, weused substantive theory related to the EDP, informationfrom the curricular context, and results from the quali-tative analyses (described below) to interpret the pat-terns revealed by the distractor plots.

Qualitative data analysisWe used the NVIVO™ software program to code andanalyze the student interviews according to a sequentialqualitative analysis process (Miles et al. 2014). We com-pleted qualitative coding in two phases. In the first phase, asingle coder categorized student responses according to themajor topic areas included in the interview protocol. In thenext phase, a group of four coders applied a provisional listof sub-codes to a cross-sectional sample of interview tran-scripts. These sub-codes included process codes aligned toindividual phases of the EDP (e.g., problem identification,prototyping, and testing solutions) and magnitude codes in-dicating positive and negative examples.

Mixed-methods analysisAfter we collected and analyzed the quantitative andqualitative data, we merged the results from the qualita-tive analysis with the results from our quantitative ana-lyses. Specifically, we used the qualitative interview datato inform our interpretation of step two (distractor ana-lysis for EDP items based on the complete sample) andstep four (distractor analysis across teachers) of ourquantitative analyses.

Fig. 2 Example distractor analysis plot. This figure shows an exampleof a distractor analysis plot for a multiple-choice item. The x-axisdisplays estimates of student achievement. Student achievementestimates are shown in logits, and these estimates were calculated usingthe Rasch model. Low values indicate lower achievement, andhigh values indicate higher achievement. The y-axis shows theobserved proportion of responses for a given answer choice. Thelines in the plot show the proportion of students who selectedeach of the four answer choices (A, B, C, or D) at each level ofachievement. For this example item, the correct response is C(pink line with triangle plotting symbols)


ResultsStep one: overall calibration of the EDP assessmentBefore we conducted our analyses, we checked theRasch model assumptions of unidimensionality (i.e.,student responses to the assessment items are primarilydue to only one latent variable), and local independence(i.e., after controlling for the latent variable of interest,there are no systematic relationships or dependenciesamong students’ responses to the items). We checkedthe unidimensionality assumption using a principalcomponents analysis of the residuals, or differences be-tween the observed responses and the responses thatwould be expected based on the Rasch estimates (Lina-cre 1998). This analysis allowed us to check for anymeaningful patterns in residuals that suggest that morethan one latent variable significantly contributed to stu-dent responses. We checked the local independenceassumption by calculating correlations between the re-siduals associated with each item.In the combined sample (all of the classrooms), the

Rasch measures explained 22.23% of the variance in re-sponses; this value is evidence of acceptable adherenceto the assumption of unidimensionality for Rasch ana-lyses, because it is higher than the generally acceptedcritical value of 20% for analyzing potentially multidi-mensional instruments (Reckase 1979). Furthermore, weexamined the inter-item correlations of model residualswithin the combined sample; these values were less than0.30. We also observed adherence to the Rasch assump-tions within the individual school sites. Specifically, theRasch measures explained > 20% of the variance in

students’ responses for each of the classrooms. Further-more, the inter-item correlations of model residualswere less than 0.30 for all of the classrooms. Because allof our sub-samples approximated the Rasch model as-sumptions, we proceeded with our analyses.Table 1 summarizes the results from the first step of

the data analysis procedure. The average estimatedstudent measure was θ = 0.11 (SD = 0.92), which isnear the average estimated item difficulty location(mean δ = 0.00; SD = 0.63). This result suggests thatthe assessment was targeted well to the group of sev-enth grade students. The relatively low standard errors(SE) for student estimates (M = 0.52, SD = 0.07) anditem estimates (M = 0.20, SD = 0.01) provide additionalevidence of precision for the Rasch estimates. Theaverage student SE is larger than the average item SEbecause there are more observations of each item (128students responded to each item) compared to the ob-servations of each student (each student is observedusing 20 items). Nonetheless, these SEs are within theranges that other researchers have reported forlow-stakes MC tests in a variety of educational do-mains (Bond and Fox 2015).Furthermore, average values of model-data fit statis-

tics are within the range of values that previous re-searchers have established as expected when there isacceptable fit to the Rasch model (MSE fit statisticsaround 1.00; Smith 2004; Wu and Adams 2013). Foritems, the Rasch reliability of separation statistic indi-cated significant differences in difficulty across the in-dividual items (Rel = 0.90); however, the reliability ofseparation statistic for students was notably lower(Rel = 0.60), suggesting that there were several groupsof students who had similar levels of achievement.Taken together, the practical implication of these re-sults is that the distractor analyses can be meaning-fully interpreted in terms of students’ understandingof the EDP.

Step two: distractor analysis for EDP items based on thecomplete sampleFor all 20 items, the correct response was associatedwith the highest average achievement level. These resultssuggest that the answer choices for the MC EDP itemswere functioning as intended. Accordingly, we examinedstudents’ responses to the EDP items using distractoranalysis plots. We illustrate the results from this stepusing four example items: item 12, item 14, item 17, anditem 19. We selected these example items for two mainreasons. First, they represent different stages of the EDP.Second, these items provide clear examples of the typesof patterns of student answer choices that we observedover all 20 items.

Table 1 Summary statistics from the dichotomous Rasch model

Statistic Student (N = 128) Item (N = 20)

Location (logits)

M 0.11 0.00

SD 0.92 0.63

Standard error

M 0.52 0.20

SD 0.07 0.01

Infit MSE

M 1.00 1.00

SD 0.14 0.14

Outfit MSE

M 1.01 1.01

SD 0.22 0.21

Separation statistics

Reliability of separation 0.68 0.90

Chi-Square 332.7* 181.7*

Degrees of freedom 127 19

*p < 0.05


Item 12Item 12 was developed to assess students’ understandingof the Identify the Problem stage of the EDP, as well astheir general understanding of the EDP as a process. Toanswer this item, students had to consider a scenario inwhich they were asked to design a new oven for a com-pany that makes ovens for restaurants and bakeries, andidentify the first step in this process according to theEDP. We present the text and distractor analysis plot foritem 12 in Fig. 3. Examination of the plot reveals severalinteresting aspects of student understanding related tothe first stage of the EDP. First of all, the proportion ofstudents who selected the correct response (D) increasedacross increasing levels of student achievement—sug-gesting that the highest-achieving students had theexpected understanding of this stage of the EDP. Next, itis interesting to note that the second-most prominentanswer choice was distractor B, followed by distractor C.This pattern suggests that students who did not selectthe correct answer appeared to skip ahead in the EDP tothe Ideate stage, as reflected in answer choice B, or thePrototype & Test stage, as reflected in answer choice C.

Item 14Item 14 was developed to assess students’ understanding ofthe Evaluate and the Communicate your Solution stages ofthe EDP. This item asked students to identify relevant de-sign details to communicate to the manager of an ovencompany (see Fig. 4). As expected, the proportion of stu-dents who selected the correct response (D) increasedacross increasing achievement levels; likewise, studentswere less attracted to each of the distractors as achievement

levels increased. Among those students who selected incor-rect responses, students selected distractor A most often,followed by distractor B. Inspection of the three mostpopular answer choices (D (correct response), A, and B) in-dicates that students recognized that a company managermay not be interested in results from early stages of theEDP, as reflected in distractor C, but had some difficultydetermining which of the other details would be mostrelevant for the specified customer. The results from ourqualitative analyses of student interviews reflected similarfindings. For example, one student described the Commu-nicate your Solution stage as follows: “Communicatingyour solution, like thinking of what to say, specific thingsyou have to think and to tell other people to get yourproduct. That’s really hard, too.” In general, student re-sponses to the interview questions suggested that studentshave a general grasp of what is needed to communicatewith a client, but that this step is difficult.

Item 17Item 17 was intended to evaluate students’ under-standing of the Understand stage of the EDP. Thisitem asked students to determine an appropriate nextstep after learning that customers do not prefer thevisual characteristics of an oven design (see Fig. 5). Asexpected, the proportion of students who selected the cor-rect response (B) increased across increasing achievementlevels, suggesting that high achieving students had the ex-pected understanding of this stage of the EDP. Across therange of achievement levels, students who provided incor-rect responses to this item selected distractor C most

Fig. 3 Distractor analysis plot for item 12: full sample. The plot shows patterns of student answer choices for item 12 within the entire sample.The x-axis displays estimates of student achievement. Student achievement estimates are shown in logits, and these estimates were calculatedusing the Rasch model. Low values indicate lower achievement, and high values indicate higher achievement. The y-axis shows the observedproportion of responses for a given answer choice. The lines in the plot show the proportion of students who selected each of the four answerchoices (A, B, C, or D) at each level of achievement. For this item, the correct response is D (black line with asterisk plotting symbols)


often. Examination of the other distractors indicates thatmost of the students recognized that the appropriate nextstep according to the EDP involved revising the design ofthe oven, as reflected in distractor C. Students who se-lected distractor C focused on the performance of the ovenrather than the visual characteristics of the design.

Item 19Item 19 was intended to assess students’ understanding ofthe Evaluate stage of the EDP. This item asked students

to identify an activity that reflects the Evaluate stage ofthe EDP with regard to the design of a recycling bin for aschool cafeteria (see Fig. 6). The correct answer for thisitem is C. Compared to the previous example items,students’ responses to item 19 were more evenly distrib-uted among all four answer choices. Although the highest-achieving students selected the correct response mostoften, the lowest-achieving students (θ < 0) were equallyattracted to the three distractors. Students with middle-range achievement (0 ≤ θ ≤ 1) selected distractor B most

Fig. 4 Distractor analysis plot for item 14: full sample. The plot shows patterns of student answer choices for item 14 within the entire sample.The x-axis displays estimates of student achievement. Student achievement estimates are shown in logits, and these estimates were calculatedusing the Rasch model. Low values indicate lower achievement, and high values indicate higher achievement. The y-axis shows the observedproportion of responses for a given answer choice. The lines in the plot show the proportion of students who selected each of the four answerchoices (A, B, C, or D) at each level of achievement. For this item, the correct response is D (black line with asterisk plotting symbols)

Fig. 5 Distractor analysis plot for item 17: full sample. The plot shows patterns of student answer choices for item 5 within the entire sample. Thex-axis displays estimates of student achievement. Student achievement estimates are shown in logits, and these estimates were calculated usingthe Rasch model. Low values indicate lower achievement, and high values indicate higher achievement. The y-axis shows the observed proportion ofresponses for a given answer choice. The lines in the plot show the proportion of students who selected each of the four answer choices (A, B, C, orD) at each level of achievement. For this item, the correct response is B (turquoise line with square plotting symbols)


often. Interestingly, the activity described in distractor B in-volves aspects of the correct response, in that the engineerconsiders the requirements for the recycling bin—indicatingthat students who selected this answer choice may have hadsome emerging understanding of the Evaluate stage of theEDP. Student responses during the interviews often indi-cated an understanding of the purpose of the Evaluate stage,which is to choose a design based on how likely it is to meetthe requirements. For example, when asked how a wing de-sign was selected from among the potential airplane designs,one student responded: “We decided based on how wewould meet your requirements and what would be thebest possible choice for it.” However, some students didnot understand this purpose. For example, in the fol-lowing quotation, a student explains that the wing de-sign choice was based upon personal preference.

Interviewer: “How did you decide which one you weregoing to try?”

Student: “Because I’ve always had a favorite plane. Youknow the planes that they got the low wings and thoselittle standup things on the end? That’s my favorite plane.I don’t know what it’s called but it’s my favorite plane. Ialways do that one. That’s how I chose my plane.”

Together, our qualitative analyses of student interviewresponses corroborate our quantitative results. That is, the

students who did not fully understand the purpose of theEvaluate stage of the EDP demonstrated some confusionwith regard to the purpose of this stage, which may haveled them to select incorrect answer choices.

Step three: items with different difficulty levels acrossclassroomsThe third step in our data analysis procedure was to iden-tify differences in item difficulty across the four classroomsof students who participated in the administration of theEDP assessment using the MFR model (Eq. 2; Linacre1989). We observed seven items for which there were sta-tistically significant differences in difficulty between at leasttwo classrooms (p < 0.05): items 1, 2, 3, 7, 9, 16, and 19.

Step four: distractor analysis across teachersFor each the items that we identified in step three, weconstructed distractor analysis plots separately using thedata from students in each of the four teachers’ class-rooms. We created these teacher-level plots for items forwhich we observed statistically significant differences initem difficulty. We illustrate our procedure for exploringstudents’ responses separately by teacher using three ex-ample items: item 7, item 9, and item 19.

Item 7Item 7 was intended to assess students’ understandingof the Prototyping & Testing stage of the EDP. This

Fig. 6 Distractor analysis plot for item 19: full sample. The plot shows patterns of student answer choices for item 16 within the entire sample.The x-axis displays estimates of student achievement. Student achievement estimates are shown in logits, and these estimates were calculatedusing the Rasch model. Low values indicate lower achievement, and high values indicate higher achievement. The y-axis shows the observedproportion of responses for a given answer choice. The lines in the plot show the proportion of students who selected each of the four answerchoices (A, B, C, or D) at each level of achievement. For this item, the correct response is C (pink line with triangle plotting symbols)


item asked students to identify the stage of the EDPduring which engineers create detailed technical draw-ings for a new design. For this item, we observed statis-tically significant differences in achievement betweenstudents who were instructed by teacher 1 and teacher2. Accordingly, we constructed distractor analysis plotsseparately for these two subgroups of students (seeFig. 7). For all of teacher 1’s students, the correct re-sponse (C) was the most popular answer choice, and nostudents selected answer choice D. Because distractors Aand B reflect earlier stages in the EDP than the correct an-swer choice (Prototype & Testing), these results suggest thatthe students in teacher 1’s classroom understood that creat-ing technical drawings is an activity that occurs before thefinal stages of the EDP. Students’ answer choices were morevaried in teacher 2’s classroom. In this classroom, stu-dents with the lowest achievement level were mostoften attracted to answer choices B and A. Similar toteacher 1’s students, none of teacher 2’s selected an-swer choice D. However, it is interesting to note that,

for teacher 2’s students, the correct response was onlythe most popular answer choice among students withachievement levels around θ = 1. Students with thehighest achievement levels (θ ≥ 1) selected answerchoice A most often. Results from our qualitative ana-lyses suggested that students’ confusion between thePrototyping & Testing stage and other stages is oftendue to the students’ desire to quickly move throughthe stages in order to create a prototype, as illustratedby the following quotations from a student interview:

Interviewer: “Have there ever been times whenyou’ve skipped any of these stages or noticed otherstudents skipping any of the stages?”

Student: “Someone skipped Ideate in my last class.Theirs just completely failed. Their catapultcompletely failed. Last year we had to make agame for a carnival, it had to do with a catapultand shooting it. It hit none of the points.”

Fig. 7 Distractor analysis plot for item 7: teacher 1 and teacher 2. The plots in this figure show patterns of student answer choices for item 7 inteacher 1’s classroom (top plot) and in teacher 2’s classroom (bottom plot). In both plots, the x-axis displays estimates of student achievement.Student achievement estimates are shown in logits, and these estimates were calculated using the Rasch model. Low values indicatelower achievement, and high values indicate higher achievement. The y-axes shows the observed proportion of responses for a givenanswer choice. The lines in each plot show the proportion of students who selected each of the four answer choices (A, B, C, or D) ateach level of achievement. For this item, the correct response is C (pink line with triangle plotting symbols)


Interviewer: “That’s interesting. In your class, how doyou know that they skipped that stage?”

Student: “Because after they thought of the idea,they just skipped right into making the prototype.”

In this interview excerpt, the student described a situ-ation where their classmates skipped a stage of theEDP—thus corroborating the results from our distractoranalysis, which indicated confusion about the sequenceof activities in the EDP.

Item 9Item 9 was intended to assess students’ understanding ofthe Understand stage of the EDP. This item is based on ascenario where students are asked to consider a set ofdesign requirements for a dog container to be used in air-planes. To answer the item, students were asked to con-sider a new customer request and determine what the

new request represents with regard to the EDP. For thisitem, we observed statistically significant differences inachievement between students who were instructed byteacher 1 and teacher 2, as well as between the studentswho were instructed by teacher 1 and teacher 4. Accord-ingly, we constructed distractor analysis plots separatelyfor the students who were instructed by teacher 1, teacher2, and teacher 4 (see Fig. 8). Among teacher 1’s students,it is interesting to note that the correct response was themost popular answer choice for the lowest-achieving stu-dents, as well as among high-achieving students. However,for students with mid-range achievement (θ around 0),distractor C was most popular. Furthermore, an equalproportion of the highest-achieving students in this class-room selected the correct response (B) as well as dis-tractor A. On the other hand, students in classroom 2were only attracted to one distractor (C). Finally, amongteacher 4’s students, the correct response was the mostpopular for students with all achievement levels. However,

Fig. 8 Distractor analysis plot for item 9: teacher 1, teacher 2, and teacher 4. The plots in this figure show patterns of student answer choices foritem 9 in teacher 1’s classroom (top plot), teacher 2’s classroom (middle plot), and teacher 4’s classroom (bottom plot). In each plot, the x-axisdisplays estimates of student achievement. Student achievement estimates are shown in logits, and these estimates were calculated using theRasch model. Low values indicate lower achievement, and high values indicate higher achievement. The y-axes shows the observed proportionof responses for a given answer choice. The lines in each plot show the proportion of students who selected each of the four answerchoices (A, B, C, or D) at each level of achievement. For this item, the correct response is B (turquoise line with square plotting symbols)


for the lowest-achieving students who did not answer item9 correctly, distractor A was most attractive. Based on thequantitative results, this item was effective at identifyingstudents who had a sophisticated understanding of therole of requirements in engineering design, specificallywith regard to adding a requirement (Understand). In thequalitative data, we identified several students who ap-peared to combine this stage with activities typically in-cluded in the Ideate stage:

Interviewer: What sort of things did you do tounderstand the problem?

Student: We looked up some websites that were goingto help us in the cell phone holder, and we designedsome sketches on paper to what we liked.

In this example, the student demonstrated some con-fusion with regard to the unique characteristics of theUnderstand stage of the EDP.

Item 19Our final example of teacher-level distractor analysis isitem 19. As we noted above, item 19 was intended to as-sess students’ understanding of the Evaluate stage of theEDP, and the item asks students to identify an activitythat is part of this stage. For this item, we observed sta-tistically significant differences in achievement betweenstudents who were instructed by teacher 2 and teacher3; accordingly, Fig. 9 includes distractor analysis plotsfor these subgroups of students. Among teacher 2’sstudents, students with the lowest achievement levels (θ< − 1) were most often attracted to distractors B and A.According to the EDP used in the experimental curricu-lum, the activities included in these stages precedeEvaluate. Accordingly, these students’ responses indicatethat they were able to eliminate activities associated withthe final stages of the EDP, as reflected in answer choiceD. Among teacher 3’s students who did not provide acorrect response to item 19, students with the lowestachievement levels (θ ≤ − 1) were attracted to answer

Fig. 9 Distractor analysis plot for item 19: teacher 2 and teacher 3. The plots in this figure show patterns of student answer choices for item 19 inteacher 2’s classroom (top plot) and in teacher 3’s classroom (bottom plot). In both plots, the x-axis displays estimates of student achievement.Student achievement estimates are shown in logits, and these estimates were calculated using the Rasch model. Low values indicate lower achievement,and high values indicate higher achievement. The y-axes shows the observed proportion of responses for a given answer choice. The lines ineach plot show the proportion of students who selected each of the four answer choices (A, B, C, or D) at each level of achievement. For thisitem, the correct response is C (pink line with triangle plotting symbols)


choice B most often. Although the proportion of stu-dents selecting the correct response decreased slightly insome intervals across increasing achievement levels, itwas the most popular answer choice for students withmid-range and high achievement levels.

ConclusionsIn this study, we used distractor analysis to explore stu-dents’ understanding of the EDP in the context of an ex-perimental engineering curriculum for middle schoolstudents. Specifically, we examined the alignment betweenstudents’ overall proficiency levels on a multiple-choiceEDP assessment and the probability that they selected cer-tain answer choices. Because the assessment items weredeveloped such that the answer choices reflected alterna-tive conceptions related to engineering in general or par-ticular stages in the EDP that could be addressed withadditional instruction, patterns of students’ responses overdifferent levels of achievement provided insight into therelationship between students’ overall achievement andtheir conceptualizations of the EDP. We also considereddifferences in patterns of students’ responses between sub-groups of students. In particular, for items on which weobserved differences in student performance, we used dis-tractor analysis techniques to explore further the differ-ences in students’ conceptualizations of the EDP that mayhave contributed to different overall performance betweensubgroups.Overall, our results suggested that distractor analysis

provides insight into students’ conceptualizations of theEDP across different levels of achievement. Our obser-vation that students with different achievement levelsselected different incorrect answer choices suggests thatstudents with different levels of overall achievement mayhave different conceptualizations of the EDP stages. Byexamining patterns of student responses to individualitems, we were able to identify particular stages of theEDP that relatively lower-achieving students struggled tounderstand in general, and stages of the EDP that studentsmay have been inclined to skip or use out of order.During our analysis of semi-structured qualitative in-

terviews with students related to the EDP interviews,students expressed many of the same findings that weobserved using distractor analysis—that is, that therewere certain stages of the EDP that relatively lower-achieving students struggled to understand, and stagesof the EDP that students may have been inclined toskip or use out of order. For example, in our distractoranalysis of item 14, which asked students to identifyrelevant details to communicate to a company manager,we observed that students had some difficulty deter-mining the specific details that would be most relevantfor the specified customer. Results from student inter-views reflected a similar finding. Specifically, students’

responses to the interview questions suggested that stu-dents have a general grasp of what is needed to com-municate with a client, but that this step is difficult.Further, in our distractor analysis of item 19, whichasked students to identify an activity that reflects theEvaluate stage of the EDP, we found that students hadsome difficulty in distinguishing the unique characteris-tics of this stage from other stages in the EDP. Studentresponses during the interviews often indicated an un-derstanding of the purpose of the Evaluate stage, whichis to choose a design based on how likely it is to meetthe requirements. However, some students demon-strated confusion with regard to the activities that cor-respond to this stage.Our distractor analysis results also revealed stages of the

EDP that students were inclined to skip over when consid-ering a design challenge. In particular, our analyses of item7 and item 12 revealed that students who did not selectthe correct answer choice experienced confusion relatedto the appropriate sequence of EDP stages. Studentsexpressed similar difficulties during our interviews. For ex-ample, several students expressed confusion between thePrototype and Test stage and other stages in the EDP. Fur-thermore, student responses suggested that their confusionbetween EDP stages is often due to their desire to quicklymove through the stages in order to create a prototype—similar to our distractor analysis results for item 7 anditem 12.The results from our comparison of patterns of stu-

dent answer choices across teachers were also interest-ing. Although all of the teachers who participated inour study received the same professional developmentwith regard to the experimental curriculum and all ofthe students participated in the same curriculum, weobserved different patterns of answer choices on theitems for which there were significant differences instudent achievement. In other words, students who se-lected incorrect responses appeared to do so for differentreasons across classrooms. These results provide a startingpoint for additional research on differences in studentachievement related to the EDP, and how achievement dif-ferences correspond to various misconceptions.

Implications for engineering educationSeveral scholars have explored the role of the EDP as akey component of student proficiency in engineering (e.g.,Kelly 2014; Wendell and Kolodner 2014), particularlywithin the context of problem-based and/or project-basedinstruction (Marra, Jonassen, Palmer, & Luft, 2014; e.g.,Kolmos and De Graff 2014). Accordingly, our results sug-gest that distractor analysis is a valuable tool that can pro-vide researchers and practitioners with insight into thedevelopment of students’ understanding of the EDP thatcan inform curriculum development and instructional


practices. As we noted earlier, although several re-searchers have applied distractor analysis techniques toscience education (Herrmann-Abell and DeBoer 2011,2014; Wind and Gale 2015), researchers have not yetused distractor analysis to explore students’ responsesto MC items in the context of engineering education.Accordingly, our study provides initial insight into theuse of distractor analysis as a method for exploring stu-dents’ conceptualizations of engineering concepts. Whencontrasted with other forms of engineering assessmentthat are intended to help researchers and practitionersunderstand students’ conceptualizations of the EDP, suchas engineering notebooks, distractor analysis of MC itemsis a promising alternative. In particular, when used withpurposefully constructed MC items, distractor analysistechniques can provide researchers and practitioners witha potentially less time-consuming method by which togain insight into students’ understanding of engineeringconcepts. This approach is particularly promising whenother approaches to assessing student understanding ofthe EDP are not practical. The approach that we demon-strated in our study can provide insight into student con-ceptualizations of EDP stages beyond the typical “correct/incorrect” scoring procedures for MC assessments.Our findings are particularly meaningful in light of re-

cent emphases on engineering design as a key compo-nent of student proficiency in the integrated Science,Technology, Engineering, and Mathematics (STEM) dis-ciplines (e.g., Borgford-Parnell et al. 2010; Cardella et al.2008; Kelly 2014; Kolmos and De Graff 2014). Further-more, current STEM curricula in the USA emphasizestudent proficiency in engineering as a key componentof college and career readiness (Auyang 2004; Carr et al.2012). Despite the importance placed on engineeringeducation in policy and practice, there has been rela-tively limited attention in research on the developmentof psychometrically sound engineering assessment tech-niques. In this study, we illustrated a method that re-searchers and practitioners can use to explore the resultsfrom MC engineering assessments from a diagnosticperspective. In practice, researchers and practitionerscan apply the techniques that we illustrated here toother MC assessments of engineering concepts. As weobserved in this study, such analyses can provide insightinto student conceptualizations of engineering concepts.Furthermore, researchers and practitioners can use the

distractor analysis techniques that we illustrated in thisstudy as a diagnostic tool to evaluate the performance ofdistractors in MC items in order to improve assessmentpractices in engineering education. For example, distractoranalysis techniques can help researchers and practitionersidentify individual distractors in MC items that are notproviding useful diagnostic information, or that might beeliciting confusion from otherwise high-achieving

students. For example, evidence that none or very few of asample of students selected a particular distractor maysuggest that the answer choice is not helpful for distin-guishing among students with different levels of under-standing for a particular concept. Furthermore, evidencethat otherwise high-achieving students are most attractedto a distractor rather than the correct answer choice maysuggest a need to revise the distractor so that it does notintroduce unnecessary confusion. Likewise, as we ob-served in our analysis of item 9, evidence that the lowest-achieving students are most attracted to the correct an-swer, while students with higher levels of achievement se-lect distractors may suggest confusion related to particulardistractors that may encourage guessing from low-achiev-ing students.Finally, our results suggest that researchers and practi-

tioners can use distractor analysis techniques as a diagnos-tic tool for exploring differences among subgroups ofstudents in terms of their conceptualization of the EDP.In this study, we used distractor analysis techniques to in-vestigate differences in patterns of students’ responseswithin different classrooms for items on which we ob-served differences in student performance. This approachallowed us to understand the degree to which differencesin students’ conceptualizations of the EDP within thesesubgroups may have contributed to differences in theirperformance on the MC items. In practice, researchersand practitioners can use distractor analysis techniques toexplore how achievement differences between other sub-groups of students, such as demographic subgroups, maybe related to different patterns of alternate conceptionsthat are reflected in MC item distractors.

Implications for research on distractor analysisOur study also has implications for research on dis-tractor analysis in general. Although several researchershave used distractor analysis techniques to explore dif-ferences over time (Wind and Gale 2015) and betweengrade levels (Herrmann-Abell and DeBoer 2011), re-searchers have not fully considered the use of distractoranalysis as a supplementary tool to explore differencesin student performance at the item level related to stu-dent subgroups. In this study, we illustrated a proced-ure for combining evidence of differential performancebetween student subgroups on individual items withdistractor analysis in order to uncover potential con-tributing factors to achievement differences. Thisapproach provides insight into differences in studentperformance across subgroups in terms of specific an-swer choices that reflect different aspects of students’understanding and application of a particular concept,such as stages of the EDP. Researchers can use the re-sults of such analyses to inform instructional decisionsin order to focus on specific concepts without the use


of more time-intensive performance assessmentprocedures.

Limitations and directions for future researchOur study has several limitations that researchers andpractitioners should consider before generalizing the re-sults to other contexts. First, we conducted our analysesusing data from seventh grade students who participatedin an experimental middle grades engineering curriculumproject in the USA. Likewise, we used data from one MCengineering assessment instrument. Our student sampleand assessment instrument may not reflect the character-istics of other student populations and MC engineeringassessments. Accordingly, we encourage researchers andpractitioners to consider the alignment between the stu-dent sample and instrument that we used in this studyand other populations of students and assessment instru-ments before generalizing our results. In future studies,researchers could explore the use of distractor analysis inother engineering education contexts in order to provideadditional insight into the generalizability of our findings.In particular, we encourage researchers to explore thegeneralizability of our findings to larger samples of stu-dents than the sample that we included in this analysis.Along the same lines, we encourage researchers to con-duct additional mixed-methods studies in which theyexplore student the alignment between student conceptu-alizations of the EDP and their specific answer choices toMC items. Specifically, researchers might use a sequentialexploratory mixed-methods design in which they beginwith a distractor analysis, and design qualitative data col-lection to purposefully target students who selected par-ticular answer choices. Such analyses would provideadditional insight into the interpretation of distractor ana-lysis as a method for exploring student understanding ofthe EDP.Second, it is important to note that we focused on one

type of distractor analysis. Other types of distractor ana-lysis, including methods based on cognitive diagnosticmodels or other distractor analysis models, may providedifferent information about students’ response patterns.In future studies, researchers should apply distractoranalysis techniques besides the methods that we used inthis study to MC engineering assessments.It is also important to note that the degree to which

one can gain insight into students’ conceptualizationsof the EDP using distractor analysis is somewhat lim-ited compared to other approaches, such as qualitativeanalyses of engineering design notebooks and inter-views with students. Nonetheless, our findings suggestthat distractor analysis provides valuable initial insightinto student conceptualizations of the EDP that canserve as a starting place from which to inform add-itional investigations. In future studies, we encourage

researchers to consider other forms of mixed-methodsdesigns in which they combine distractor analysistechniques with qualitative methods in order to morefully understand students’ conceptualizations of theEDP.Finally, it is important to note that we used distractor

analysis techniques to explore students’ understandingof the EDP. We did not directly explore how teacherscould use the results from such analyses to inform theirapproach and pedagogy, or the effectiveness of distractoranalysis results for this purpose. In future studies, re-searchers could explore the implications of using dis-tractor analysis results to inform teaching practices.

Endnotes1This study was approved by the IRB at Georgia Insti-

tute of Technology, protocol number H13201.

Additional file

Additional file 1: Background information about the EngineeringDesign Process assessment. (DOCX 23 kb)

AcknowledgementsThe authors wish to acknowledge Dr. Jessica D. Gale and Dr. Sunni Newtonfor their assistance with data collection.

FundingThis research was supported by a grant from the National ScienceFoundation, grant # 1238089. Any opinions, findings, and conclusionsor recommendations expressed in these materials are those of theauthor(s) and do not necessarily reflect the views of the NationalScience Foundation.

Availability of data and materialsPlease contact the corresponding author for data requests.

Authors’ contributionsSW, MA, JL, and RM designed the assessments used to collect data for thisstudy. SW proposed the research questions and analytic approach, analyzedthe data, and drafted most of the manuscript sections. AA assisted with theliterature review and construction of the distractor analysis plots. MA, JL, andRM offered comments, revised the manuscript, and provided editorialsuggestions. All of the authors read and approved of the final manuscript.

Ethics approval and consent to participateThe Institutional Review Board for Georgia Institute of Technology has approvedthe study reported in this manuscript. Parental consent and studentassent procedure are in place per IRB requirement. The IRB protocolidentification number is #H13201. Additional documentation related toIRB approval is available from the corresponding author.

Consent for publicationNot applicable.

Competing interestsThe authors declare that they have no competing interests.

Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims in publishedmaps and institutional affiliations.


https://doi.org/10.1186/s40594-018-0156-x

Author details1Educational Studies in Psychology, Research Methodology, and Counseling,The University of Alabama, Box 870231, Tuscaloosa 35487, AL, USA. 2GeorgiaInstitute of Technology, Atlanta, GA, USA.

Received: 19 June 2018 Accepted: 20 December 2018

ReferencesAdams, R. S., Turns, J., & Atman, C. J. (2003). Education effective engineering

designers: the role of reflective practice. Design Studies, 24(3), 275–294.Alemdar, M., Lingle, J. A., Moore, R., & Wind, S. A. (2017a). Developing an

engineering design process assessment using think-aloud interviews.International Journal of Engineering Education, 33(1(B)), 441–452.

Alemdar, M., Lingle, J. A., Newton, S., Moore, R., Rosen, J., & Usselman, M. (2017b).Students’ perceptions of middle school STEM innovation and design course.In Proceedings of the ASEE 2017 Annual Conference. Columbus, OH.

Atman, C. J., Eris, O., McDonnell, J., Cardella, M. E., & Borgford-Parnell, J. L. (2014).Engineering design education: research, practice, and examples that link thetwo. In J. Aditya & B. M. Olds (Eds.), Cambridge handbook of engineeringeducation research (pp. 201–225). Cambridge: Cambridge University Press.

Auyang, S. Y. (2004). Engineering—an endless frontier. Cambridge, MA: HarvardUniversity Press.

Borgford-Parnell, J., Deibel, K., & Atman, C. J. (2010). From engineering designresearch to engineering pedagogy: bringing research results directly to thestudents. International Journal of Engineering Education, 26(4), 748–759.

Briggs, D. C., Alonzo, A. C., Schwab, C., & Wilson, M. (2006). Diagnostic assessmentwith ordered multiple-choice items. Educational Assessment, 11(1), 33–63https://doi.org/10.1207/s15326977ea1101_2.

Carberry, A. R., Lee, H.-S., & Ohland, M. W. (2010). Measuring engineering designself-efficacy. Journal of Engineering Education, 99(1), 71–79 https://doi.org/10.1002/j.2168-9830.2010.tb01043.x.

Cardella, M., Atman, C. J., Turns, J., & Adams, R. (2008). Students with differingdesign processes as freshmen: case studies on change. International Journalof Engineering Education, 24(2), 246–259.

Carr, R. L., Bennett, L. D., & Strobel, J. (2012). Engineering in the K-12 STEMstandards of the 50 U.S. states: an analysis of presence and extent. Journal ofEngineering Education, 101(3), 1–26.

Creswell, J. W., & Plano Clark, V. L. (2011). Designing and conducting mixedmethods research. Thousand Oaks, CA: Sage.

de la Torre, J. (2009). A cognitive diagnosis model for cognitively based multiple-choice options. Applied Psychological Measurement, 33(3), 163–183 https://doi.org/10.1177/0146621608320523.

Eliot, M., & Turns, J. (2011). Constructing professional portfolios: sense-making andprofessional identity development for engineer undergraduates. Journal ofEngineering Education, 100(4), 630–654.

Gierl, M. J., Bulut, O., Guo, Q., & Zhang, X. (2017). Developing, analyzing, andusing distractors for multiple-choice tests in education: a comprehensivereview. Review of Educational Research, 87(6), 1082–1116.

Haladyna, G., & Rodriguez, M. C. (2013). Developing and validating test items. NewYork, NY: Routledge.

Herrmann-Abell, C. F., & DeBoer, G. E. (2011). Using distractor-drivenstandards-based multiple-choice assessments and Rasch modeling toinvestigate hierarchies of chemistry misconceptions and detect structuralproblems with individual items. Chemistry Education Research andPractice, 12(2), 184–192.

Herrmann-Abell, C. F., & DeBoer, G. E. (2014). Developing and using distractor-driven multiple-choice assessments aligned to ideas about energy forms,transformation, transfer, and conservation. In R. F. Chen, A. Eisenkraft, D.Fortus, J. Krajcik, K. Neumann, J. Nordine, & A. Scheff (Eds.), Teaching andlearning of energy in K–12 education (pp. 103–133). Springer.

Hestenes, D., Wells, M., & Swackhamer, G. (1992). Force concept inventory. ThePhysics Teacher, 30(3), 141–158.

Jorion, N., Gane, B. D., James, K., Schroeder, L., DiBello, L. V., & Pellegrino, J. W.(2015). An analytic framework for evaluating the validity of concept inventoryclaims. Journal of Engineering Education, 104(4), 454–496.

Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with fewassumptions, and connections with nonparametric item response theory.Applied Psychological Measurement, 25(3), 258–272 https://doi.org/10.1177/01466210122032064.

Kelly, A. F. (2014). Design-based research in engineering education: current stateand next steps. In A. Johri & B. M. Olds (Eds.), Handbook of engineeringeducation research (pp. 497–518). New York, NY: Cambridge University Press.

Klassen, S. (2006). Contextual assessment in science education: background,issues, and policy. Science Education, 90(5), 820–851.

Kolmos, A., & De Graff, E. (2014). Problem-based and project-based learning inengineering education. In Handbook of engineering education research (pp.797–518). New York, NY: Cambridge University Press.

Kong, Y., Diefes-Dux, H. A., Rodgers, K. J., Douglas, K. A., & Madhavan, K. (2014).Development and validation of a nano size and scale instrument (NSSI). In2014 IEEE Frontiers in Education Conference (FIE) (pp. 1–4). IEEE https://doi.org/10.1109/FIE.2014.7044332.

Linacre, J. M. (1989). Many-facet Rasch measurement. Chicago, IL: MESA Press.Linacre, J. M. (1994). Sample size and item calibration stability. Rasch

Measurement Transactions, 7(4), 328.Linacre, J. M. (1998). Structure in Rasch residuals: Why principal components

analysis (PCA)? Rasch Measurement Transactions, 12(2), 636.Linacre, J. M. (2015). Facets Rasch measurement (version 3.71.4). Chicago, IL:

Winsteps.com.Marra, R., Jonassen, D. H., Palmer, B., & Luft, S. (2014). Why problem-based

learning works: Theoretical foundations. Journal on Excellence in CollegeTeaching, 25(3&4), 221–238.

Miles, M. B., Huberman, A. M., & Saldana, J. (2014). Qualitative data analysis (3rded.). Thousand Oaks, CA: Sage.

Mourtes, N. J. (1999). Portfolio assessment in aerodynamics. Journal of EngineeringEducation, 88, 223–229.

NGSS Lead States. (2013). Next generation science standards: For states, by states.Washington, DC: The National Academies Press.

Purzer, S. (2011). The relationship between team discourse, self-efficacy, andindividual achievement: a sequential mixed-methods study. Journal ofEngineering Education, 100(4), 655–679.

Rasch, G. (1960). Probabilistic models for some intelligence and achievement tests(expanded edition, 1980). Chicago: University of Chicago Press.

Reckase, M. D. (1979). Unifactor latent trait models applied to multifactor tests:Results and implications. Journal of Educational and Behavioral Statistics, 4(3),207–230 https://doi.org/10.3102/10769986004003207.

Sengupta, S., Ergas, S., Cunningham, J., Goel, R., Feldman, A., & Chen, Y.-H. (2017).Concept inventory for fundamentals of environmental engineering courses:concept inventory development and testing. Environmental EngineeringScience, 34(12), 895–907.

Smith, R. M. (2004). Fit analysis in latent trait models. In E. V. Smith & R. M. Smith (Eds.),Introduction to Rasch measurement (pp. 73–92). Maple Grove, MN: JAM Press.

Sobek, D., II. (2002). Use of journals to evaluate student design processes. Montreal:Paper presented at the American Society for Engineering Education conference.

Thissen, D., Steinberg, L., & Fitzpatrick, A. R. (1989). Multiple-choice models: thedistractors are also part of the item. Journal of Educational Measurement,26(2), 161–176.

Treagust, D. F. (1995). Diagnostic assessment of students’ science knowledge. InS. M. Glynn & R. Duit (Eds.), Learning science in the schools: Research reformingpractice (pp. 327–346). Mahwah, NJ: Erlbaum.

Wage, K. E., Buck, J. R., Wright, C. H. G., & Welch, T. B. (2005). The signals andsystems concept inventory. IEEE Transactions on Education, 48(3), 448–461.

Wang, W.-c. (1998). Rasch analysis of distractors in multiple-choice items. Journalof Outcome Measurement, 2(1), 43–65.

Williams, J. M. (2001). The engineering portfolio: communication, reflection, andstudent learning outcomes assessment. International Journal of EngineeringEducation, 18(2), 199–207.

Wilson, M. (1992). The ordered partition model: an extension of the partial creditmodel. Applied Psychological Measurement, 16(4), 309–325 https://doi.org/10.1177/014662169201600401.

Wind, S. A., Alemdar, M., Gale, J. D., Lingle, J. A., & Moore, R. (2017). Developingan engineering design process assessment using mixed methods: anillustration with Rasch measurement theory and cognitive interviews. Journalof Applied Measurement, 18(2), 100–121.

Wind, S. A., & Gale, J. D. (2015). Diagnostic opportunities using Raschmeasurement in the context of a misconceptions-based physical scienceassessment. Science Education, 99(4), 721–741 https://doi.org/10.1002/sce.21172.

Wright, B. D., & Stone, M. H. (1979). Best Test Design. Chicago, IL: MESA Press.Wu, M., & Adams, R. J. (2013). Properties of Rasch residual fit statistics. Journal of

Applied Measurement, 14(4), 339–355.


https://doi.org/10.1177/01466210122032064

https://doi.org/10.1177/01466210122032064

Documents

Exploring student understanding of the engineering design