Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 303–312Copenhagen, Denmark, September 8, 2017. c©2017 Association for Computational Linguistics

Multiple Choice Question Generation Utilizing An Ontology

Katherine Stasaski and Marti HearstUC Berkeley

{katie stasaski, hearst}@berkeley.edu

Abstract

Ontologies provide a structured represen-tation of concepts and the relationshipswhich connect them. This work investi-gates how a pre-existing educational Biol-ogy ontology can be used to generate use-ful practice questions for students by usingthe connectivity structure in a novel way.It also introduces a novel way to generatemultiple-choice distractors from the ontol-ogy, and compares this to a baseline of us-ing embedding representations of nodes.

An assessment by an experienced scienceteacher shows a significant advantage overa baseline when using the ontology fordistractor generation. A subsequent studywith three science teachers on the resultsof a modified question generation algo-rithm finds significant improvements. Anin-depth analysis of the teachers’ com-ments yields useful insights for any re-searcher working on automated questiongeneration for educational applications.

1 Introduction

An important educational application of NLP isthe generation of study questions to help studentspractice and study a topic, as a step toward masterylearning (Polozov et al., 2015). Although much re-search exists in automated question generation thetechniques needed for educational applications re-quire a level of precision that is not always presentin these approaches.

Ontologies have the potential to be uniquelybeneficial for educational question generation be-cause they allow concepts to be connected in non-traditional ways. Questions can be generatedabout different concepts’ properties which span

different areas of a textbook or even different edu-cational resources.

However, ontologies are not commonly used inNLP approaches to generate complex, multi-partquestions. This may be due to concern about on-tology’s incompleteness and the fact that they areusually structured for other purposes.

In this work, we describe a novel method forgenerating complex multiple choice questions us-ing an ontology, with the aim of testing a stu-dent’s understanding of the bigger picture of howconcepts interact, beyond just a definition ques-tion. This technique generates questions thathelp achieve understanding at the second level ofBloom’s taxonomy (Bloom et al., 1956). We alsogenerate multiple choice distractors using severalontology- and embedding-based approaches.

We report on two different studies. The firstassesses both the questions and the question dis-tractors with one domain expert, a middle schoolscience teacher. This finds evidence that theontology-based approach generates novel and use-ful practice questions. Based on the findings fromthat study, we adjust the question generation al-gorithm and report on a subsequent evaluation inwhich three experts quantitatively rank and qual-itatively comment on a larger selection of ques-tions. The results are strong, with more than 60questions out of 90 receiving positive ratings fromtwo of the judges. Additionally, we categorize andprovide in-depth analysis of qualitative feedbackand use this to inform multiple future directions toimprove educational practice question generation.

2 Related Work

Prior work has explored both automatically gener-ating educational ontologies from text and utiliz-ing expert-created ontologies for other tasks. Forinstance, Olney et al. (2011) explored extracting

303

nodes and relationships from text to build a con-cept map ontology automatically from textbooks.Other work has also attempted to build ontolo-gies from non-educational texts (Benafia et al.,2015; Szulman et al., 2010) and has explored uti-lizing crowd-sourcing to build an ontology fromtext (Getman and Karasiuk, 2014).

Prior approaches to question generation fromontologies have involved hand-crafted rules totransform a relationship into a question (Olneyet al., 2012b; Papasalouros et al., 2008; Ou et al.,2008). However, these approaches mainly gener-ate questions for a single fact and do not combinemultiple pieces of information together to createmore complex questions. There is the potentialto explore other, more complex, types of ques-tion generation procedures from the ontology. Ap-proaches have also utilized online questions forontology-driven generation, but this is less gener-alizable (Abacha et al., 2016).

Prior work aimed at generating educationalpractice questions has generated questions directlyfrom text using a series of manual translations anda ranking procedure to determine quality (Heilmanand Smith, 2010, 2009; Heilman, 2011).

Other work has focused on question genera-tion, independent of an educational context. Alarge-scale question generation task posed to thecommunity prompted a focus on factual questiongeneration from texts and knowledge bases (Ruset al., 2008; Graesser et al., 2012). Approacheshave included factual generation directly from text(Brown et al., 2005; Mannem et al., 2010; Mazidiand Tarau, 2016; Yao et al., 2012) as well as gener-ation from knowledge bases (Olney et al., 2012a).

Recent advances in text generation have usedneural generative models to create interestinglyworded questions (Serban et al., 2016; Indurthiet al., 2017). However, because we are using a hu-man created ontology and lack specialized trainingdata, we utilize hand-crafted rules for generation.

3 Question Generation

We utilize an educational Biology ontology togenerate multiple choice questions, which consistof the text of a question, the correct answer, andthree distractor multiple choice candidates.

3.1 Dataset

We use an expert-curated ontology documentingK-12 Biology concepts (Fisher, 2010) designed

Figure 1: Selected part of the Biology ontology.

for educational applications. While more re-sources could be used to accomplish this task, weonly utilize the ontology to explore the efficacyof this question generation approach. By utilizingan expert-curated ontology instead of an automati-cally generated one, we operate under the assump-tion that the ontology is correct and complete.Future work can explore utilizing this method inconjunction with other educational resources andtechniques.

The ontology contains 1,260 unique conceptnodes and 227 unique relationship types with a to-tal of 3,873 node-relationship-node triples. Theaverage outgoing degree is 7. Figure 1 shows asmall sample.

3.2 Using The Structure of the Ontology

The novel aspect of our approach is the manner inwhich we use an ontology to go beyond simplefactoid question generation. Rather than gener-ating a question from a node-relation-node triple,this algorithm makes use of the graph structure ofthe ontology to create complex questions that linkdifferent concepts, with the aim of challenging thestudent to piece together different concepts.

The goal of this evaluation was to determine ifthis novel way of combining concepts would bejudged as creating useful, coherent questions fortesting students.

To create these novel structured questions, thealgorithm chooses a node to act as the answer, andfrom three randomly-chosen outgoing links it gen-erates a question. The relations of the outgoinglinks and the nodes on the other ends are usedto form the question words. For instance, fromthe node “Water” emanates the links (DissolvesIn,“salt”), (HasProperty, “cohesion”), and (InputTo,“evaporation”) from which is generated the ques-tion “What dissolves salt, has cohesion, and is aninput to evaporation? (Water)”

A total of 992,926 questions can be generatedvia this method from the ontology. These ques-tions are distributed over 426 nodes, with the av-

304

erage number of questions that can be generatedper node being 2,330. While 834 nodes do nothave three outgoing links to generate a questionfrom, these nodes can be chosen as properties tocommpose other questions.

3.3 Generating Distractors

Good multiple choice questions should have dis-tractors (alternative answers to distract the studentfrom the correct answer). These should not besynonymous with the correct answer, but shouldbe a plausible answer which should not be so far-fetched as to be obviously incorrect.

We experimented with several different waysof generating multiple choice distractors using thestructure of the ontology, and compared these withtwo embedding based methods. In each case, ifthe text of a distractor overlaps with the correctanswer, we do not use it.

3.4 Ontology Distractor Generation

We experimented with 5 different ontology-baseddistractor methods. For each distractor generationmethod, the correct answer node, n is connectedto three property nodes n1, n2, and n3 via rela-tionships r1, r2, and r3 respectively. In order toensure that distractor node m does not correctlyanswer the question, we make sure at least one ofn1, n2, or n3 does not connect to m. The follow-ing methods are illustrated in Figure 2.

Two Matching Relationships: This methodchooses m such that m is connected to n1 via r1and m is connected to n2 via r2.

One Matching and One New Relationship:This method chooses m such that m is connectedto n1 via r1 and m is connected to n2 via a differ-ent relationship, r4 6= r2.

Two New Relationships: This method choosesm such that m is connected to n1 via a differentrelationship type r4 6= r2 and m is connected ton2 via a different relationship, r5 6= r3.

One Matching Relationship: This methodchooses m such that m is connected to n1 via r1.

We also examined an additional question-independent ontology-grounded approach.

Node Structure: This approach rates pairs ofnodes by similarity, where similarity is determinedby their tendency to link to similar relation typesand to link to the same intermediate nodes. Moreformally, let cn denote the set of nodes which areconnected to any node n and let ln,r denote the

number of connections that n has of type r. Thesimilarity between n and m is computed as:

sn,m = count(cn ∩ cm)−∑

r

|ln,r − lm,r|

3.5 Embedding Distractors

We implemented two methods to generate dis-tractors grounded in the embeddings of thenodes. Both utilize pre-trained word embeddings(Mikolov et al., 2013). A given node n consistsof a series of words, w1, w2, ..., wn. We create amulti-word embedding by distributing weight andplacing more emphasis on the last word in a se-quence, which we assume to be the head word.The similarity s between the two embedded nodesen and em is determined by cosine similarity.

Correct Answer Embeddings: are generatedby comparing the correct answer, n with the mostsimilar node in the graph G:

distractor = arg maxm∈G

sen,em

Question Component Embeddings: are gen-erated by finding the most similar node to thequestion components n1, n2, and n3. The aboveequation is computed for each component.

3.6 Ontology Coverage

Each of these methods is applicable to a subset ofnodes in the ontology. From a randomly sampledselection of 10,000 questions, 15.6% met require-ments for Two Matching Relationship Distractors,16.2% met requirements for One Matching, OneNew Distractors, 29.1% met requirements for TwoNew Relationship Distractors, and 25.6% met re-quirements for One Matching Relationship Dis-tractors. Node Structure, Correct Answer Embed-dings, and Question Component Embeddings allhad complete coverage due to the nature of themethods.

3.7 Pedagogical Motivation

This question generated method is similar to an in-verse of the “Feature Specification” questions de-scribed by Graessner et al (1992) in which stu-dents are asked to describe properties of a con-cept. An example format of this type of question is“Which qualitative attributes does entity X have?”(Graesser et al., 1992). Instead of prompting stu-dents to list properties of a concept, we provide

305

Two Matching Relationships: “What is a type of organicmolecule, is a class of compound in living things, and iscomposed of phosphorous atoms?”

One Matching, One New Relationship: “What has struc-ture spindle fibers, has organelle cell wall, and has organellecytoplasm?”

Two New Relationships: “What can be protist cell, can beanimal cell, and is a part of a Eukaryote?”

One Matching Relationship: “What is required in dissolv-ing, can be table sugar, and dissolves in water?”

Figure 2: Question-specific distractor generation methods. Correct answer nodes are leftmost in eachgraph, and chosen distractor nodes are rightmost.

three features of concepts and ask the students tochoose the correct concept given the features.

Through these questions, we aim to challengestudents to connect different features of a conceptwhile working within the constraints that the ques-tions and corresponding answers be able to be gen-erated via the ontology. The type of questions thatarise are intended for mastery learning, in whichstudents learn simpler facts about a concept beforetackling more difficult conceptual problems (Polo-zov et al., 2015). While the questions are not crit-ical thinking ones, they are designed to be morecomplex than a simple definition question and tobe a gateway to more difficult questions.

Another potential application of these questionsis preparation for extracurricular trivia competi-tions, such as Quiz Bowl1. One type of ques-tion asked at these competitions is one which listsmany characteristics of a concept and challengesstudents to quickly identify the concept. Connect-ing multiple facets of a concept are essential to an-swer these questions.

3.8 Generating the TextBecause the purpose of this study was to examinethe feasibility of the ontology structure for ques-tion generation, we use hand-crafted rules to pro-duce the question text. The human-generated on-tology has nodes and relationships that are worded

1https://www.naqt.com/about-quiz-bowl.html

Relationship RuleHas -Characteristic

If n = verb→ “n.”If n = noun→ “has n. ”If n = adjective→ “is n.”

HasProcess “has a process called n.”CanBe “can be a/an n.”

Table 1: Common relationship types and rule-methods used to generate the question segment.

somewhat naturally, which helped this process.We devised simple relationship-to-text transla-tions rules; examples are shown in Table 1.

4 Study 1

4.1 Method

We conducted a study to assess the quality ofthe questions and distractors. We asked a middleschool science teacher with 20 years of experienceto rate the quality of 20 complex questions on ascale of 1-7. The scale was explained such that 1was “Poor,” 4 was “OK,” and 7 was “Excellent.”To create the test set, we randomly selected nodesand generated questions about them 2.

We also asked the teacher to rate distractors on ascale of 1-5 (a narrower scale was chosen as it wasthought these would be more difficult to differen-

2Full text of questions evaluated in both studies can befound in an appendix in the supplementary materials.

306

tiate than the questions). Each distractor methodwas tested with 10 different questions (each with3 distractors). Questions were randomly chosenfrom all possible questions that could be gener-ated from the ontology, and were disregarded ifa given distractor method was not able to gener-ate three valid distractors. For both processes de-scribed above, the teacher was prompted to enteroptional comments about the question or distrac-tors.

4.2 ResultsQuantitative distractor generation results can beseen in Table 2. The difference in ontology-generated distractors compared to embedding-generated distractors was significant using a t-testwith p value < 0.001. Comparing the highest-performing ontology and embedding methods(One Matching, One New Relationship and Cor-rect Answer Embedding) is also significant underthe t-test with p < 0.05. This indicates that whilethe overall feedback was critical, there is promisein using ontology over embedding distractors.

The questions’ ratings averaged 2.25 out of 7.After analyzing the qualitative comments, this canbe attributed primarily to the unnatural wordingof the questions. Qualitative comments about thequestions are categorized in Table 3.

4.3 DiscussionAll ontology distractor methods except for OneMatching Relationship received explicit com-ments pointing out that the distractors were“good,” while no embedding approach receivedthese comments. This indicates that the combi-nation of different methods have strengths thatcontribute to a good set of distractors. TheNode Structure method provides broad distractors,while the other methods provide question-specificones. For example, the distractors “cell wall,”“chloroplast,” and “central vacuole” for the ques-tion “What is an organelle of eukaryotic cell, is anorganelle of animal cell, and is an organelle of fun-gal cell? (Golgi body)” are plausible but incorrect.

However, the teacher also commented that somedistractors of both embedding and ontology meth-ods were “poor.” Examining the “poor” distrac-tors for embedding questions shows that the dis-tractors can come from concept areas unrelated tothe question. For instance, for the question “Whatcontains chromosome, is an organelle of eukary-otic cell, and is a type of organelle? (nucleus),”

Distractor Type AvgTwo Matching Relationships 2.37One Matching, One New Relationship 2.78Two New Relationships 2.03One Matching Relationship 2.07Node Structure 2.63Correct Answer Embedding 2.10Question Component Embedding 1.60

Table 2: Averaged distractor scores.

Type of Comment CountUnnatural Wording of Question 31Good question 24OK question 17Unnatural Grouping of Characteristics 2Text of Node was Confusing 2Imprecise Relationship 1Not Middle School Level 1

Table 3: Categorization of qualitative feedback forquestions. Feedback was included for all ques-tions, including those in the distractor section.

the Question Component Embeddings generated“new genetic recombinations” as a distractor.

By contrast, the ontology-generated distractorswere marked “poor” when the question includedone unique property. For example, for the ques-tion “What eats mice, eats deer, and is a type ofpredator? (mountain lion),” the improbable dis-tractor “vole” was chosen because it eats mice andis a predator. This suggests the necessity of moreformal reasoning and real world knowledge cou-pled with the ontology information.

There were also instances in which the pro-posed distractors were unintentionally correct an-swers. For the embedding-generated distractors,this happened because the method favors distrac-tors that are more similar to the correct answer.For the ontology-generated distractors, this oc-curred where the ontology was incomplete.

While the distractor evaluation is quite prelim-inary as it only involves one expert evaluating 10sets of distractors per generation method, these re-sults suggest the potential to explore an ontologymethod of distractor generation in future work.

5 Study 2

Based on these initial results, we extended thework in several ways. First, based on the results

307

of Study 1, we found that the teacher was sensi-tive to any flaws of the wording of the questions,so we modified the assessment with questions thatwere manually touched up to remove grammati-cal errors. Second, although we had evidence thatthe ontology-based method was producing high-quality questions, we noticed that the target an-swers of many of the questions were quite general(e.g. “solids”, “water”). Therefore, we modifiedthe algorithm to take out-degree and relation com-monality into account. We also decided to investi-gate questions composed of only two relations aswell as three relations. Finally, we wanted to im-prove the evaluation in two ways: (i) by assessingmore questions, and (ii) by having more indepen-dent judges per question. We accomplished this byfinding assessors with appropriate backgrounds onan expert-oriented crowdwork site. Each of thesemodifications is described in detail below, alongwith results of this second evaluation.

5.1 Modifications to Generation Algorithm

We wanted to assess if the ontology generationmethod worked well, but were wondering if per-haps including three relations made the questionstoo complex or unusual. For this reason, we de-cided to include questions with only two relationsin Study 2.

After adapting the question generation methodto generate questions with two properties as op-posed to three, given that the node “Evaporation”is connected to the two relations (Outputs, “wa-ter vapor”) and (OppositeOf, “condensation”), thequestion generated from these two properties is:“What yields water vapor and is the opposite ofcondensation? (evaporation).”

Improving diversity of questions and coverageof the ontology were priorities in this study. Wemodified our question generation algorithm to pri-oritize these goals. We placed restrictions on thenumber of outgoing connections of a node connn

such that 5 < connn < 30.In addition, for two-property questions, we im-

posed the constraint that the collection of cho-sen properties yields exactly one unique correctanswer. This added an additional check that thequestion generated was not about general proper-ties that multiple nodes in the ontology fulfill.

For a random selection of 45 questions, we donot allow the algorithm to generate more than 2questions about the same concept, to evaluate a

more diverse set of questions. We also do not al-low a node to be asked as a property involved ina question more than 5 times. This procedure wasused for the selection of questions for the studybelow.

5.2 Manual Adjustments of QuestionExpression

The first experiment showed that the grammaticalerrors were distracting and affected the evaluationof the content of the questions. Therefore, we ad-justed the grammatical correction rules as well asmade minor edits by hand to ensure grammaticalcorrectness. Some minor grammatical errors stillexist, but major ones which obscure the meaningof the question were manually corrected. So, forinstance, we fixed errors in the specification of ar-ticles, as seen in the removal of a and addition ofs when transforming the question “What yields aRNA and is contained in chromosome? (gene)”to “What yields RNA and is contained in chromo-somes? (gene).” These changes were made to atotal of 16 questions. When a question was mod-ified, the average number of changed characterswas 3.3.

5.3 Evaluation of Question QualityThree middle school science teachers, each withat least 10 years of experience, were recruitedto evaluate the generated questions via Upwork3,an expert-oriented freelance work matching site.Each teacher evaluated 90 questions (45 two-property and 45 three-property) both quantita-tively on a scale from 1 to 7 and qualitatively viaopen-ended comments.

The title shown to the assessors was “MiddleSchool Science Practice Question Evaluation” andthe instructions for the assessment were:

Below are shown some automaticallygenerated biology questions, intendedfor practice studying. Please rate thequality of these questions for these pur-poses. Use the rating 1 to 7 where 1 =Poor, 4 = OK, and 7 = Excellent.

Please ignore grammatical errors. Foreach question, please briefly explainyour rating in one sentence.

We recognize that by informing teachers thatthe questions were generated automatically, they

3http://www.upwork.com

308

Figure 3: Distribution of judges’ quantitative scoreover the 90 evaluated questions.

Evaluator 2R Questions 3R QuestionsEvaluator 1 5.76 5.94Evaluator 2 5.33 4.63Evaluator 3 1.36 1.13Average Score 4.15 3.89

Table 4: Quantitative feedback of question qualityfrom the second study, scored on a scale from 1to 7. 2R Questions are created from two relation-ships, 3R from three relationships.

could potentially be biased (either positively ornegatively) when evaluating. However, becausethe generated questions are a new type of multiple-choice question, there is no naturally-arisinghuman-generated baseline.

5.4 Results

Quantitative results from the second study can beviewed in Table 4. Both 2R and 3R questionsachieved similar rankings, with no significant dif-ferences between the two (t-test, p=0.37). A his-togram with judges’ score distribution can be seenin Figure 3.

The qualitative results were analyzed and cate-gorized in Table 5 and are discussed in the nextsubsection.

5.5 Discussion

Compared to the previous study, the fixing ofgrammatical errors allowed us to better determinethe quality of the content of questions. While theevaluators did comment on the grammar of thequestions and suggested corrections a total of 37times, the quality of the content was able to be

evaluated.On the positive side, 77 questions were praised

as being clear and easy to understand (see Table 5).We believe this is due to a combination of the twochanges we made in response to the first study–improving the syntax and orienting the questionstowards more specific answers.

Additionally, one teacher commented that 10questions had particularly good properties. Twoexamples are: “What has a deoxyribose and oc-curs at the mitochondria? (DNA)” and “What in-cludes carbon, is a part of organic molecule, andincludes oxygen? (CHNOPS).” The updated algo-rithm ensured that the chosen properties were lessvague and also not too specific.

In some cases, the evaluators had an explicitpositive response to questions’ logically groupingproperties within questions. One teacher specifi-cally pointed out that certain questions containedvaluable properties which guide the students to thecorrect answer, as in: “What contrasts with plantcell and has an organelle called rough ER? (ani-mal cell)”. This provides positive support for thegoal of this style of generating questions by in-clude multiple pieces of information from the on-tology.

The descriptive vocabulary of 8 questions wasalso pointed out by one evaluator, such as “Whatincludes glucose, includes deoxyribose, and is atype of sugar? (monosaccharide).” Since we uti-lize a human-created ontology to generate ques-tions, the nodes and relationships are often de-scriptive. Our generation method leverages this tocreate questions. Given an ontology with descrip-tive, precisely-worded relationships and nodes,questions generate via our method will reflect thediverse vocabulary.

On the negative side, one teacher was particu-larly skeptical of the ability of this method of ques-tions to prepare students for standardized testing.She pointed out that this factual question style didnot challenge students to think critically. Whilethis is true, this is not the main focus of this work.We aim to over-generate simple practice questionsto ensure students have adequate materials to prac-tice and study with, before they have mastered aconcept.

Two teachers specifically mentioned that thestyle of questions were repetitive. The lack ofdiversity of questions is something which can beaddressed in future work. However, these stud-

309

Qualitative Feedback Evaluator 1 Evaluator 2 Evaluator 3 TotalClear and easy to understand 13 64 – 77Simple to answer 20 – – 20Poor wording – 3 16 19Does not prepare for standardized test – – 19 19Too many “What“ questions 17 – – 17Vague/Broad – 2 15 17Confusing/Poor properties chosen 1 4 10 15Too many options in question 9 1 1 11Good chosen properties 10 – – 10Small rewording suggestion – 8 2 10Ontology flaw pointed out – 5 4 9More precise rewording suggestion – 3 6 9Descriptive vocab 8 – – 8Detailed question 7 – – 7“Trivia” question – – 6 6Simple vocab 4 – – 4Too specific – – 4 4Not a critical thinking question – – 4 4Visual diagram needed – – 4 4Good intermediate steps to guide to correct answer 4 – – 4Alternate phrasing for ontology node – 2 1 3Academic language 2 – – 2Redundant concepts chosen 2 – – 2Poor format of multiple choice – – 2 2Too many similar questions 1 – – 1OK questions 1 – – 1Confusing property represented in the ontology – 1 – 1Should be a higher Bloom’s Taxonomy question – – 1 1

Table 5: Qualitative feedback from evaluators, categorized by type of comment.

ies show promise for using an ontology to informfactual question generation, and future work canextend this method to other question types.

Two of the teachers also pointed out parts of thegenerated questions that were scientifically incor-rect. Examining these questions shows that the on-tology contains incorrect information. This pointsto the necessity of validating and updating createdontologies. Our method, while generalizable toother ontologies, assumes the correctness of theinformation represented. Future work can exam-ine verifying ontologies, via methods such as dia-logue, parsing educational materials, or direct val-idation from experts.

In nine of the comments, the wording of ques-tions was stated as being imprecise. For instance,solid is linked to (CharacteristicOf, ”Solvent”). Itwas pointed out that while most solvents are solid,some can be liquids or gases. This underscores the

finding of the first study that errors in the ontologycan lead to errors in the questions.

It also seems that the selection of nodes to formquestions can be improved. Certain questionswere pointed out to have poor groupings of proper-ties. For instance, “What is produced by an ovaryand via fertilization creates an embryo? (egg)”wasthought not to be a good question, perhaps becauseto know either portions of the question one mustknow what an egg is.

6 Conclusion

We presented a novel way of generating complexmultiple choice questions and distractors from aneducational ontology. We showed significant im-provement when the ontology was used to gen-erate distractors compared to an embedding ap-proach. Insights gained from evaluation indicatea necessity of ontology augmentation and a more

310

advanced reasoning model.We also showed in a subsequent study that ques-

tion content, when adapted to account for poten-tially vague questions, has promising results. Fu-ture work may benefit from incorporating moreknowledge rich approaches such as Berant et al.’s(2014) work on deep analysis of biology texts.

Our algorithm for choosing properties to in-clude in questions and generating distractors isgeneralizable to other ontologies, although ourmethod assumes a near-complete ontology, as dis-tractors are generated via assumptions that theabsence of a link implies the absence of a rela-tionship. Changing the text-generating rules maybe necessary to generalize our approach as theseare tied to the specific relationships of our on-tology. Our ontology contains many naturally-worded relationships, which aided this process.Other text generation methods can be explored infuture work, as well, to rectify this.

Insights gained from the teachers’ qualitativefeedback are applicable to other question genera-tion methods as well. Future work should focus ongenerating increasingly more complex questionswhich focus on higher levels of Bloom’s taxon-omy. External knowledge not represented in theontology can be used to both increase the diffi-culty of the questions as well as improve simplermethods of question generation. Finally, verify-ing the completeness and accuracy of ontologiesas well as wording questions diversely and pre-cisely should be a focus going forward.

Acknowledgments

We thank Ann Peterson and the three Upwork sci-ence teachers for assessing the results of this anal-ysis, Yiyi Chen for coding support, as well asJonathan Gillick, David Bamman, Julian Park, andthe anonymous reviewers for helpful comments.This work was supported by a UC Berkeley Chan-cellor’s Fellowship.

References

Asma Ben Abacha, Julio Cesar Dos Reis, Yassine Mra-bet, Cedric Pruski, and Marcos Da Silveira. 2016.Towards natural language question generation forthe validation of ontologies and mappings. Journalof biomedical semantics 7(1):48.

Ali Benafia, Smaine Mazouzi, and Sara Benafia. 2015.Building ontologies from text corpora. In ICEMIS

’15 Proceedings of the The International Conferenceon Engineering MIS 2015.

Jonathan Berant, Vivek Srikumar, Pei-Chun Chen,Abby Vander Linden, Brittany Harding, BradHuang, Peter Clark, and Christopher D Manning.2014. Modeling biological processes for readingcomprehension. In EMNLP.

Benjamin S. Bloom, David Krathwohl, and Bertram B.Masia. 1956. Taxonomy of educational objectives:The classification of educational goals. HandbookI: Cognitive domain.. Green.

Jonathan C. Brown, Gwen A. Frishkoff, and MaxineEskenazi. 2005. Automatic question generation forvocabulary assessment. In Proceedings of the Con-ference on Human Language Technology and Em-pirical Methods in Natural Language Processing.HLT ’05, pages 819–826.

Kathleen Fisher. 2010. Biology lessons at sdsu.http://naturalsciences.sdsu.edu/.

Anatoly P. Getman and Volodymyr V. Karasiuk. 2014.A crowdsourcing approach to building a legal on-tology from text. Artificial Intelligence and Law22:313–335.

Arthur C. Graesser, Natalie Person, and John Huber.1992. Mechanisms that generate questions. InThomas W. Lauer, Eileen Peacock, and Arthur C.Graesser, editors, Questions and Information Sys-tems, Lawrence Erlbaum Associates, Inc., Hillsdale,New Jersey, chapter 9, pages 167–187.

Arthur C. Graesser, Vasile Rus, and Zhiqiang Cai.2012. Question answering and generation questionanswering and generation. In Applied Natural Lan-guage Processing: Identification, Investigation andResolution.

Michael Heilman. 2011. Automatic Factual QuestionGeneration from Text. Ph.D. thesis, Carnegie Mel-lon University.

Michael Heilman and Noah A. Smith. 2009. Ques-tion generation via overgenerating transformationsand ranking. Technical Report CMU-LTI-09-013,Carnegie Mellon University.

Michael Heilman and Noah A Smith. 2010. Goodquestion! statistical ranking for question generation.In Human Language Technologies: The 2010 An-nual Conference of the North American Chapter ofthe Association for Computational Linguistics. As-sociation for Computational Linguistics, pages 609–617.

Sathish Indurthi, Dinesh Raghu, Mitesh M. Khapra,and Sachindra Joshi. 2017. Generating naturallanguage question-answer pairs from a knowledgegraph using a rnn based question generation model.In ACL 2016.

311

Prashanth Mannem, Rashmi Prasad, and Aravind Joshi.2010. Question generation from paragraphs atupenn: Qgstec system description. In Proceedingsof QG2010: The Third Workshop on Question Gen-eration.

Karen Mazidi and Paul Tarau. 2016. Infusing nlu intoautomatic question generation. In INLG.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S.Corrado, and Jeffrey Dean. 2013. Distributed repre-sentations of words and phrases and their composi-tionality. CoRR abs/1310.4546.

Andrew Olney, Arthur C. Graesser, and Natalie K.Person. 2012a. Question generation from conceptmaps. DD 3:75–99.

Andrew M Olney, Whitney L Cade, and ClaireWilliams. 2011. Generating concept map exercisesfrom textbooks. In Proceedings of the 6th work-shop on innovative use of NLP for building educa-tional applications. Association for ComputationalLinguistics, pages 111–119.

Andrew McGregor Olney, Arthur C Graesser, and Na-talie K Person. 2012b. Question generation fromconcept maps. Dialogue & Discourse 3(2):75–99.

Shiyan Ou, Constantin Orasan, Dalila Mekhaldi, andLaura Hasler. 2008. Automatic question patterngeneration for ontology-based question answering.In FLAIRS Conference. pages 183–188.

Andreas Papasalouros, Konstantinos Kanaris, and Kon-stantinos Kotis. 2008. Automatic generation of mul-tiple choice questions from domain ontologies. Ine-Learning. pages 427–434.

Oleksandr Polozov, Eleanor O’Rourke, Adam M.Smith, Luke S. Zettlemoyer, Sumit Gulwani, andZoran Popovic. 2015. Personalized mathematicalword problem generation. In IJCAI.

Vasile Rus, Zhiqiang Cai, and Art Graesser. 2008.Question generation: Example of a multi-year eval-uation campaign. Proc WS on the QGSTEC .

Iulian Vlad Serban, Alberto Garcıa-Duran, CaglarGulcehre, Sungjin Ahn, Sarath Chandar, AaronCourville, and Yoshua Bengio. 2016. Generatingfactoid questions with recurrent neural networks:The 30m factoid question-answer corpus. In Pro-ceedings of the 54th Annual Meeting of the Associa-tion for Computational Linguistics. pages 588–598.

Sylvie Szulman, Nathalie Aussenac-Gilles, AdelineNazarenko, Henry Valery Teguiak, Eric Sardet, andJean Charlet. 2010. Dafoe: A platform for build-ing ontologies from texts. In EKAW 2010 Knowl-edge Engineering and Knowledge Management bythe Masses.

Xuchen Yao, Gosse Bouma, and Yi Zhang. 2012.Semantics-based question generation and imple-mentation. DD 3:11–42.

312