Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processingand the 9th International Joint Conference on Natural Language Processing, pages 1618–1629,Hong Kong, China, November 3–7, 2019. c©2019 Association for Computational Linguistics

1618

Asking Clarification Questions in Knowledge-Based Question Answering

Jingjing Xu1∗, Yuechen Wang2, Duyu Tang3, Nan Duan3, Pengcheng Yang1,Qi Zeng1, Ming Zhou3, Xu Sun1

1 MOE Key Lab of Computational Linguistics, School of EECS, Peking University2 University of Science and Technology of China

3 Microsoft Research Asia, Beijing, China{jingjingxu,yang pc,pkuzengqi,xusun}@pku.edu.cn

wyc9725@mail.ustc.edu.cn{dutang,nanduan,mingzhou}@microsoft.com

Abstract

The ability to ask clarification questions isessential for knowledge-based question an-swering (KBQA) systems, especially for han-dling ambiguous phenomena. Despite itsimportance, clarification has not been wellexplored in current KBQA systems. Fur-ther progress requires supervised resources fortraining and evaluation, and powerful modelsfor clarification-related text understanding andgeneration. In this paper, we construct a newclarification dataset, CLAQUA, with nearly40K open-domain examples. The dataset sup-ports three serial tasks: given a question, iden-tify whether clarification is needed; if yes,generate a clarification question; then predictanswers base on external user feedback. Weprovide representative baselines for these tasksand further introduce a coarse-to-fine modelfor clarification question generation. Experi-ments show that the proposed model achievesbetter performance than strong baselines. Thefurther analysis demonstrates that our datasetbrings new challenges and there still remainseveral unsolved problems, like reasonableautomatic evaluation metrics for clarificationquestion generation and powerful models forhandling entity sparsity.1

1 Introduction

Clarification is an essential ability for knowledge-based question answering, especially when han-dling ambiguous questions (Demetras et al.,1986). In real-world scenarios, ambiguity is acommon phenomenon as many questions are notclearly articulated, e.g. “What are the languagesused to create the source code of Midori?” in Fig-ure 1. There are two “Midori” using different pro-gramming languages, which confuses the system.

∗ The work was done while Jingjing Xu and YuechenWang were interns in Microsoft Research, Asia.

1The dataset and code will be released at https://github.com/msra-nlc/MSParS_V2.0

What are the languages used to create the source code of Midori?

I mean the first one.

When you say the source code language used in the program Midori, are you referring to web browser Midori or operating system Midori?

C.

Midori

Type

Description

computer.web_browsercomputer.software…

Midori is a free and open-source light-weight web browser....

Midori

Type

Description

computer.operating_systemcomputer.software…

Midori was the code name for a managed code operating system being developed…

Language C Language M#

Figure 1: An example of a clarification question inKBQA. There are two entities named “Midori” usingdifferent programming languages, which confuses thesystem.

For these ambiguous or confusing questions, it ishard to directly give satisfactory responses unlesssystems can ask clarification questions to confirmthe participant’s intention. Therefore, this workexplores how to use clarification to improve cur-rent KBQA systems.

We introduce an open-domain clarification cor-pus, CLAQUA, for KBQA. Unlike previousclarification-related datasets with limited anno-tated examples (De Boni and Manandhar, 2003;Stoyanchev et al., 2014) or in specific domains (Liet al., 2017; Rao and III, 2018), our dataset cov-ers various domains and supports three tasks. Thecomparison of our dataset with relevant datasets isshown in Table 1. Our dataset considers two kindsof ambiguity for single-turn and multi-turn ques-tions. In the single-turn case, an entity name refersto multiple possible entities in a knowledge basewhile the current utterance lacks necessary iden-tifying information. In the multi-turn case, am-biguity mainly comes from the omission where apronoun refers to multiple possible entities fromthe previous conversation turn. Unlike CSQA

1619

Dataset Domain Size Task

De Boni and Manandhar (2003) Open domain 253 Clarification question generation.Stoyanchev et al. (2014) Open domain 794 Clarification question generation.

Li et al. (2017) Movie 180K Learning to generate responses based on previous clar-ification questions and user feedback in dialogue.

Guo et al. (2017) Synthetic 100K Learning to ask clarification questions in reading com-prehension.

Rao and III (2018) Operating system 77K Ranking clarification questions in an online QA forum,StackExchange2.

CLAQUA (Our dataset) Open domain 40KClarification in KBQA, supporting clarification iden-tification, clarification question generation, andclarification-based question answering.

Table 1: The comparison of our dataset with relevant datasets. The first and second datasets focus on generatingclarification questions while the small size limits their applications. The middle three tasks are either not designedfor KBQA or limited in specific domains. Unlike them, we present an open-domain dataset for KBQA.

dataset (Saha et al., 2018) that constructs clarifica-tion questions base on predicate-independent tem-plates, our clarification questions are predicate-aware and more diverse. Based on the clarifica-tion raising pipeline, we formulate three tasks inour dataset, including clarification identification,clarification question generation, and clarification-based question answering. These tasks can natu-rally be integrated into existing KBQA systems.Since asking too many clarification questions sig-nificantly lowers the conversation quality, we firstdesign the task of identifying whether clarificationis needed. If no clarification is needed, systemscan directly respond. Otherwise, systems need togenerate a clarification question to request moreinformation from the participant. Then, externaluser feedback is used to predict answers.

Our main contribution is constructing a newclarification dataset for KBQA. To build high-quality resources, we elaborately design a data an-notation pipeline, which can be divided into threesteps: sub-graph extraction, ambiguous ques-tion annotation, and clarification question anno-tation. We design different annotation interfacesfor single-turn and multi-turn cases. We first ex-tract “ambiguous sub-graphs” from a knowledgebase as raw materials. To enable systems to per-form robustly across domains, the sub-graphs areextracted from an open-domain KB, covering do-mains like music, tv, book, film, etc. Based on thesub-graphs, annotators are required to write am-biguous questions and clarification questions.

We also contribute by implementing representa-tive neural networks for the three tasks and furtherdeveloping a new coarse-to-fine model for clari-fication question generation. Take multi-turn asan example. In the task of clarification identifica-

tion, the best performing system obtains an accu-racy of 86.6%. For clarification question genera-tion, our proposed coarse-to-fine model achievesa BLEU score of 45.02, better than strong base-line models. For clarification-based question an-swering, the best accuracy is 74.7%. We also con-duct a detailed analysis for three tasks and find thatour dataset brings new challenges that need to befurther explored, like reasonable automatic eval-uation metrics and powerful models for handlingthe sparsity of entities.

2 Data Collection

The construction process consists of three steps,sub-graph extraction, ambiguous question annota-tion, and clarification question annotation. We de-sign different annotation interfaces for single-turnand multi-turn cases.

2.1 Single-Turn Annotation

Sub-graph Extraction. As shown in Figure 2,we extract ambiguous sub-graphs from an open-domain knowledge base3, like FreeBase. For sim-plification, we set the maximum number of am-biguous entities to 2. In single-turn cases, we fo-cus on shared-name ambiguity where two entitieshave the same name and there is a lack of neces-sary distinguishing information. To construct suchcases, we extract two entities sharing the same en-tity name and the same predicate. Predicate repre-sents the relation between two entities. The sub-graphs provide reference for human annotators towrite diverse ambiguous questions based on theshared predicates.

2An online QA community.3To persevere anonymity, the name of the used KB is hid-

den for now. We will release our dataset and codes.

1620

Midori

Midori

C

M#

Computer.software.

language_used

Computer.software.

language_used

Figure 2: An extracted ambiguous sub-graph inthe single-turn case. Two entities share the samename “Midori” and the same predicate “Com-puter.software,language used”.

Shared Name MidoriPredicate # 1 computer.software.language usedPredicateDescription

A property relating an instanceof computer.software to a pro-gramming language that was usedto create the source code for thesoftware.

Qa: What are the languages used to create the sourcecode of Midori?

Table 2: An ambiguous question annotation example inthe single-turn case.

Ambiguous Question Annotation. In this step,the input is a table listing the shared entity name,predicate name, and predicate description. Basedon this table, annotators need to write ambiguousquestions, e.g., “What are the languages used tocreate the source code of Midori?”. An annotationexample is shown in Table 2. For diversity, anno-tators are encouraged to paraphrase the intentionwords in the predicate description.

Clarification Question Annotation. Based onentities and the annotated ambiguous question, an-notators are required to summarize distinguish-ing information and write a multi-choice clarifi-cation question with a special marker for sepa-rating entity information and pattern information,e.g., “When you say the source code languageused in the program Midori, are you referring to[web browser Midori] or [operating system Mi-dori]?”. Annotators are asked to write predicate-aware clarification questions instead of generalquestions, like “Which one do you mean, A or B?”or “do you mean A?” as adopted in (Saha et al.,2018). An example is shown in Table 3.

We use multi-choice as our basic type of clar-ification. There are three possible clarificationquestion types, including zero-choice type (e.g., Idon’t understand, can you provide more details? ),single-choice type (e.g., Do you mean A? ), andmulti-choice type (e.g., Which one do you mean,A or B? ). The zero-choice type means that the sys-tem does not understand the question and expects

E1 Name MidoriE1 Type computer.software

computer.web browsermedia common.creative work... ...

E1 Description Midori is a free and open-sourcelight-weight web browser. It usesthe WebKit rendering engine and theGTK+ 2 or GTK+ 3 ... ...

E2 Name MidoriE2 Type computer.operating system

computer.softwareE2 Description Midori was the code name for a man-

aged code operating system beingdeveloped by Microsoft with jointeffort of Microsoft Research... ...

Qa What are the languages used to cre-ate the source code of Midori?

Qc: When you say the source code language used inthe program Midori, are you referring to [web browserMidori] or [operating system Midori]?

Table 3: A clarification question annotation example inthe single-turn case.

more details from the participant. Though simple,it costs more user efforts as it pushes the partic-ipant to figure out what confuses the system. Incomparison, the advantage of single-choice typelies in the control of conversation direction. How-ever, if the system cannot provide user-expectedchoices, conversation may be longer. As for themulti-choice type, its advantage is less conversa-tion turns to ask for more valuable informationwhile it requires more annotation work.

2.2 Multi-Turn Annotation

Sub-graph Extraction. In multi-turn cases, am-biguity comes from the omission of the target en-tity name from the previous conversation turn. Wefirst extract two connected entities. If they alsoshare another predicate, we extract all related in-formation as an ambiguous sub-graph, as shown inFigure 3. By asking the shared predicate, we getan ambiguous question.

Lineodes

caracasia

Lineodes

Species

Genes

organism_classification

organism_classification

higher_classification

Figure 3: An extracted ambiguous sub-graph inthe multi-turn case. “Lineodes caracasia” and “Li-neodes” are linked and share the same relation “organ-ism classification”.

1621

E1 Name Lineodes caracasiaE2 Name LineodesPredicate #1 biology.higher classificationPredicate Description Property relating an biol-

ogy.organism classificationto its higher or parent biol-ogy.organism classification.

Predicate #2 biology.organism classificationPredicate Description Relating an biol-

ogy.organism to its biol-ogy.organism classification,the formal or informal tax-onomy grouping for theorganism.

Qh: What is the higher classification for Lineodescaracasia?Rh: Lineodes.Qa: What is the biological classification?

Table 4: An ambiguous question annotation example inthe multi-turn case.

E1 Name Lineodes caracasiaE1 Type media common.cataloged instance

biology.organism classification... ...

E1 Description Lineodes caracasia is a moth in thefamily Crambidae... ...

E2 Name LineodesE2 Type media common.cataloged instance

biology.organism classificationE2 Description Lineodes is a genus of moths of the

Crambidae family. ... ...Qh What is the higher classification for

Lineodes caracasia?Rh Lineodes.Qa What is the biological classification?Qc: Are you referring to [Lineodes caracasia] or Li-neodes], when you ask the biological classification?

Table 5: A clarification question annotation example inthe multi-turn case.

Ambiguous Question Annotation. An annota-tion example is shown in Table 4. Based on twoentities in the extracted sub-graph, annotators con-struct a conversation turn and then write an am-biguous question where the entity name is omitted,e.g, “What is the biological classification?”.

Clarification Question Annotation. The anno-tation guideline of this step is the same as insingle-turn annotation. The input includes twocandidate entities and the annotated ambiguousquestion. The output is a clarification question.An annotation example is shown in Table 5.

3 Tasks

Our dataset aims to enable systems to ask clarifi-cation questions in open-domain question answer-ing. Three tasks are designed in this work, includ-

music13.8%

tv 13.2%

book13.2%

film12.4%

traffic10.3%

people9.3%

sports7.4%

geography7.4%

location6.9%

organization6.1%

Figure 4: The distribution of top-10 domains.

ing clarification identification, clarification ques-tion generation, and clarification-based questionanswering. Figure 4 shows the distribution of thetop-10 domains in our dataset.

3.1 Clarification IdentificationClarification identification can be regarded as aclassification problem. It takes conversation con-text and candidate entity information as input. Theoutput is a label identifying whether clarificationis needed. Specifically, the input is {Qa, E1, E2}for single-turn cases or {Qp, Rp, Qa, E1, E2} formulti-turn cases. Qa represents the current ques-tion. E1 and E2 represent the candidate entities.Qp and Rp are question and response from the pre-vious conversation turn. The output is a binary la-bel from set Y = {0, 1} where 1 indicates that theinput question is ambiguous and 0 indicates theopposite.

- Which film does David Wills

act?

- Which city does river Nile

go through?

- Cairo

- Where is its mouth place?

David Wills

David Wills

Sonic Outlawsactor

Nile

Cairo

Ugandamouth place

basin_city

mouth place

actor

Figure 5: Negative examples of unambiguous sub-graphs and annotated conversations. The dotted linemeans the non-existent relation. The upper figure is inthe single-turn case and the lower one is in the multi-turn case.

As Figure 5 shows, negative examples are anno-tated in the same way as the positive (ambiguous)examples do, but without the clarification relatedsteps. In their sub-graphs, one of the entities hasits unique predicate. The unique predicates areincluded in the user questions like “Where is itsmouth place?”. As only river “Nile” has mouthplaces, this question is unambiguous.

1622

Single-Turn Multi-TurnPositive Negative Positive Negative

Train 8,139 6,841 12,173 8,289Dev 487 422 372 601Test 637 673 384 444

Table 6: Statistics of the dataset. It is important to notethat three tasks share the same split. Since clarifica-tion question generation and clarification-based ques-tion answering are built upon ambiguous situations,they only use the positive data in their training, devel-opment, and test sets.

The statistics of our dataset are shown in Ta-ble 6. It is important to note that three tasks sharethe same split. Since clarification question gen-eration and clarification-based question answeringare built upon ambiguous situations, they only usethe positive (ambiguous) data in their training, de-velopment, and test sets. For generalization, weadd some examples with unseen entities and pred-icates into the development and test sets.

3.2 Clarification Question Generation

This text generation task takes ambiguous con-text and entity information as input, and then out-puts a clarification question. For single-turn cases,the input is {Qa, E1, E2}. For multi-turn cases,the input is {Qp, Rp, Qa, E1, E2}. In both cases,the output is a clarification question Qc. We useBLEU as the automatic evaluation metric.

3.3 Clarification-Based Question Answering

As our dataset focuses on simple questions, thistask can be simplified as the combination of entityidentification and predicate identification. Entityidentification is to extract entity e from candidateentities based on current context. Predicate identi-fication is to choose predicate p that describes theambiguous question. The extracted entity e andpredicate p are combined to query the knowledgetriple (e, p, o). The model is correct when bothtasks are successfully completed.

Additional user responses toward clarificationquestions are also necessary for this task. Due tothe strong pattern in responses, we use a template-based method to generate the feedback. We firstdesign four templates, “I mean the first one”, “Imean the second one”, “I mean [entity name]”, “Imean [entity type] [entity name]”. Then for eachclarification example, we randomly select a candi-date entity and fill its information into the templateas the user response, e.g., “I mean web browserMidori.” for the clarification question in Table 3.

For entity identification, we use the selected entityas the gold label. For predicate prediction, we usethe predicate in the current ambiguous question asthe gold label. In muli-turn cases, the predicate la-bel is Predicate #1 as shown in Table 2. In single-turn cases, the predicate label is Predicate #2 asshown in Table 4.

Entity identification and predicate identifica-tion are both classification tasks. They takethe same information as input, including am-biguous context, clarification question, and userresponse. For single-turn cases, the inputis {Qa, E1, E2, Qc, Rc} where Rc is the feed-back of the participant toward clarification ques-tion Qc. For multi-turn cases, the input is{Qp, Rp, Qa, E1, E2, Qc, Rc}. For entity identi-fication, the output is a label from set YE ={E1, E2}. For predicate identification, we ran-domly choose 1 negative predicates from the train-ing set and combine them with the gold predicatetogether as the candidate set YP . The output ofpredicate identification is the index of the goldpredicate in YP .

4 Approach

For three tasks, we implement several representa-tive neural networks as baselines. Here we intro-duce these models in detail.

4.1 Classification Identification Models

The input of classification models is the ambigu-ous context and entity information. For simpli-fication, we use a special symbol, [S], to con-catenate all the entity information together. Thesetwo parts can be regarded as different source in-formation. Based on whether the inter-relationbetween different source information is explicitlyexplored, the classification models can be classi-fied into two categories, unstructured models andstructured models.

Unstructured models concatenate differentsource inputs into a long sequence with a spe-cial separation symbol [SEL]. We implementseveral widely-used sequence classification mod-els, including Convolutional Neural Network(CNN) (Kim, 2014), Long-Short Term MemoryNetwork (LSTM) (Schuster and Paliwal, 1997),and Recurrent Convolutional Neural Network(RCNN) (Lai et al., 2015), Transformer. Thedetails of the models are shown in SupplementaryMaterials.

1623

Decoder

Decoder

Encoder

Encoder

Entity Rendering Module

Template Generating Module

What are the languages

used to create the source

code of Midori?

Entity Info

When you say the source code language used in the program Midori, are you referring to [A] or [B]?

web browser Midori

operating system Midori

Decoder

Hidden vector of [B]

Hidden vector of [A]

TemplateAmbiguous Context

[A] [B]

Figure 6: An illustration of the proposed coarse-to-finemodel. The proposed model consists of a template gen-erating module and an entity rendering module. Theformer is used to generate a clarification template basedon ambiguous question, e.g., “When you say the sourcecode language used in the program Midori, are you re-ferring to [A] or [B]?”. The latter is used to fill up thegenerated template with detailed entity information.

Structured models use separate structures to en-code different source information and adopt anadditional structure to model the inter-relation ofthe source information. Specifically, we use tworepresentative neural networks, Hierarchical At-tention Network (HAN) (Yang et al., 2016) andDynamic Memory Network (DMN) (Kumar et al.,2016), as our structured baselines. The details ofthe models are shown in Supplementary Materials.

4.2 Clarification Question GenerationModels

The input of the generation model is the ambigu-ous context and entity information. In single-turncases, the ambiguous context is current questionQa. In multi-cases, the input is current questionand the previous conversation turn {Qp, Rp, Qa}.We use [S] to concatenate all entity informa-tion together, and use [SEL] to concatenate entityinformation and context information into a longsequence. We adopt Seq2Seq (Bahdanau et al.,2015) and Transformer (Vaswani et al., 2017) asour baselines and further develop a new genera-tion model. The details of baselines can be foundin Supplementary Materials.

Coarse-to-fine Model. Generally speaking, aclarification question contains two parts, entityphrases (e.g., “web browser Midori”) and patternphrases ( e.g. “When you say the source code lan-guage used in the program Midori, are you refer-ring to [A] or [B]?” ). The entity phrase is summa-rized from the given entity information for distin-guishing between two entities. The pattern phrase

is used to locate the position of ambiguity, whichis closely related with the context. In summary,two kinds of phrases refer to different source in-formation. Based on this feature, we propose anew coarse-to-fine model, as shown in Figure 6.Similar ideas have been successfully applied to se-mantic parsing (Dong and Lapata, 2018).

The proposed model consists of a template gen-erating module Tθ and an entity rendering moduleRφ. Tθ first generates a template containing pat-tern phrases and the symbolic representation of theentity phrases. Then, the symbolized entities con-tained in the generated template are further prop-erly rendered by the entity rendering module Rφ toreconstruct complete entity information. Since theannotated clarification questions explicitly sepa-rate entity phrases and pattern phrases, we caneasily build training data for these two modules.For clarity, the template is constructed by replac-ing entity phrases in a clarification question withspecial symbols, [A] and [B], which representthe positions of two entities.

The template generating module Tθ uses Trans-former (Vaswani et al., 2017) as implementation.The input is the ambiguous context and the outputis a clarification template.

Similar to Tθ, the entity rendering module Rφ

is also implemented with a Transformer structure.More specifically, for each symbolized entity inthe template, we add the hidden state of the de-coder of Tθ corresponding to this symbolized en-tity into the decoder input.

4.3 Clarification-Based Question AnsweringModels

This task contains two sub-tasks, entity identifi-cation and predicate identification. They sharethe same input. In single-turn cases, the input is{Qa, Qc, Rc, E1, E2}. In multi-cases, the input is{Qp, Rp, Qa, Qc, Rc, E1, E2}. Considering thesetwo sub-tasks both belong to classification tasks,we use the same models as used in the task of clar-ification identification for implementation.

5 Experiments

We report the experiment results on three formu-lated tasks and give a detailed analysis. Moredetails of hyper-parameters are in SupplementaryMaterials.

1624

Model Single-Turn Multi-TurnCNN 83.3 80.0RNN 83.8 72.7Transformer 84.0 72.8HAN 83.1 73.0DMN 86.6 80.5

Table 7: Results of clarification identification models.

5.1 Experiment Results

Clarification Identification. As shown in Ta-ble 7, the structured models have the best perfor-mance, with a 86.6% accuracy on single-turn casesand a 80.5% accuracy on multi-turn cases. Com-pared to the best-performing unstructured model,the structured architecture brings obvious perfor-mance improvements, with 2.6% and 6.7% accu-racy increases. It indicates that the structured ar-chitectures have learned the inter-relation betweenthe context and entity details, which is a vital partfor reasoning whether a question is ambiguous.We also find that the models all achieve better re-sults on multi-turn cases. Multi-turn cases containadditionally previous conversation turn, which canhelp the models capture more key phrases from theentity information.

Clarification Question Generation. Table 8shows that Seq2Seq achieves low BLEU scores,which indicates its tendency to generate irrelevanttext. Transformer achieves higher performancethan Seq2Seq. Our proposed coarse-to-fine modeldemonstrates a new state of the art, improving thecurrent highest baseline result by 3.35 and 0.60BLEU scores, respectively.

We also find that multi-turn cases generally ob-tain higher BLEU scores than single-turn cases.In single-turn cases, the same-name characteris-tic makes it hard to summarize key distinguish-ing information. As a comparison, in multi-turncases, two candidate entities usually have differ-ent names and it is easier to generate clarificationquestions. It is due to the flexible structure of am-biguous questions, which makes it hard to defineuniversal rules to identify the asked objects. Ta-ble 9 shows a an example generated by the pro-posed model.

Clarification-Based Question Answering. AsTable 10 shows, the unstructured models performbetter than the structured models, indicating thatthis task tends to be over-fitting and small modelshave better the generalization ability.

Model Single-Turn Multi-TurnSeq2Seq 18.84 31.62Transformer 20.69 44.42The proposed Model 24.04 45.02

Table 8: Results of clarification question generationmodels.

E1 Name Lineodes interruptaE1 Type biology.organism classificationE1 Description Lineodes interrupta is a moth in the

Crambidae family. It was describedby Zeller in 1873. It is found inMexico and the United States, whereit has been recorded from Florida,Louisiana, Oklahoma and Texas.

E2 Name LineodesE2 Type biology.organism classificationE2 Description Lineodes is a genus of moths of the

Crambidae family.Qp What is higher classification of Li-

neodes interrupta?Rp Lineodes.Qa Biologically speaking, what is the

classification?Output: Are you referring to Lineodes interrupta orLineodes, when you ask the biological classification?

Table 9: An example of the generated clarificationquestions.

5.2 DiscussionAutomatic Evaluation. In the task of clarifica-tion question generation, we use BLEU, a widelyused evaluation metric on language generationtasks, as the automatic evaluation metric. BLEUevaluates the generated text by computing its sim-ilarity with the gold text. However, we find thismetric not suitable for our task. A good answercan ask different aspects as long as it can distin-guish the entities. The given answer in our datasetis one of possible answers but not the only cor-rect answer. A good answer may get a low BLEUscore. Therefore, a more reasonable evaluationmetric needs to be explored in the future, likeparaphrase-based BLEU.

Error Analysis. In clarification question gener-ation, although our proposed model has achievedthe best performance, it still generates some low-quality questions. We conduct a human evaluationon 100 generated clarification questions and sum-marize two error categories. The first one is theentity error where the generated clarification ques-tion has a correct multi-choice structure but withirrelevant entity information. Here we present oneexample in Table 11. In this example, our modelgenerates entity phrases irrelevant to the input am-

1625

Model Single-Turn Multi-TurnCNN 60.8 74.7RNN 56.4 76.0Transformer 52.1 58.9DMN 50.9 54.5HAN 51.7 54.7

Table 10: Results of clarification-based question an-swering models.

E1 Name Come Back, Little ShebaE1 Type award.winning work

award.nominated work... ...

E1 Description Come Back, Little Sheba is a 1950theater production of the play byWilliam Inge... ...

E2 Name Come Back, Little ShebaE2 Type theater.production

award.nominated workE2 Description Come Back, Little Sheba is a 2008

theater production of the play byWilliam Inge.... ...

Qh: Who directed Come Back, Little Sheba?Output: Which one do you mean, winning work visualartist Claudia Coleman or winning work Elmer Gantry,when you ask the directors?

Table 11: A bad generated case.

biguous question. This failure is mainly due to theunseen entity name “Come Back, Little Sheba”,which leads the model to wrong generation de-cisions. Since there are lots of entities with dif-ferent names and descriptions, how to handle thesparse information is a non-trivial problem. Thesecond one is the grammar error where modelssometimes generate low-quality phrases after gen-erating a low-frequency entity word. These errorsboth can be attributed to the sparsity of entities tosome extent. How to deal with the sparsity of en-tity information still needs further exploration.

6 Related Work

Our work is related to clarification question andquestion generation.

6.1 Clarification Question in Other Tasks

There are several studies on asking clarificationquestions. Stoyanchev et al. (2014) randomlydrop one phrase from a question and require an-notators to ask a clarification question toward thedropped information, e.g.“Do you know the birthdata of XXX”. However, the small dataset sizemakes it hard to help downstream tasks. Follow-ing this work, Guo et al. (2017) provide a largersynthetic dataset QRAQ by replacing some of en-tities with variables. Li et al. (2017) use mis-

spelled words to replace entities in questions tobuild ambiguous questions. Although these stud-ies are good pioneering studies, the synthetic con-structing way makes them unnatural and far fromreal world questions.

Different from these studies, Braslavski et al.(2017) investigate a community question answer-ing (CQA) dataset and study how to predict thespecific subject of a clarification question. Simi-larly, Rao and III (2018) focus on learning to rankhuman-written clarification questions in an onlineQA forum, StackExchange. Our work differs fromthese two studies in that we have a knowledgegraph at the backend, and the clarification-relatedcomponents are able to facilitate KBQA.

6.2 Question Generation

For different purposes, there are various questiongeneration tasks. Hu et al. (2018) aim to ask ques-tions to play the 20 question game. Dhingra et al.(2017) teach models to ask questions to limit thenumber of answer candidates in task-oriented dia-logues. Ke et al. (2018) train models to ask ques-tions in open-domain conversational systems tobetter interact with people. Guo et al. (2018) de-velop a sequence-to-sequence model to generatenatural language questions.

7 Conclusion and Future Work

In this work, we construct a clarification ques-tion dataset for KBQA. The dataset supportsthree tasks, clarification identification, clarifica-tion question generation, and clarification-basedquestion answering. We implement representa-tive neural networks as baselines for three tasksand propose a new generation model. The de-tailed analysis shows that our dataset brings newchallenges. More powerful models and reasonableevaluation metrics need further explored.

In the future, we plan to improve our datasetby including more complex ambiguous situationsfor both single-turn and multi-turn questions, suchas multi-hop questions, aggregation questions, etc.We also plan to integrate the clarification-basedmodels into existing KBQA system, and studyhow to iteratively improve the model based on hu-man feedback.

Acknowledgments

We thank all reviewers for providing the thought-ful and constructive suggestions. This work was

1626

supported in part by National Natural ScienceFoundation of China (No. 61673028). JingjingXu and Xu Sun are the corresponding authors ofthis paper.

ReferencesDzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-

gio. 2015. Neural machine translation by jointlylearning to align and translate. In 3rd Inter-national Conference on Learning Representations,ICLR 2015, San Diego, CA, USA, May 7-9, 2015,Conference Track Proceedings.

Pavel Braslavski, Denis Savenkov, Eugene Agichtein,and Alina Dubatovka. 2017. What do you mean ex-actly?: Analyzing clarification questions in cqa. InProceedings of the 2017 Conference on ConferenceHuman Information Interaction and Retrieval, pages345–348. ACM.

Marco De Boni and Suresh Manandhar. 2003. An anal-ysis of clarification dialogue for question answering.In Proceedings of the 2003 Conference of the NorthAmerican Chapter of the Association for Computa-tional Linguistics on Human Language Technology-Volume 1, pages 48–55. Association for Computa-tional Linguistics.

Marty J Demetras, Kathryn Nolan Post, and Cather-ine E Snow. 1986. Feedback to first language learn-ers: The role of repetitions and clarification ques-tions. Journal of child language, 13(2):275–292.

Bhuwan Dhingra, Lihong Li, Xiujun Li, Jianfeng Gao,Yun-Nung Chen, Faisal Ahmed, and Li Deng. 2017.Towards end-to-end reinforcement learning of dia-logue agents for information access. In Proceedingsof the 55th Annual Meeting of the Association forComputational Linguistics, ACL 2017, Vancouver,Canada, July 30 - August 4, Volume 1: Long Papers,pages 484–495.

Li Dong and Mirella Lapata. 2018. Coarse-to-fine de-coding for neural semantic parsing. arXiv preprintarXiv:1805.04793.

Daya Guo, Yibo Sun, Duyu Tang, Nan Duan, Jian Yin,Hong Chi, James Cao, Peng Chen, and Ming Zhou.2018. Question generation from sql queries im-proves neural semantic parsing. In Proceedings ofthe 2018 Conference on Empirical Methods in Nat-ural Language Processing, pages 1597–1607. Asso-ciation for Computational Linguistics.

Xiaoxiao Guo, Tim Klinger, Clemens Rosenbaum,Joseph P Bigus, Murray Campbell, Ban Kawas,Kartik Talamadupula, Gerry Tesauro, and SatinderSingh. 2017. Learning to query, reason, and answerquestions on ambiguous texts. In ICLR 2017.

Huang Hu, Xianchao Wu, Bingfeng Luo, ChongyangTao, Can Xu, Wei Wu, and Zhan Chen. 2018. Play-ing 20 question game with policy-based reinforce-ment learning. arXiv preprint arXiv:1808.07645.

Pei Ke, Jian Guan, Minlie Huang, and Xiaoyan Zhu.2018. Generating informative responses with con-trolled sentence function. In Proceedings of the56th Annual Meeting of the Association for Com-putational Linguistics, ACL 2018, Melbourne, Aus-tralia, July 15-20, 2018, Volume 1: Long Papers,pages 1499–1508.

Yoon Kim. 2014. Convolutional neural networks forsentence classification. In Proceedings of the 2014Conference on Empirical Methods in Natural Lan-guage Processing, EMNLP 2014, October 25-29,2014, Doha, Qatar, A meeting of SIGDAT, a SpecialInterest Group of the ACL, pages 1746–1751.

Ankit Kumar, Ozan Irsoy, Peter Ondruska, MohitIyyer, James Bradbury, Ishaan Gulrajani, VictorZhong, Romain Paulus, and Richard Socher. 2016.Ask me anything: Dynamic memory networks fornatural language processing. In International Con-ference on Machine Learning, pages 1378–1387.

Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. 2015.Recurrent convolutional neural networks for textclassification. In AAAI, volume 333, pages 2267–2273.

Jiwei Li, Alexander H. Miller, Sumit Chopra,Marc’Aurelio Ranzato, and Jason Weston. 2017.Dialogue learning with human-in-the-loop. In 5thInternational Conference on Learning Representa-tions, ICLR 2017, Toulon, France, April 24-26,2017, Conference Track Proceedings.

Sudha Rao and Hal Daume III. 2018. Learning to askgood questions: Ranking clarification questions us-ing neural expected value of perfect information. InProceedings of the 56th Annual Meeting of the As-sociation for Computational Linguistics, ACL 2018,Melbourne, Australia, July 15-20, 2018, Volume 1:Long Papers, pages 2736–2745.

Amrita Saha, Vardaan Pahuja, Mitesh M Khapra,Karthik Sankaranarayanan, and Sarath Chandar.2018. Complex sequential question answering: To-wards learning to converse over linked question an-swer pairs with a knowledge graph. In Thirty-Second AAAI Conference on Artificial Intelligence.

Mike Schuster and Kuldip K Paliwal. 1997. Bidirec-tional recurrent neural networks. IEEE Transactionson Signal Processing, 45(11):2673–2681.

Svetlana Stoyanchev, Alex Liu, and Julia Hirschberg.2014. Towards natural clarification questions in di-alogue systems. In AISB symposium on questions,discourse and dialogue, volume 20.

Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N. Gomez, LukaszKaiser, and Illia Polosukhin. 2017. Attention is allyou need. In Advances in Neural Information Pro-cessing Systems 30: Annual Conference on NeuralInformation Processing Systems 2017, 4-9 Decem-ber 2017, Long Beach, CA, USA, pages 6000–6010.

1627

Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He,Alex Smola, and Eduard Hovy. 2016. Hierarchi-cal attention networks for document classification.In Proceedings of the 2016 Conference of the NorthAmerican Chapter of the Association for Computa-tional Linguistics: Human Language Technologies,pages 1480–1489.

1628

A Supplementary Materials

A.1 Classification Models

Model Hyper-parameter Setting

CNN

Batch size 64Embedding size 128Hidden size 128Filter size 128Kernel size 100Learning rate 0.0003Optimizer Adam

RNN

Batch size 8Embedding size 128Hidden size 128Learning rate 2Optimizer Adam

Transformer

Batch size 8Embedding size 128Hidden size 128Learning rate 2Dropout 0.1Optimizer Adam

HAN

Batch size 64Embedding size 100Hidden size 100Learning rate 0.01Optimizer Adam

DMN

Batch size 64Embedding size 300Hidden size 300Learning rate 0.005Optimizer Adam

Table 12: Settings of classification models.

Table 12 shows the detailed hyper-parametersettings of these classification models. Here weintroduce the classification models used in our ex-periments.

CNN. It first feeds the input embedding se-quence into a convolutional layer. The convolu-tionial layer contains K filters f ∈ Rs,d where sis the size of filters and d is the dimension of wordembeddings. We use different sizes of filters to getrich features. Then, a max pooling layer takes Kvectors as input and generates a distributed inputrepresentation h. Finally, the vector h is projectedto the probability of labels.

RNN. It is built on a traditional bi-directionalgated recurrent unit (GRU) structure, which isused to capture global and local dependencies in-side the input sequence. The last output of GRU isthen fed into the output layer to generate the prob-ability of labels.

Transformer. A model similar with RNNs. Itremoves the recurrent unit in RNNs and replaces it

Model Hyper-parameter Setting

Seq2Seq

Batch size 16Embedding size 32Hidden size 32Learning rate 2Beam size 1Dropout 0.1Optimizer Adam

Transformer

Batch size 16Embedding size 32Hidden size 32Learning rate 2Beam size 1Dropout 0.1Optimizer Adam

Our Model

Batch size 16Embedding size 32Hidden size 32Learning rate 2Beam size 1Dropout 0.1Optimizer Adam

Table 13: Settings of generation models.

with a special attention mechanism. called multi-head attention. The model is composed of a stackof 2 identical layers. Each identical layer has twosub-layers. The first sub-layer is a multi-head self-attention layer, and the second is a position-wisefully connected feed-forward layer. Since inputnodes are not sequentially in order, the positionalembedding layer is removed from the model. Thedecoder is also composed of a stack of 2 identicallayers.

HAN. It is built upon two self-attention layers.The first layer is responsible for encoding contextinformation and entity information. Then, the sec-ond layer uses a self-attention mechanism to learnthe inter-relations between them. Finally, the out-put of the second layer is fed into the output layerfor predicting labels.

DMN. This model regards context informationas query and entity information as input to learntheir inter-relations. It consists of four modules,an input module, a question module, an episodicmemory module, and an answer module. Theinput module and thed question module encodequery and input into distributed vector represen-tations. The episodic memory module decodeswhich parts of the entity information to focus onthrough the attention mechanism. Finally, the an-swer module generates an answer based on the fi-nal memory vector of the memory module.

1629

A.2 Clarification Question GenerationModels

Here is the detailed introduction about the genera-tion baselines.

Seq2Seq (Bahdanau et al., 2015). This modelis based on a traditional encoder-decoder frame-work which encodes the input sequence into adense vector and then decodes the target sequenceword by word. Attention is used to capture globaldependencies between input and output.

Transformer (Vaswani et al., 2017). It is basedsolely on attention mechanism to capture globaldependencies. Similar to Seq2Seq, it uses anencoder-decoder framework but with different im-plementation.

Table 13 shows the detailed hyper-parametersettings of generation models.