Undefined 0 (2016) 1–0 1IOS Press

Survey on Challenges of Question Answeringin the Semantic WebKonrad Höffner a, Sebastian Walter b, Edgard Marx a, Ricardo Usbeck a, Jens Lehmann a,Axel-Cyrille Ngonga Ngomo a

a Leipzig University, Institute of Computer Science, AKSW GroupAugustusplatz 10, D-04109 Leipzig, GermanyE-mail: {hoeffner,marx,lehmann,ngonga,usbeck}@informatik.uni-leipzig.deb CITEC, Bielefeld UniversityInspiration 1, D - 33615 Bielefeld, GermanyE-mail: swalter@techfak.uni-bielefeld.de

Abstract. Semantic Question Answering (SQA) removes two major access requirements to the Semantic Web: the mastery of aformal query language like SPARQL and knowledge of a specific vocabulary. Because of the complexity of natural language, SQApresents difficult challenges and many research opportunities. Instead of a shared effort, however, many essential components areredeveloped, which is an inefficient use of researcher’s time and resources. This survey analyzes 62 different SQA systems, whichare systematically and manually selected using predefined inclusion and exclusion criteria, leading to 72 selected publicationsout of 1960 candidates. We identify common challenges, structure solutions, and provide recommendations for future systems.This work is based on publications from the end of 2010 to July 2015 and is also compared to older but similar surveys.

Keywords: Question Answering, Semantic Web, Survey

1. Introduction

Semantic Question Answering (SQA) is defined byusers (1) asking questions in natural language (NL) (2)using their own terminology to which they (3) receive aconcise answer generated by querying an RDF knowl-edge base.1 Users are thus freed from two major ac-cess requirements to the Semantic Web: (1) the mas-tery of a formal query language like SPARQL and (2)knowledge about the specific vocabularies of the knowl-edge base they want to query. Since natural languageis complex and ambiguous, reliable SQA systems re-quire many different steps. For some of them, like part-of-speech tagging and parsing, mature high-precisionsolutions exist, but most of the others still present diffi-cult challenges. While the massive research effort hasled to major advances, as shown by the yearly Ques-tion Answering over Linked Data (QALD) evaluation

1Definition based on Hirschman and Gaizauskas [72].

campaign, it suffers from several problems: Instead ofa shared effort, many essential components are redevel-oped. While shared practices emerge over time, theyare not systematically collected. Furthermore, most sys-tems focus on a specific aspect while the others arequickly implemented, which leads to low benchmarkscores and thus undervalues the contribution. This sur-vey aims to alleviate these problems by systematicallycollecting and structuring methods of dealing with com-mon challenges faced by these approaches. Our con-tributions are threefold: First, we complement exist-ing work with 72 publications about 62 systems de-veloped from 2010 to 2015. Second, we identify chal-lenges faced by those approaches and collect solutionsfor them from the 72 publications. Finally, we drawconclusions and make recommendations on how to de-velop future SQA systems. The structure of the paperis as follows: Section 2 states the methodology usedto find and filter surveyed publications. Section 3 com-pares this work to older, similar surveys as well as eval-

0000-0000/16/$00.00 c© 2016 – IOS Press and the authors. All rights reserved

2 Challenges of Question Answering in the Semantic Web

uation campaigns and work outside the SQA field. Sec-tion 4 introduces the surveyed systems. Section 5 iden-tifies challenges faced by SQA approaches and presentsapproaches that tackle them. Section 6 summarizes theefforts made to face challenges to SQA and their impli-cation for further development in this area.

2. Methodology

This survey follows a strict discovery methodology;Objective inclusion and exclusion criteria are used tofind and restrict publications on SQA.

Inclusion Criteria Candidate articles for inclusion inthe survey need to be part of relevant conference pro-ceedings or searchable via Google Scholar (see Ta-ble 1). The included papers from the publication searchengine Google Scholar are the first 300 results in thechosen timespan (see exclusion criteria) that contain“’question answering’ AND (’Semantic Web’ OR ’dataweb’)” in the article including title, abstract and textbody. Conference candidates are all publications in ourexamined time frame in the proceedings of the ma-jor Semantic Web Conferences ISWC, ESWC, WWW,NLDB, and the proceedings which contain the annualQALD challenge participants.

Exclusion Criteria Works published before Novem-ber 20102 or after July 2015 are excluded, as well asthose that are not related to SQA, determined in a man-ual inspection in the following manner: First, proceed-ing tracks are excluded that clearly do not contain SQArelated publications. Next, publications both from pro-ceedings and from Google Scholar are excluded basedon their title and finally on their content.

Notable exclusions We exclude the following ap-proaches since they do not fit our definition of SQA(see Section 1): Swoogle [45] is independent of anyspecific knowledge base but instead builds its own in-dex and knowledge base using RDF documents foundby multiple web crawlers. Discovered ontologies areranked based on their usage intensity and RDF doc-uments are ranked using authority scoring. Swooglecan only find single terms and cannot answer naturallanguage queries and is thus not a SQA system. Wol-fram|Alpha is a natural language interface based on thecomputational platform Mathematica [138] and aggre-gates a large number of structured sources and a al-

2The time before is already covered in Cimiano and Minock [32].

gorithms. However, it does not support Semantic Webknowledge bases and the source code and the algorithmis are not published. Thus, we cannot identify whetherit corresponds to our definition of a SQA system.

Result The inspection of the titles of the GoogleScholar results by two authors of this survey led to 153publications, 39 of which remained after inspecting thefull text (see Table 1). The selected proceedings con-tain 1660 publications, which were narrowed down to980 by excluding tracks that have no relation to SQA.Based on their titles, 62 of them were selected and in-spected, resulting in 33 publications that were catego-rized and listed in this survey. Table 1 shows the num-ber of publications in each step for each source. In total,1960 candidates were found using the inclusion crite-ria in Google Scholar and conference proceedings andthen reduced using track names (conference proceed-ings only, 1280 remaining), then titles (214) and finallythe full text, resulting in 72 publications describing 62distinct SQA systems.

3. Related Work

This section gives an overview of recent QA andSQA surveys and differences to this work, as well asQA and SQA evaluation campaigns, which quantita-tively compare systems.

3.1. Other Surveys

QA Surveys Cimiano and Minock [32] present a data-driven problem analysis of QA on the Geobase dataset.The authors identify eleven challenges that QA hasto solve and which inspired the problem categories ofthis survey: question types, language “light”3, lexicalambiguities, syntactic ambiguities, scope ambiguities,spatial prepositions, adjective modifiers and superla-tives, aggregation, comparison and negation operators,non-compositionality, and out of scope4. In contrast toour work, they identify challenges by manually inspect-ing user provided questions instead of existing systems.Mishra and Jain [94] propose eight classification crite-ria, such as application domain, types of questions andtype of data. For each criterion, the different classifica-tions are given along with their advantages, disadvan-tages and exemplary systems.

3semantically weak constructions4cannot be answered as the information required is not contained

in the knowledge base

Challenges of Question Answering in the Semantic Web 3

Table 1Sources of publication candidates along with the number of publica-tions in total, after excluding based on conference tracks (I), based onthe title (II), and finally based on the full text (selected). Works thatare found both in a conference’s proceedings and in Google Scholarare only counted once, as selected for that conference. The QALD2 proceedings are included in ILD 2012, QALD 3 [24] and QALD4 [133] in the CLEF 2013 and 2014 working notes.

Venue All I II Selected

Google Scholar Top 300 300 300 153 39

ISWC 2010 [106] 70 70 1 1

ISWC 2011 [7] 68 68 4 3

ISWC 2012 [35] 66 66 4 2

ISWC 2013 [4] 72 72 4 0

ISWC 2014 [91] 31 4 2 0

WWW 2011 [139] 81 9 0 0

WWW 2012 [140] 108 6 2 1

WWW 2013 [141] 137 137 2 1

WWW 2014 [142] 84 33 3 0

WWW 2015 [143] 131 131 1 1

ESWC 2011 [6] 67 58 3 0

ESWC 2012 [121] 53 43 0 0

ESWC 2013 [33] 42 34 0 0

ESWC 2014 [108] 51 31 2 1

ESWC 2015 [59] 42 42 1 1

NLDB 2011 [96] 21 21 2 2

NLDB 2012 [21] 36 36 0 0

NLDB 2013 [89] 36 36 1 1

NLDB 2014 [90] 39 30 1 2

NLDB 2015 [16] 45 10 2 1

QALD 1 [130] 3 3 3 2

ILD 2012 [132] 9 9 9 3

CLEF 2013 [52] 208 7 6 5

CLEF 2014 [27] 160 24 8 6

Σ(conference) 1660 980 61 33

Σ(all) 1960 1280 214 72

SQA Surveys For each participant, problems and theirsolution strategies are given: Athenikos and Han [8]give an overview of domain specific QA systems forbiomedicine. After summarising the state of the artfor biomedical QA systems in 2009, the authors de-scribe different approaches from the point of view ofmedical and biological QA. In contrast to our survey,the authors do not sort the presented approaches bychallenges, but by more broader terms such as “Non-semantic knowledge base medical QA systems andapproaches” or “Inference-based biological QA sys-tems and approaches”. Lopez et al. [85] present anoverview similar to Athenikos and Han [8] but with a

Table 2Other surveys by year of publication. Surveyed years are given ex-cept when a dataset is theoretically analyzed. Approaches addressingspecific types of data are also indicated.

QA Survey Year Coverage Data

Cimiano and Minock [32] 2010 — geobaseMishra and Jain [94] 2015 2000–2014 general

SQA Survey Year Coverage Data

Athenikos and Han [8] 2010 2000–2009 biomedicalLopez et al. [85] 2010 2004–2010 generalFreitas et al. [56] 2012 2004–2011 generalLopez et al. [86] 2013 2005–2012 general

wider scope. After defining the goals and dimensionsof QA and presenting some related and historic work,the authors summarize the achievements of SQA sofar and the challenges that are still open. Another re-lated survey from 2012, Freitas et al. [56], gives a broadoverview of the challenges involved in constructing ef-fective query mechanisms for Web-scale data. The au-thors analyze different approaches, such as Treo [55],for five different challenges: usability, query expressiv-ity, vocabulary-level semantic matching, entity recog-nition and improvement of semantic tractability. Thesame is done for architectural elements such as userinteraction and interfaces and the impact on these chal-lenges is reported. Lopez et al. [86] analyze the SQAsystems of the participants of the QALD 1 and 2 evalu-ation challenge, see Section 3.2. While there is an over-lap in the surveyed approaches between Lopez et al.[86] and our paper, our survey has a broader scope asit also analyzes approaches that do not take part in theQALD challenges.

In contrast to the surveys mentioned above, we donot focus on the overall performance or domain of asystem, but on analyzing and categorizing methods thattackle specific problems. Additionally, we build uponthe existing surveys and describe the new state of theart systems, which were published after the before men-tioned surveys in order to keep track of new researchideas.

3.2. Evaluation Campaigns

Contrary to QA surveys, which qualitatively com-pare systems, there are also evaluation campaigns,which quantitatively compare them using benchmarks.Those campaigns show how different open-domain QAsystems perform on realistic questions on real-world

4 Challenges of Question Answering in the Semantic Web

knowledge bases. This accelerates the evolution of QAin four different ways: First, new systems do not haveto include their own benchmark, shortening system de-velopment. Second, standardized evaluation allows forbetter research resource allocation as it is easier to de-termine, which approaches are worthwhile to developfurther. Third, the addition of new challenges to thequestions of each new benchmark iteration motivatesaddressing those challenges. And finally, the competi-tive pressure to keep pace with the top scoring systemscompells emergence and integration of shared best prac-tises. On the other hand, evaluation campaign proceed-ings do not describe single components of those sys-tems in great detail. By focussing on complete systems,research effort gets spread around multiple components,possibly duplicating existing efforts, instead of beingfocussed on a single one.

Question Answering on Linked Data (QALD) isthe most well-known all-purpose evaluation campaignwith its core task of open domain SQA on lexico-graphic facts of DBpedia [82]. Since its inception in2011, the yearly benchmark has been made progres-sively more difficult. Additionally, the general core taskhas been joined by special tasks providing challengeslike multilinguality, hybrid (textual and Linked Data)and its newest addition, SQA on statistical data in theform of RDF Data Cubes [73].

BioASQ [127,103,9,10] is a benchmark challengewhich ran until September 2015 and consists of seman-tic indexing as well as an SQA part on biomedical data.In the SQA part, systems are expected to be hybrids,returning matching triples as well as text snippets butpartial evaluation (text or triples only) is possible aswell. The introductory task separates the process intoannotation which is equivalent to named entity recog-nition (NER) and disambiguation (NED) as well as theanswering itself. The second task combines these twosteps.

TREC LiveQA, starting in 2015 [3], gives systemsunanswered Yahoo Answers questions intended forother humans. As such, the campaign contains the mostrealistic questions with the least restrictions, in contrastto the solely factual questions of QALD, BioASQ andTREC’s old QA track [39].

3.3. System Frameworks

System frameworks provide an abstraction in whichan generic functionality can be selectively changed by

additional third-party code. In document retrieval, thereare many existing frameworks, such as Lucene5, Solr6

and Elastic Search7. For SQA systems, however, thereis still a lack of tools to facilitate implementation andevaluation process of SQA systems.

Document retrieval frameworks usually split the re-trieval process in tree steps (1) query processing, (2)retrieval and (3) ranking. In the (1) query processingstep, query analyzers identify documents in the datastore. Thereafter, the query is used to (2) retrieve doc-uments that match the query terms resulting from thequery processing. Later, the retrieved documents are (3)ranked according to some ranking function, commonlytf-idf [122]. Developing an SQA framework is a hardtask because many systems work with a mixture of NLtechniques on top of traditional IR systems. Some sys-tems make use of the syntactic graph behind the ques-tion [131] to deduce the query intention whereas oth-ers, the knowledge graph [117]. There are hybrid sys-tems that to work both on structured and unstructureddata [134] or on a combination of systems [63]. There-fore, they contain very peculiar steps. This has led to anew research sub field that focuses on QA frameworks,that is, the design and development of common featuresfor SQA systems.

openQA [87]8 is a modular open-source frameworkfor implementing and instantiating SQA approaches.The framework’s main work-flow consists of fourstages (interpretation, retrieval, synthesis, rendering)and adjacent modules (context and service). The adja-cent modules are intended to be accessed by any of thecomponents of the main work-flow to share commonfeatures to the different modules e.g. cache. The frame-work proposes the answer formulation process in avery likely traditional document retrieval fashion wherethe query processing and ranking steps are replaced bythe more general Interpretation and Synthesis. The in-terpretation step comprises all the pre-processing andmatching techniques required to deduce the questionwhereas the synthesis is the process of ranking, merg-ing and confidence estimation required to produce theanswer. The authors claims that openQA enables a uni-fication of different architectures and methods.

5https://lucene.apache.org6https://solr.apache.org7https://www.elastic.co8http://openqa.aksw.org

Challenges of Question Answering in the Semantic Web 5

4. Systems

The 72 surveyed publications describe 62 distinctsystems or approaches. The implementation of a SQAsystem can be very complex and depending on, thusreusing, several known techniques. SQA systems aretypically composed of two stages: (1) the query ana-lyzer and (2) retrieval. The query analyzer generatesor formats the query that will be used to recover theanswer at the retrieval stage. There is a wide varietyof techniques that can be applied at the analyzer stage,such as tokenisation, disambiguation, internationaliza-tion, logical forms, semantic role labels, question re-formulation, coreference resolution, relation extractionand named entity recognition amongst others. For someof those techniques, such as natural language (NL)parsing and part-of-speech (POS) tagging, mature all-purpose methods are available and commonly reused.Other techniques, such as the disambiguating betweenmultiple possible answers candidates, are not availableat hand in a domain independent fashion. Thus, highquality solutions can only be obtained by the devel-opment of new components. This section exemplifiessome of the reviewed systems and their novelties tohighlight current research questions, while the next sec-tion presents the contributions of all analyzed papers tospecific challenges.

Hakimov et al. [64] proposes a SQA system us-ing syntactic dependency trees of input questions. Themethod consists of three main steps: (1) Triple patternsare extracted using the dependency tree and POS tagsof the questions. (2) Entities, properties and classesare extracted and mapped to the underlying knowl-edge base. Recognized entities are disambiguated usingpage links between all spotted named entities as wellas string similarity. Properties are disambiguated byusing relational linguistic patterns from PATTY [97],which allows a more flexible mapping, such as “die” todbo:deathPlace9 Finally, (3) question words arematched to the respective answer type, such as “who” toperson, organization or company and “while” to place.

9URL prefixes are defined in Table 3.

dbo http://dbpedia.org/ontology/

dbr http://dbpedia.org/resource/

owl http://www.w3.org/2002/07/owl#

Table 3URL prefixes used throughout this work.

The results are then ranked and the best ranked resultis returned as the answer.

PARALEX [48] only answers questions for subjectsor objects of property-object or subject-property pairs,respectively. It contains phrase to concept mappingsin a lexicon that is trained from a corpus of para-phrases, which is constructed from the question-answersite WikiAnswers10. If one of the paraphrases can bemapped to a query, this query is the correct answer forthe paraphrases as well. By mapping phrases betweenthose paraphrases, the linguistic patterns are extended.For example, “what is the r of e” leads to “how r ise ”, so that “What is the population of New York” canbe mapped to “How big is NYC”. There is a variety ofother systems, such as Bordes et al. [18], that make useof paraphrase learning methods and integrate linguis-tic generalization with knowledge graph biases. Theyare however not included here as they do query RDFknowledge bases and thus do not fit the inclusion crite-ria.

Xser [144] is based on the observation that SQA con-tains two independent steps. First, Xser determines thequestion structure solely based on a phrase level depen-dency graph and second uses the target knowledge baseto instantiate the generated template. For instance, mov-ing to another domain based on a different knowledgebase thus only affects parts of the approach so that theconversion effort is lessened.

QuASE [124] is a three stage open domain ap-proach based on web search and the Freebase knowl-edge base11. First, QuASE uses entity linking, semanticfeature construction and candidate ranking on the inputquestion. Then, it selects the documents and accordingsentences from a web search with a high probability tomatch the question and presents them as answers to theuser.

DEV-NLQ [123] is based on lambda calculus andan event-based triple store12 using only triple based re-trieval operations. DEV-NLQ claims to be the only QAsystem able to solve chained, arbitrarily-nested, com-plex, prepositional phrases.

CubeQA [73,74] is a novel approach of SQA overmulti-dimensional statistical Linked Data using theRDF Data Cube Vocabulary13, which existing ap-proaches cannot process. Using a corpus of questionswith open domain statistical information needs, the au-

10http://wiki.answers.com/11https://www.freebase.com/12http://www.w3.org/wiki/LargeTripleStores13http://www.w3.org/TR/vocab-data-cube/

6 Challenges of Question Answering in the Semantic Web

thors analyze how those questions differ from others,which additional verbalizations are commonly usedand how this influences design decisions for SQA onstatistical data.

QAKiS [23,34,25] queries several multilingual ver-sions of DBpedia at the same time by filling theproduced SPARQL query with the correspondinglanguage-dependent properties and classes. Thus, it canretrieve correct answers even in cases of missing infor-mation in the language-dependent knowledge base.

Freitas and Curry [53] evaluate a distributional-compositional semantics approach that is independentfrom manually created dictionaries but instead relies onco-occurring words in text corpora. The vector spaceover the set of terms in the corpus is used to create adistributional vector space based on the weighted termvectors for each concept. An inverted Lucene index isadapted to the chosen model.

Instead of querying a specific knowledge base, Sunet al. [124] use web search engines to extract relevanttext snippets, which are then linked to Freebase, wherea ranking function is applied and the highest rankedentity is returned as the answer.

HAWK [134] is the first hybrid source SQA sys-tem which processes Linked Data as well as textualinformation to answer one input query. HAWK usesan eight-fold pipeline comprising part-of-speech tag-ging, entity annotation, dependency parsing, linguisticpruning heuristics for an in-depth analysis of the nat-ural language input, semantic annotation of propertiesand classes, the generation of basic triple patterns foreach component of the input query as well as discard-ing queries containing not connected query graphs andranking them afterwards.

SWIP (Semantic Web intercase using Pattern) [107]generates a pivot query, a hybrid structure between thenatural language question and the formal SPARQL tar-get query. Generating the pivot queries consists of threemain steps: (1) Named entity identification, (2) Queryfocus identification and (3) sub query generation. Toformalize the pivot queries, the query is mapped to lin-guistic patterns, which are created by hand from do-main experts. If there are multiple applicable linguis-tic patterns for a pivot query, the user chooses betweenthem.

Hakimov et al. [65] adapt a semantic parsing algo-rithm to SQA which achieves a high performance butrelies on large amounts of training data which is notpractical when the domain is large or unspecified.

Several industry-driven SQA-related projects haveemerged over the last years. For example, DeepQA of

IBM Watson [63], which was able to win the Jeopardy!challenge against human experts.

YodaQA [12] is a modular open source hybrid ap-proach built on top of the Apache UIMA framework14

that is part of the Brmson platform and is inspiredby DeepQA. YodaQA allows easy parallelization andleverage og pre-existing NLP UIMA components byrepresenting each artifact (question, search result, pas-sage, candidate answer) as a separate UIMA CAS.Yoda pipeline is divided in five different stages: (1)Question Analysis, (2) Answer Production, (3) AnswerAnalysis, (4) Answer Merging and Scoring as well as(5) Successive Refining.

Further, KAIST’s Exobrain15 project aims to learnfrom large amounts of data while ensuring a naturalinteraction with end users. However, it is limited to Ko-rean.

Answer Presentation Another, important part of SQAsystems outside the SQA research challenges is resultpresentation. Verbose descriptions or plain URIs are un-comfortable for human reading. Entity summarizationdeals with different types and levels of abstractions.

Cheng et al. [30] proposes a random surfer modelextended by a notion of centrality, i.e., a computationof the central elements involving similarity (or related-ness) between them as well as their informativeness.The similarity is given by a combination of the related-ness between their properties and their values.

Ngonga Ngomo et al. [101] present another ap-proach that automatically generates natural languagedescription of resources using their attributes. The ra-tionale behind SPARQL2NL is to verbalize16 RDFdata by applying templates together with the metadataof the schema itself (label, description, type). Enti-ties can have multiple types as well as different lev-els of hierarchy which can lead to different levels ofabstractions. The verbalization of the DBpedia entitydbr:Microsoft can vary depending on the typedbo:Agent rather than dbo:Company.

5. Challenges

In this section, we address seven challenges that haveto be faced by state-of-the-art SQA systems. All men-

14https://uima.apache.org/15http://exobrain.kr/16For example, "123"ˆˆ<http://dbpedia.org/

datatype/squareKilometre> can be verbalized as 123square kilometres.

Challenges of Question Answering in the Semantic Web 7

Table 4Different techniques for bridging the lexical gap along with examplesof deviations of the word “running” that these techniques cover.

Identity runningSimilarity Measure runnignStemming/Lemmatizing runAQE—Synonyms sprintPattern libraries X made a break for Y

tioned challenges are currently open research fields.For each challenge, we describe efforts mentioned inthe 72 selected publications. Challenges that affectSQA, but that are not to be solved by SQA systems,such as speech interfaces, data quality and system in-teroperability, are analyzed in Shekarpour et al. [118].

5.1. Lexical Gap

In a natural language, the same meaning can be ex-pressed in different ways. Natural language descrip-tions of RDF resources are provided by values of therdfs:label property (label in the following). Whilesynonyms for the same RDF resource can be mod-eled using multiple labels for that resource, knowledgebases typically do not contain all the different termsthat can refer to a certain entity. If the vocabulary usedin a question is different from the one used in the la-bels of the knowledge base, we call this the lexicalgap17 [65]. Because a question can usually only be an-swered if every referred concept is identified, bridgingthis gap significantly increases the proportion of ques-tions that can be answered by a system. Table 4 showsthe methods employed by the 72 selected publicationsfor bridging the lexical gap along with examples. Asan example of how the lexical gap is bridged outside ofSQA, see Lee et al. [80].

String Normalization and Similarity Functions Nor-malizations, such as conversion to lower case or to baseforms, such as “é” to “e”, allow matching of slightly dif-ferent forms and some simple mistakes, such as “DejaVu” for “déjà vu”, and are quickly implemented andexecuted. More elaborate normalizations use naturallanguage processing (NLP) techniques for stemming(both “running” and “ran” to “run”).

If normalizations are not enough, the distance—andits complementary concept, similarity—can be quanti-fied using a similarity function and a threshold. Com-

17In linguistics, the term lexical gap has a different meaning, re-ferring to a word that has no equivalent in another language.

mon examples of similarity functions are Jaro-Winkler,an edit-distance that measures transpositions and n-grams, which compares sets of substrings of length nof two strings. Also, one of the surveyed publications,Zhang et al. [150], uses the largest common substring,both between Japanese and translated English words.However, applying such similarity functions can carryharsh performance penalties. While an exact stringmatch can be efficiently executed in a SPARQL triplepattern, similarity scores generally need to be calcu-lated between a phrase and every entity label, which isinfeasible on large knowledge bases [134]. There arehowever efficient indexes for some similarity functions.For instance, the edit distances of two characters or lesscan be mitigated by using the fuzzy query implementa-tion of a Lucene Index18 that implements a LevenshteinAutomaton [112]. Furthermore, Ngonga Ngomo [99]provides a different approach to efficiently calculatingsimilarity scores that could be applied to QA. It usessimilarity metrics where a triangle inequality holds thatallows for a large portion of potential matches to be dis-carded early in the process. This solution is not as fastas using a Levenshtein Automaton but does not placesuch a tight limit on the maximum edit distance.

Automatic Query Expansion While normalizationand string similarity methods match different forms ofthe same word, they do not recognize synonyms. Syn-onyms, like design and plan, are pairs of words that,either always or only in a specific context, have thesame meaning. In hyper-hyponym-pairs, like chemicalprocess and photosynthesis, the first word is less spe-cific then the second one. These word pairs, taken fromlexical databases such as WordNet [92], are used asadditional labels in Automatic query expansion (AQE).AQE is commonly used in information retrieval andtraditional search engines, as summarized in Carpinetoand Romano [28]. These additional surface forms al-low for more matches and thus increase recall but leadto mismatches between related words and thus can de-crease the precision.

In traditional document-based search engines withhigh recall and low precision, this trade-off is morecommon than in SQA. SQA is typically optimized forconcise answers and a high precision, since a SPARQLquery with an incorrectly identified concept mostly re-sults in a wrong set of answer resources. However, AQEcan be used as a backup method in case there is nodirect match. One of the surveyed publications is an

18http://lucene.apache.org

8 Challenges of Question Answering in the Semantic Web

experimental study [115] that evaluates the impact ofAQE on SQA. It has analyzed different lexical19 andsemantic20 expansion features and used machine learn-ing to optimize weightings for combinations of them.Both lexical and semantic features were shown to bebeneficial on a benchmark dataset consisting only ofsentences where direct matching is not sufficient.

Pattern libraries RDF individuals can be matchedfrom a phrase to a resource with high accuracy usingsimilarity functions and normalization alone. Proper-ties however require further treatment, as (1) they deter-mine the subject and object, which can be in differentpositions21 and (2) a single property can be expressedin many different ways, both as a noun and as a verbphrase which may not even be a continuous substring22

of the question. Because of the complex and varyingstructure of those linguistic patterns and the requiredreasoning and knowledge23, libraries to overcome thisissues have been developed.

PATTY [97] detects entities in sentences of a corpusand determines the shortest path between the entities.The path is then expanded with occurring modifiersand stored as a pattern. Thus, PATTY is able to buildup a pattern library on any knowledge base with anaccompanying corpus.

BOA [61] generates linguistic patterns using a cor-pus and a knowledge base. For each property in theknowledge base, sentences from a corpus are chosencontaining examples of subjects and objects for thisparticular property. BOA assumes that each resourcepair that is connected in a sentence exemplifies anotherlabel for this relation and thus generates a pattern fromeach occurrence of that word pair in the corpus.

PARALEX [48] contains phrase to concept map-pings in a lexicon that is trained from a corpus of para-phrases from the QA site WikiAnswers. The advantageis that no manual templates have to be created as theyare automatically learned from the paraphrases.

Entailment A corpus of already answered questionsor linguistic question patterns can be used to infer theanswer for new questions. A phrase A is said to entaila phrase B, if B follows from A. Thus, entailment is di-rectional: Synonyms entail each other, whereas hyper-

19lexical features include synonyms, hyper and hyponyms20semantic features making use of RDF graphs and the RDFS

vocabulary, such as equivalent, sub- and superclasses21E.g., “X wrote Y” and “Y is written by X”22E.g., “X wrote Y together with Z” for “X is a coauthor of Y”.23E.g., “if X writes a book, X is called the author of it.”

and hyponyms entail in one direction only: “birds fly”entails “sparrows fly”, but not the other way around. Ouand Zhu [102] generate possible questions for an ontol-ogy in advance and identify the most similar match to auser question based on a syntactic and semantic similar-ity score. The syntactic score is the cosine-similarity ofthe questions using bag-of-words. The semantic scorealso includes hypernyms, hyponyms and denorminal-izations based on WordNet [92]. While the preprocess-ing is algorithmically simple compared to the complexpipeline of NLP tools, the number of possible questionsis expected to grow superlinearly with the size of theontology so the approach is more suited to specific do-main ontologies. Furthermore, the range of possiblequestions is quite limited which the authors aim to par-tially alleviate in future work by combining multiplebasic questions into a complex question.

Document Retrieval Models Blanco et al. [17] adaptentity ranking models from traditional document re-trieval algorithms to RDF data. The authors applyBM25 as well as the tf-idf ranking function to an in-dex structure with different text fields constructed fromthe title, object URIs, property values and RDF inlinks.The proposed adaptation is shown to be both time effi-cient and qualitatively superior to other state-of-the-artmethods in ranking RDF resources.

Composite Approaches Elaborate approaches onbridging the lexical gap can have a high impact on theoverall runtime performance of an SQA system. Thiscan be partially mitigated by composing methods andexecuting each following step only if the one beforedid not return the expected results.

BELA [136] implements four layers. First, the ques-tion is mapped directly to the concept of the ontologyusing the index lookup. Second, the question is mappedbased on Levenshtein distance to the ontology, if theLevenshtein distance of a word from the question anda property from an ontology exceed a certain threshold.Third, WordNet is used to find synonyms for a givenword. Finally, BELA uses explicit semantic analysis(ESA) Gabrilovich and Markovitch [58]. The evalua-tion is carried out on the QALD 2 [132] test dataset andshows that the more simple steps, like index lookup andLevenshtein distance, had the most positive influenceon answering questions so that many questions can beanswered with simple mechanisms.

Park et al. [105] answer natural language questionsvia regular expressions and keyword queries with aLucene-based index. Furthermore, the approach uses

Challenges of Question Answering in the Semantic Web 9

DBpedia [84] as well as their own triple extractionmethod on the English Wikipedia.

5.2. Ambiguity

Ambiguity is the phenomenon of the same phrasehaving different meanings; this can be structural andsyntactic (like “flying planes”) or lexical and seman-tic (like “bank”). We distinguish between homonymy,where the same string accidentally refers to differentconcepts (as in money bank vs. river bank) and poly-semy, where the same string refers to different but re-lated concepts (as in bank as a company vs. bank as abuilding). We distinguish between synonymy and taxo-nomic relations such as metonymy and hypernymy. Incontrast to the lexical gap, which impedes the recall of aSQA system, ambiguity negatively effects its precision.Ambiguity is the flipside of the lexical gap.

This problem is aggravated by the very methods usedfor overcoming the lexical gap. The more loose thematching criteria become (increase in recall), the morecandidates are found which are generally less likelyto be correct than closer ones. Disambiguation is theprocess of selecting one of multiple candidate conceptsfor an ambiguous phrase. We differentiate between twotypes of disambiguation based on the source and typeof information used to solve this mapping:

Corpus-based methods are traditionally used andrely on counts, often used as probabilities, from unstruc-tured text corpora. Such statistical approaches [120] arebased on the distributional hypothesis, which states that“difference of meaning correlates with difference of[contextual] distribution” [68]. The context of a phraseis identified here as its central characteristic [93]. Com-mon context features used are word co-occurrences,such as left or right neighbours, but also synonyms, hy-ponyms, POS-tags and the parse tree structure. Moreelaborate approaches also take advantage of the con-text outside of the question, such as past queries of theuser [119] .

In SQA, Resource-based methods exploit the factthat the candidate concepts are RDF resources. Re-sources are compared using different scoring schemesbased of their properties and the connections betweenthem. The assumption is that high score between all theresources chosen in the mapping implies a higher prob-ability of those resources being related, and that thisimplies a higher propability of those resource being cor-rectly chosen. RVT [62] uses Hidden Markov Models(HMM) to select the proper ontological triples accord-ing to the graph nature of DBpedia. CASIA [70] em-

ploys Markov Logic Networks (MLN): First-order logicstatements are assigned a numerical penalty, which isused to define hard constraints, like “each phrase canmap to only one resource”, alongside soft constraints,like “the larger the semantic similarity is between tworesources, the higher the chance is that they are con-nected by a relation in the question”. Underspecifica-tion [128] discards certain combinations of possiblemeanings before the time consuming querying step, bycombining restrictions for each meaning. Each termis mapped to a Dependency-based Underspecified Dis-course REpresentation Structure (DUDE [31]), whichcaptures its possible meanings along with their class re-strictions. Treo [55,54] performs entity recognition anddisambiguation using Wikipedia-based semantic relat-edness and spreading activation. Semantic relatednesscalculates similarity values between pairs of RDF re-sources. Determining semantic relatedness between en-tity candidates associated to words in a sentence allowsto find the most probable entity by maximizing the totalrelatedness. EasyESA [29] is based on distributionalsemantic models which allow to represent an entity bya vector of target words and thus compresses its repre-sentation. The distributional semantic models allow tobridge the lexical gap and resolve ambiguity by avoid-ing the explicit structures of RDF-based entity descrip-tions for entity linking and relatedness. gAnswer [76]tackles ambiguity with RDF fragments, i.e., star-likeRDF subgraphs. The number of connections betweenthe fragments of the resource candidates is then usedto score and select them. Wikimantic [19] can be usedto disambiguate short questions or even sentences. Ituses Wikipedia article interlinks for a generative model,where the probability of an article to generate a term isset to the terms relative occurrence in the article. Dis-ambiguation is then an optimization problem to locallymaximize each article’s (and thus DBpedia resource’s)term probability along with a global ranking method.Shekarpour et al. [113,116] disambiguate resource can-didates using segments consisting of one or more wordsfrom a keyword query. The aim is to maximize the hightextual similarity of keywords to resources along withrelatedness between the resources (classes, propertiesand entities). The problem is cast as a Hidden MarkovModel (HMM) with the states representing the set ofcandidate resources extended by OWL reasoning. Thetransition probabilities are based on the shortest pathbetween the resources. The Viterbi algorithm gener-ates an optimal path though the HMM that is used fordisambiguation. DEANNA [145,146] manages phrasedetection, entity recognition and entity disambiguation

10 Challenges of Question Answering in the Semantic Web

by formulating the SQA task as an integer linear pro-gramming (ILP) problem. It employs semantic coher-ence which measures co-occurrence of resources in thesame context. DEANNA constructs a disambiguationgraph, which encodes the selection of candidates for re-sources and properties. The chosen objective functionmaximizes the combined similarity while constraintsguarantee that the selections are valid. The resultingproblem is NP-hard but it is efficiently solvable in ap-proximations by existing ILP solvers. The follow-upapproach [147] uses DBpedia and Yago with a map-ping of input queries to semantic relations based ontext search. At QALD 2, it outperformed almost everyother system on factoid questions and every other sys-tem on list questions. However, the approach requiresdetailed textual descriptions of entities and only cre-ates basic graph pattern queries. LOD-Query [114] is akeyword-based SQA system that tackles both ambigu-ity and the lexical gap by selecting candidate conceptsbased on a combination of a string similarity score andthe connectivity degree. The string similarity is thenormalized edit distance between a labels and a key-word. The connectivity degree of a concept is approxi-mated by the occurrence of that concept in all the triplesof the knowledge base. Pomelo [66] answers biomed-ical questions on the combination of Drugbank, Dis-easome and Sider using owl:sameAs links betweenthem. Properties are disambiguated using predefinedrewriting rules which are categorized by context. Raniet al. [110] use fuzzy logic co-clustering algorithms toretrieve documents based on their ontology similarity.Possible senses for a word are assigned a probabilitydepending on the context. Zhang et al. [150] translatesRDF resources to the English DBpedia. It uses feed-back learning in the disambiguation step to refine theresource mapping

Instead of trying to resolve ambiguity automati-cally, some approaches let the user clarify the exact in-tent, either in all cases or only for ambiguous phrases:SQUALL [50,51] defines controlled, English-based,vocabulary that is enhanced with knowledge from agiven triple store. While this ideally results in a highperformance, it moves the problem of the lexical gapand disambiguation fully to the user. As such, it cov-ers a middle ground between SPARQL and full-fledgedSQA with the author’s intent that learning the gram-matical structure of this proposed language is easierfor a non-expert than to learn SPARQL. A cooper-ative approach that places less of a burden on theuser is proposed in [88], which transforms the ques-tion into a discourse representation structure and starts

a dialogue with the user for all occurring ambigui-ties. CrowdQ [42] is a SQA system that decomposescomplex queries into simple parts (keyword queries)and uses crowdsourcing for disambiguation. It avoidsexcessive usage of crowd resources by creating gen-eral templates as an intermediate step. FREyA (Feed-back, Refinement and Extended VocabularY Aggrega-tion) [36] represents phrases as potential ontology con-cepts which are identified by heuristics on the syntacticparse tree. Ontology concepts are identified by match-ing their labels with phrases from the question withoutregarding its structure. A consolidation algorithm thenmatches both potential and ontology concepts. In caseof ambiguities, feedback from the user is asked. Disam-biguation candidates are created using string similar-ity in combination with WordNet synonym detection.The system learns from the user selections, therebyimproving the precision over time. TBSL [131] usesboth an domain independent and a domain dependentlexicon so that it performs well on specific topic butis still adaptable to a different domain. It uses Au-toSPARQL [81] to refine the learned SPARQL usingthe QTL algorithm for supervised machine learning.The user marks certain answers as correct or incorrectand triggers a refinement. This is repeated until theuser is satisfied with the result. An extension of TBSLis DEQA [83], which combines Web extraction withOXPath [57], interlinking with LIMES [100] and SQAwith TBSL. It can thus answer complex questions aboutobjects which are only available as HTML. Anotherextension of TBSL is ISOFT [104], which uses explicitsemantic analysis to help bridging the lexical gap. NL-Graphs [47] combines SQA with an interactive visu-alization of the graph of triple patterns in the querywhich is close to the SPARQL query structure yet stillintuitive to the user. Users that find errors in the querystructure can either reformulate the query or modifythe query graph. KOIOS [15] answers queries on natu-ral environment indicators and allows the user to refinethe answer to a keyword query by faceted search. In-stead of relying on a given ontology, a schema index isgenerated from the triples and then connected with thekeywords of the query. Ambiguity is resolved by userfeedback on the top ranked results.

A different way to restrict the set of answer candi-dates and thus handle ambiguity is to determine theexpected answer type of a factual question. The stan-dard approach to determine this type is to identify thefocus of the question and to map this type to an on-tology class. In the example “Which books are writ-ten by Dan Brown?”, the focus is “books”, which is

Challenges of Question Answering in the Semantic Web 11

mapped to dbo:Book. There is however a long tailof rare answer types that are not as easily alignable toan ontology, which, for instance, Watson [63] tacklesusing the TyCor [79] framework for type coercion. In-stead of the standard approach, candidates are first gen-erated using multiple interpretations and then selectedbased on a combination of scores. Besides trying toalign the answer type directly, it is coerced into othertypes by calculating the probability of an entity of classA to also be in class B. DBpedia, Wikipedia and Word-Net are used to determine link anchors, list member-ships, synonyms, hyper- and hyponyms. The follow-up [137] compares two different approaches for answertyping. Type-and-Generate (TaG) approaches restrictcandidate answers to the expected answer types usingpredictive annotation, which requires manual analysisof a domain. Tycor on the other hand employs multiplestrategies using generate-and-type (GaT), which gen-erates all answers regardless of answer type and triesto coerce them into the expected answer type. Exper-imental results hint that GaT outperforms TaG whenaccuracy is higher than 50%. The significantly higherperformance of TyCor when using GaT is explained byits robustness to incorrect candidates while there is norecovery from excluded answers from TaG.

5.3. Multilingualism

Knowledge on the Web is expressed in various lan-guages. While RDF resources can be described inmultiple languages at once using language tags, thereis not a single language that is always used in Webdocuments. Additionally, users have different nativelanguages. A more flexible approach is thus to haveSQA systems that can handle multiple input languages,which may even differ from the language used to en-code the knowledge. Deines and Krechel [40] useGermaNet [67] which is integrated into the multilin-gual knowledge base EuroWordNet [135] together withlemon-LexInfo [22], to answer German questions. Ag-garwal et al. [2] only need to successfully translate partof the query, after which the recognition of the otherentities is aided using semantic similarity and related-ness measures between resources connected to the ini-tial ones in the knowledge base. QAKiS (Question An-swering wiKiframework-based system) [34] automati-cally extends existing mappings between different lan-guage versions of Wikipedia, which is carried over toDBpedia.

5.4. Complex Queries

Simple questions can most often be answered bytranslation into a set of simple triple patterns. Problemsarise when several facts have to be found out, connectedand then combined. Queries may also request a specificresult order or results that are aggregated or filtered.

YAGO-QA [1] allows nested queries when the sub-query has already been answered, for example “Whois the governor of the state of New York?” after “Whatis the state of New York?” YAGO-QA extracts factsfrom Wikipedia (categories and infoboxes), WordNetand GeoNames. It contains different surface forms suchas abbreviations and paraphrases for named entities.

PYTHIA [129] is an ontology-based SQA systemwith an automatically build ontology-specific lexicon.Due to the linguistic representation, the system is ableto answer natural language question with linguisticallymore complex queries, involving quantifiers, numerals,comparisons and superlatives, negations and so on.

IBM Watson [63] handles complex questions by firstdetermining the focus element, which represents thesearched entity. The information about the focus ele-ment is used to predict the lexical answer type and thusrestrict the range of possible answers. This approachallows for indirect questions and multiple sentences.

Shekarpour et al. [113,116], as mentioned in Sec-tion 5.2, propose a model that use a combination ofknowledge base concepts with a HMM model to handlecomplex queries.

Intui2 [43] is an SQA system based on DBpediabased on synfragments which map to a subtree of thesyntactic parse tree. Semantically, a synfragment is aminimal span of text that can be interpreted as an RDFtriple or complex RDF query. Synfragments interop-erate with their parent synfragment by combining allcombinations of child synfragments, ordered by syntac-tic and semantic characteristics. The authors assumethat an interpretation of a question in any RDF querylanguage can be obtained by the recursively interpreta-tion of its synfragments. Intui3 [44] replaces self-madecomponents with robust libraries such as the neuralnetworks-based NLP toolkit SENNA and the DBpediaLookup service. It drops the parser determined inter-pretation combination method of its predecessor thatsuffered from bad sentence parses and instead uses afixed order right-to-left combination.

GETARUNS [41] first creates a logical form outof a query which consists of a focus, a predicateand arguments. The focus element identifies the ex-pected answer type. For example, the focus of “Who

12 Challenges of Question Answering in the Semantic Web

is the major of New York?” is “person”, the predi-cate “be” and the arguments “major of New York”. Ifno focus element is detected, a yes/no question is as-sumed. In the second step, the logical form is con-verted to a SPARQL query by mapping elements toresources via label matching. The resulting triple pat-terns are then split up again as properties are refer-enced by unions over both possible directions, as in({?x ?p ?o} UNION {?o ?p ?x}) because thedirection is not known beforehand. Additionally, thereare filters to handle additional restrictions which can-not be expressed in a SPARQL query, such as “Whohas been the 5th president of the USA”.

5.5. Distributed Knowledge

If concept information–which is referred to in aquery–is represented by distributed RDF resources, in-formation needed for answering it may be missing ifonly a single one or not all of the knowledge basesare found. In single datasets with a single source, suchas DBpedia, however, most of the concepts have atmost one corresponding resource. In case of combineddatasets, this problem can be dealt with by creatingsameAs, equivalentClass or equivalentProperty links,respectively. However, interlinking while answering asemantic query is a separate research area and thus notcovered here.

Some questions are only answerable with multipleknowledge bases and we assume already created linksfor the sake of this survey. The ALOQUS [78] systemtackles this problem by using the PROTON [37] upperlevel ontology first to phrase the queries. The ontologyis than aligned to those of other knowledge bases us-ing the BLOOMS [77] system. Complex queries aredecomposed into separately handled subqueries aftercoreferences24 are extracted and substituted. Finally,these alignments are used to execute the query on thetarget systems. In order to improve the speed and qual-ity of the results, the alignments are filtered using athreshold on the confidence measure.

Herzig et al. [71] search for entities and consolidateresults from multiple knowledge bases. Similarity met-rics are used both to determine and rank results can-didates of each datasource and to identify matches be-tween entities from different datasources.

24Such as “List the Semantic Web people and their affiliation.”,where the coreferent their refers to the entity people.

5.6. Procedural, Temporal and Spatial Questions

Procedural Questions Factual, list and yes-no ques-tions are easiest to answer as they conform directlyto SPARQL queries using SELECT and ASK. Others,such as why (causal) or how (procedural) questions re-quire more additional processing. Procedural QA cancurrently not be solved by SQA, since, to the best of ourknowledge, there are no existing knowledge bases thatcontain procedural knowledge. While it is not an SQAsystem, we describe the document-retrieval based KO-MODO [26] to motivate further research in this area. In-stead of an answer sentence, KOMODO returns a Webpage with step-by-step instructions on how to reach thegoal specified by the user. This reduces the problemdifficulty as it is much easier to find a Web page whichcontains instructions on how to, for example, assemblean “Ikea Billy bookcase” than it would be to extract,parse and present the required steps to the user. Ad-ditionally, there are arguments explaining reasons fortaking a step and warnings against deviation. Insteadof extracting the sense of the question using an RDFknowledge base, KOMODO submits the question to atraditional search engine. The highest ranked returnedpages are then cleaned and procedural text is identifiedusing statistical distributions of certain POS tags.

In basic RDF, each fact, which is expressed by atriple, is assumed to be true, regardless of circum-stances. In the real world and in natural language how-ever, the truth value of many statements is not a con-stant but a function of either or both the location ortime.

Temporal Questions Tao et al. [126] answer tempo-ral question on clinical narratives. They introduce theClinical Narrative Temporal Relation Ontology (CN-TRO), which is based on Allen’s Interval Based Tempo-ral Logic [5] but allows usage of time instants as wellas intervals. This allows inferring the temporal relationof events from those of others, for example by using thetransitivity of before and after. In CNTRO, measure-ment, results or actions done on patients are modeledas events whose time is either absolutely specified indate and optionally time of day or alternatively in re-lations to other events and times. The framework alsoincludes an SWRL [75] based reasoner that can deduceadditional time information. This allows the detectionof possible causalities, such as between a therapy for adisease and its cure in a patient.

Melo et al. [88] propose to include the implicit tem-poral and spatial context of the user in a dialog in order

Challenges of Question Answering in the Semantic Web 13

to resolve ambiguities. It also includes spatial, temporaland other implicit information.

QALL-ME [49] is a multilingual framework basedon description logics and uses the spatial and temporalcontext of the question. If this context is not explicitlygiven, the location and time are of the user posing thequestion are added to the query. This context is alsoused to determine the language used for the answer,which can differ from the language of the question.

Spatial Questions In RDF, a location can be ex-pressed as 2-dimensional geocoordinates with latitudeand longitude, while three-dimensional representations(e.g. with additional height) are not supported by themost often used schema25. Alternatively, spatial rela-tionships can be modeled which are easier to answeras users typically ask for relationships and not exactgeocoordinates .

Younis et al. [149] employ an inverted index fornamed entity recognition that enriches semantic datawith spatial relationships such as crossing, inclusionand nearness. This information is then made availablefor SPARQL queries.

5.7. Templates

For complex questions, where the resulting SPARQLquery contains more than one basic graph pattern, so-phisticated approaches are required to capture the struc-ture of the underlying query. Current research fol-lows two paths, namely (1) template based approaches,which map input questions to either manually or au-tomatically created SPARQL query templates or (2)template-free approaches that try to build SPARQLqueries based on the given syntactic structure of theinput question.

For the first solution, many (1) template-driven ap-proaches have been proposed like TBSL [131] orSINA [113,116]. Furthermore, Casia [69] generates thegraph pattern templates by using the question type,named entities and POS tags techniques. The gener-ated graph patterns are then mapped to resources usingWordNet, PATTY and similarity measures. Finally, thepossible graph pattern combinations are used to buildSPARQL queries. The system focuses in the generationof SPARQL queries that do not need filter conditions,aggregations and superlatives.

25see http://www.w3.org/2003/01/geo/wgs84_posat http://lodstats.aksw.org

Ben Abacha and Zweigenbaum [13] focus on a nar-row medical patients-treatment domain and use manu-ally created templates alongside machine learning.

Damova et al. [38] return well formulated naturallanguage sentences that are created using a templatewith optional parameters for the domain of paintings.Between the input query and the SPARQL query, thesystem places the intermediate step of a multilingualdescription using the Grammatical Framework [111],which enables the system to support 15 languages.

Rahoman and Ichise [109] propose a template basedapproach using keywords as input. Templates are auto-matically constructed from the knowledge base.

However, (2) template-free approaches require addi-tional effort of making sure to cover every possible ba-sic graph pattern [134]. Thus, only a few SQA systemstackle this approach so far.

Xser [144] first assigns semantic labels, i.e., vari-ables, entities, relations and categories, to phrases bycasting them to a sequence labelling pattern recognitionproblem which is then solved by a structured percep-tron. The perceptron is trained using features includingn-grams of POS tags, NER tags and words. Thus, Xseris capable of covering any complex basic graph pattern.

Going beyond SPARQL queries is TPSM, the opendomain Three-Phases Semantic Mapping [60] frame-work. It maps natural language questions to OWLqueries using Fuzzy Constraint Satisfaction Problems.Constraints include surface text matching, preferenceof POS tags and the similarity degree of surface forms.The set of correct mapping elements acquired using theFCSP-SM algorithm is combined into a model usingpredefined templates.

An extension of gAnswer [151] (see Section 5.2) isbased on question understanding and query evaluation.First, their approach uses a relation mining algorithmto find triple patterns in queries as well as relation ex-traction, POS-tagging and dependency parsing. Second,the approach tries to find a matching subgraph for theextracted triples and scores them based on a confidencescore. Finally, the top-k subgraph matches are returned.Their evaluation on QALD 3 shows that mapping NLquestions to graph pattern is not as powerful as gener-ating SPARQL (template) queries with respect to ag-gregation and filter functions needed to answer severalbenchmark input questions.

6. Conclusion

In this survey, we analyzed 62 systems and their con-tributions to seven challenges for SQA systems. SQA is

14 Challenges of Question Answering in the Semantic Web

an active research field with many existing and diverseapproaches covering a multitude of research challenges,domains and knowledge bases.

We only cover QA on the Semantic Web, that is, ap-proaches that retrieve resources as Linked Data fromRDF knowledge bases. As similar challenges are facedby QA unrelated to the Semantic Web, we refer to Sec-tion 3. We choose to not go into detail for approachesthat do not retrieve resources from RDF knowledgebases. Moreover, our consensus can be found in Ta-ble 6 for best practices. The upcoming HOBBIT26

project will clarify, which modules can be aligned withstate-of-the art performance and will quantify the im-pact of those modules. To cover the field of SQA indepth, we exluded works solely about similarity [11]or paraphrases [14]. The existence of common SQA

26http://project-hobbit.eu/

Table 5Number of publications per year per addressed challenge. Percent-ages are given for the fully covered years 2011–2014 separately andfor the whole covered timespan, with 1 decimal place. For a full list,see Table 7.

Year

Tota

l

Lex

ical

Gap

Am

bigu

ity

Mul

tilin

gual

ism

Com

plex

Ope

rato

rs

Dis

trib

uted

Kno

wle

dge

Proc

edur

al,T

empo

ralo

rSp

atia

l

Tem

plat

es

absolute

2010 1 0 0 0 0 0 1 0

2011 16 11 12 1 3 1 2 2

2012 14 6 7 1 2 1 1 4

2013 20 18 12 2 5 1 1 5

2014 13 7 8 1 2 0 1 0

2015 6 5 3 1 0 1 0 0

all 70 46 42 6 12 4 6 11

percentage

2011 68.8 75.0 6.3 18.8 6.3 12.5 12.5

2012 42.9 50.0 7.1 14.3 7.1 7.1 28.6

2013 85.0 60.0 10.0 25.0 5.0 5.0 25.0

2014 53.8 61.5 7.7 15.4 7.7 7.7 0.0

all 65.7 60.0 8.6 17.1 5.7 8.6 15.7

challenges implies that a unifying architecture can im-prove the precision as well as increase the numberof answered questions [87]. Research into such archi-tectures, includes openQA [87], OAQA [148], QALL-ME [49] and QANUS [98] (see Section 3.3). Our goal,however, is not to quantify submodule performance orinterplay. That will be the task of upcoming projects oflarge consortiums. A new community27 is forming inthat field and did not find a satisfying solution yet.28 Inthis section, we discuss each of the seven research chal-lenges and give a short overview of already establishedas well as future research directions per challenge, seeTable 6.

Overall, the authors of this survey cannot observe aresearch drift to any of the challenges. The number ofpublications in a certain research challenge does not de-crease significantly, which can be seen as an indicatorthat none of the challenges is solved yet – see Table 5.Naturally, since only a small number of publicationsaddressed each challenge in a given year, one cannotdraw statistically valid conclusions. The challenges pro-posed by Cimiano and Minock [32] and reduced withinthis survey appear to be still valid.

Bridging the (1) lexical gap has to be tackled by ev-ery SQA system in order to retrieve results with a highrecall. For named entities, this is commonly achievedusing a combination of the reliable and mature nat-ural language processing algorithms for string simi-larity and either stemming or lemmatization, see Ta-ble 6. Automatic Query Expansion (AQE), for examplewith WordNet synonyms, is prevalent in informationretrieval but only rarely used in SQA. Despite its po-tential negative effects on precision29, we consider it anet benefit to SQA systems. Current SQA systems du-plicate already existing efforts or fail to decide on theright technique. Thus, reusable libraries to lower theentrance effort to SQA systems are needed. Mappingto RDF properties from verb phrases is much harder,as they show more variation and often occur at mul-tiple places of a question. Pattern libraries, such asBOA [61], can improve property identification, how-ever they are still an active research topic and are spe-cific to a knowledge base.

The next challenge, (2) ambiguity is addressed by themajority of the publications but the percentage does notincrease over time, presumably because of use cases

27https://www.w3.org/community/nli/28http://eis.iai.uni-bonn.de/blog/2015/11/29Synonyms and other related words almost never have exactly the

same meaning.

Challenges of Question Answering in the Semantic Web 15

Table 6Established and actively researched as well as envisioned techniquesfor solving each challenge.

Challenge Established Future

Lexical Gap stemming, lemmatization, string similarity, synonyms, vector space model,indexing, pattern libraries, explicit semantic analysis

combined efforts,reuse of libraries

Ambiguity user information (history, time, location), underspecification, machine learning,spreading activation, semantic similarity, crowdsourcing, Markov Logic Network

holistic,knowledge-baseaware systems

Multilingualism translation to core language, language-dependent grammar usage of multilingualknowledge bases

Complex Operators reuse of former answers, syntactic tree-based formulation, answer typeorientation, HMM, logic

non-factual questions,domain-independence

Distributed Knowledge andProcedural, Temporal,Spatial

temporal logic domain specificadaptors, proceduralSQA

Templates fixed SPARQL templates, template generation, syntactic tree based generation complex questions

with small knowledge bases, where its impact is minus-cule. For systems intended for longtime usage by thesame persons, we regard as promising the integrationof previous questions, time and location, as is alreadycommon in web of document search engines. There is avariety of established disambiguation methods, whichuse the context of a phrase to determine the most likelyRDF resource, some of which are based on unstruc-tured text collections and others on RDF resources. Aswe could make out no clear winner, we recommend sys-tem developers to make their decisions based on the re-sources (such as query logs, ontologies, thesauri) avail-able to them. Many approaches reinvent disambigua-tion efforts and thus–like for the lexical gap–holistic,knowledge-base aware, reusable systems are needed tofacilitate faster research.

Despite its inclusion since QALD 3 and following,publications dealing with (3) multilingualism remain asmall minority. Automatic translation of parts of or thewhole query requires the least development effort, butsuffers from imperfect translations. A higher qualitycan be achieved by using components, such as parsersand synonym libraries, for multiple languages. A possi-ble future research direction is to make use of variouslanguage versions at once to use the power of a uni-fied graph [34]. For instance, DBpedia [84] providesa knowledge base in more than 100 languages, whichcould form the base of a multilingual SQA system.

Complex operators (4) seem to be used only in spe-cific tasks or factual questions. Most systems either usethe syntactic structure of the question or some formof knowledge-base aware logic. Future research will

be directed towards domain-independence as well asnon-factual queries.

Approaches using (5) distributed knowledge as wellas those incorporating (6) procedural, temporal and spa-tial data remain niches. Procedural SQA does not ex-ist yet as present approaches return unstructured textin the form of already written step-by-step instructions.While we consider future development of proceduralSQA as feasible with the existing techniques, as far aswe know there is no RDF vocabulary for and knowl-edge base with procedural knowledge yet.

The (7) templates challenge which subsumes thequestion of mapping a question to a query structure isstill unsolved. Although the development of templatebased approaches seems to have decreased in 2014, pre-sumably because of their low flexibility on open do-main tasks, this still presents the fastest way to developa novel SQA system but the limitiation to simple querystructures has yet to be overcome.

Future research should be directed at more modu-larization, automatic reuse, self-wiring and encapsu-lated modules with their own benchmarks and evalu-ations. Thus, novel research field can be tackled byreusing already existing parts and focusing on the re-search core problem itself. A step in this directionis QANARY [20], which describes how to modular-ize QA systems by providing a core QA vocabularyagainst which existing vocabularies are bound. An-other research direction are SQA systems as aggre-gators or framework for other systems or algorithmsto benefit of the set of existing approaches. Further-more, benchmarking will move to single algorithmicmodules instead of benchmarking a system as a whole.

16 Challenges of Question Answering in the Semantic Web

The target of local optimization is benchmarking a pro-cess at the individual steps, but global benchmarkingis still needed to measure the impact of error propaga-tion across the chain. A Turing test-like spirit wouldsuggests that the latter is more important, as the localmeasure are never fully representative. Additionally,we foresee the move from factual benchmarks overcommon sense knowledge to more domain specificquestions without purely factual answers. Thus, thereis a movement towards multilingual, multi-knowledge-source SQA systems that are capable of understandingnoisy, human natural language input.

Acknowledgements This work has been supportedby the FP7 project GeoKnow (GA No. 318159), theBMWI Project SAKE (Project No. 01MD15006E)and by the Eurostars projects DIESEL (E!9367) andQAMEL (E!9725) as well as the European Union’sH2020 research and innovation action HOBBIT (GA688227).

Challenges of Question Answering in the Semantic Web 17

Table 7: Surveyed publications from November 2010 to July of 2015, inclusive, along with the challenges theyexplicitely address and the approach or system they belong to. Additionally annotated is the use light expressions aswell as the use of intermediate templates. In case the system or approach is not named in the publication, a name isgenerated using the last name of the first author and the year of the first included publication.

publ

icat

ion

syst

emor

appr

oach

Year

Lex

ical

Gap

Am

bigu

ity

Mul

tilin

gual

ism

Com

plex

Ope

rato

rs

Dis

trib

uted

Kno

wle

dge

Proc

edur

al,T

empo

ralo

rSp

atia

l

Tem

plat

es

QA

LD

1Q

AL

D2

QA

LD

3

QA

LD

4

Tao et al. [126] Tao10 2010 X

Adolphs et al. [1] YAGO-QA 2011 X XBlanco et al. [17] Blanco11 2011 XCanitrot et al. [26] KOMODO 2011 X XDamljanovic et al. [36] FREyA 2011 X X XFerrandez et al. [49] QALL-ME 2011 X XFreitas et al. [55] Treo 2011 X X X XGao et al. [60] TPSM 2011 X XKalyanpur et al. [79] Watson 2011 XMelo et al. [88] Melo11 2011 X XMoussa and Abdel-Kader [95] QASYO 2011 X XOu and Zhu [102] Ou11 2011 X X X XShen et al. [119] Shen11 2011 XUnger and Cimiano [128] Pythia 2011 XUnger and Cimiano [129] Pythia 2011 X X X XBicer et al. [15] KOIOS 2011 X XFreitas et al. [54] Treo 2011 X

Ben Abacha and Zweigenbaum [13] MM+/BIO-CRF-H 2012 XBoston et al. [19] Wikimantic 2012 XGliozzo and Kalyanpur [63] Watson 2012 XJoshi et al. [78] ALOQUS 2012 X XLehmann et al. [83] DEQA 2012 X X XYahya et al. [145] DEANNA 2012Yahya et al. [146] DEANNA 2012 X XShekarpour et al. [113] SINA 2012 X XUnger et al. [131] TBSL 2012 X X XWalter et al. [136] BELA 2012 X X X XYounis et al. [149] Younis12 2012 X XWelty et al. [137] Watson 2012 XElbedweihy et al. [46] Elbedweihy12 2012 XCabrio et al. [23] QAKiS 2012 X

Demartini et al. [42] CrowdQ 2013 X X

18 Challenges of Question Answering in the Semantic Web

Aggarwal et al. [2] Aggarwal12 2013 X X XDeines and Krechel [40] GermanNLI 2013 XDima [43] Intui2 2013 X XFader et al. [48] PARALEX 2013 X X X XFerre [50] SQUALL2SPARQL 2013 X X XGiannone et al. [62] RTV 2013 X XHakimov et al. [64] Hakimov13 2013 X X XHe et al. [69] CASIA 2013 X X X X XHerzig et al. [71] CRM 2013 X X XHuang and Zou [76] gAnswer 2013 X XPradel et al. [107] SWIP 2013 X XRahoman and Ichise [109] Rahoman13 2013 XShekarpour et al. [116] SINA 2013 X X XShekarpour et al. [115] SINA 2013 XShekarpour et al. [114] SINA 2013 X X XDelmonte [41] GETARUNS 2013 X X XCojan et al. [34] QAKiS 2013 XYahya et al. [147] SPOX 2013 X XZhang et al. [150] Kawamura13 2013 X X

Carvalho et al. [29] EasyESA 2014 X XRani et al. [110] Rani14 2014 XZou et al. [151] Zhou14 2014 XStewart [123] DEV-NLQ 2014 XHöffner and Lehmann [73] CubeQA 2014 X X XCabrio et al. [25] QAKiS 2014 XFreitas and Curry [53] Freitas14 2014 X XDima [44] Intui3 2014 X X XHamon et al. [66] POMELO 2014 X XPark et al. [104] ISOFT 2014 X XHe et al. [70] CASIA 2014 X X XXu et al. [144] Xser 2014 X XElbedweihy et al. [47] NL-Graphs 2014 XSun et al. [125] QuASE 2015 X X

Park et al. [105] Park15 2015 X XDamova et al. [38] MOLTO 2015 XSun et al. [124] QuASE 2015 X XUsbeck et al. [134] HAWK 2015 X X XHakimov et al. [65] Hakimov15 2015 X

Challenges of Question Answering in the Semantic Web 19

References

[1] P. Adolphs, M. Theobald, U. Schäfer, H. Uszkoreit, andG. Weikum. YAGO-QA: Answering questions by struc-tured knowledge queries. In ICSC 2011 IEEE Fifth In-ternational Conference on Semantic Computing, pages 158–161, Los Alamitos, USA, 2011. IEEE Computer Society.doi:10.1109/ICSC.2011.30.

[2] N. Aggarwal, T. Polajnar, and P. Buitelaar. Cross-lingual nat-ural language querying over the Web of Data. In E. Mé-tais, F. Meziane, M. S. V. Sugumaran, and S. Vadera, editors,18th International Conference on Applications of Natural Lan-guage to Information Systems, NLDB 2013, pages 152–163.Springer, Berlin Heidelberg, Germany, 2013. doi:10.1007/978-3-642-38824-8_13.

[3] E. Agichtein, D. Carmel, D. Pelleg, Y. Pinter, and D. Harman.Overview of the TREC 2015 LiveQA track. In E. M. Voorheesand A. Ellis, editors, The Twenty-Fourth Text REtrieval Con-ference (TREC 2015) Proceedings. National Institute of Stan-dards and Technology (NIST), 2015.

[4] H. Alani, L. Kagal, A. Fokoue, P. Groth, C. Biemann, J. X.Parreira, L. Aroyo, N. Noy, C. Welty, and K. Janowicz, editors.The Semantic Web–ISWC 2013, Berlin Heidelberg, Germany,2013. Springer. doi:10.1007/978-3-642-41335-3.

[5] J. F. Allen. Maintaining knowledge about temporal in-tervals. Communications of the ACM, 26:832–843, 1983.doi:10.1145/182.358434.

[6] G. Antoniou, M. Grobelnik, E. Simperl, B. Parsia, D. Plex-ousakis, P. de Leenheer, and J. Z. Pan, editors. The Seman-tic Web: Research and applications, volume 6643 of LectureNotes in Computer Science, Berlin Heidelberg, Germany, 2011.Springer. doi:10.1007/978-3-642-21034-1.

[7] L. Aroyo, H. Welty, Chris Alani, J. Taylor, A. Bernstein,L. Kagal, N. Noy, and E. Blomqvist, editors. The SemanticWeb–ISWC 2011, Berlin Heidelberg, Germany, 2011. Springer.doi:10.1007/978-3-642-25073-6.

[8] S. Athenikos and H. Han. Biomedical Question Answering:A survey. Computer methods and programs in biomedicine,99:1–24, 2010. doi:10.1016/j.cmpb.2009.10.003.

[9] G. Balikas, A. Kosmopoulos, A. Krithara, G. Paliouras, andI. Kakadiaris. Results of the BioASQ track of the QuestionAnswering lab at CLEF 2014. In CLEF 2014 Working Notes,volume 1180 of CEUR Workshop Proceedings, 2014. OnlineWorking Notes.

[10] G. Balikas, A. Kosmopoulos, A. Krithara, G. Paliouras, andI. Kakadiaris. Results of the BioASQ tasks of the QuestionAnswering lab at clef 2015. In CEUR Workshop Proceedings,volume 1391 of CEUR Workshop Proceedings, 2015. OnlineWorking Notes.

[11] D. Bär, C. Biemann, I. Gurevych, and T. Zesch. Ukp: Com-puting semantic textual similarity by combining multiple con-tent similarity measures. In S. Manandhar and D. Yuret, edi-tors, *SEM 2012: The First Joint Conference on Lexical andComputational Semantics, pages 435–440, Stroudsburg, USA,2012. Association for Computational Linguistics.

[12] P. Baudiš and J. Šedivy. Modeling of the Question Answeringtask in the YodaQA system. In Experimental IR Meets Mul-tilinguality, Multimodality, and Interaction, pages 222–228.Springer, Berlin Heidelberg, Germany, 2015. doi:10.1007/978-3-319-24027-5_20.

[13] A. Ben Abacha and P. Zweigenbaum. Medical Question An-swering: Translating medical questions into SPARQL queries.In IHI ’12: Proceedings of the 2nd ACM SIGHIT InternationalHealth Informatics Symposium, pages 41–50, New York, USA,2012. Association for Computing Machinery.

[14] J. Berant and P. Liang. Semantic parsing via paraphrasing. InProceedings of the 52nd Annual Meeting, pages 1415–1425,Stroudsburg, USA, 2014. Association For Computational Lin-guistics. doi:10.3115/v1/P14-1133.

[15] V. Bicer, T. Tran, A. Abecker, and R. Nedkov. KOIOS:Utilizing semantic search for easy-access and visualizationof structured environmental data. In The Semantic Web–ISWC 2011, pages 1–16. Springer, Berlin Heidelberg, Ger-many, 2011. doi:10.1007/978-3-642-25093-4_1.

[16] C. Biemann, S. Handschuh, A. Freitas, F. Meziane, and E. Mé-tais. Natural Language Processing and Information Systems,volume 9103 of Lecture Notes in Computer Science. SpringerPublishing, New York, USA, 2015. doi:10.1007/978-3-319-19581-0.

[17] R. Blanco, P. Mika, and S. Vigna. Effective and efficient entitysearch in RDF data. In L. Aroyo, C. Welty, H. Alani, J. Tay-lor, and A. Bernstein, editors, Proceedings of the 10th Interna-tional Conference on The Semantic Web—Volume Part I, vol-ume 7031 of Lecture Notes in Computer Science, pages 83–97,Berlin Heidelberg, Germany, 2011. Springer. doi:10.1007/978-3-642-25073-6_6.

[18] A. Bordes, J. Weston, and N. Usunier. Open Question Answer-ing with weakly supervised embedding models. In MachineLearning and Knowledge Discovery in Databases, volume8724 of Lecture Notes in Computer Science, pages 165–180.Springer, Berlin Heidelberg, Germany, 2014. doi:10.1007/978-3-662-44848-9_11.

[19] C. Boston, S. Carberry, and H. Fang. Wikimantic: Disam-biguation for short queries. In G. Bouma, A. Ittoo, E. Mé-tais, and H. Wortmann, editors, 17th International Conferenceon Applications of Natural Language to Information Systems,NLDB 2012, pages 140–151, Berlin Heidelberg, Germany,2012. Springer. doi:10.1007/978-3-642-31178-9_13.

[20] A. Both, D. Diefenbach, K. Singh, S. Shekarpour, D. Cherix,and C. Lange. Qanary–An extensible vocabulary for openQuestion Answering systems. In H. Sack, E. Blomqvist,M. d’Aquin, C. Ghidini, S. P. Ponzetto, and C. Lange, edi-tors, The Semantic Web. Latest Advances and New Domains,volume 9678 of Lecture Notes in Computer Science, BerlinHeidelberg, Germany, 2016. Springer.

[21] G. Bouma, A. Ittoo, E. Métais, and H. Wortmann, editors. Nat-ural Language Processing and Information Systems, volume7337 of Lecture Notes in Computer Science, Berlin Heidelberg,Germany, 2012. Springer. doi:10.1007/978-3-642-31178-9.

[22] P. Buitelaar, P. Cimiano, P. Haase, and M. Sintek. Towardslinguistically grounded ontologies. In L. Aroyo, P. Traverso,F. Ciravegna, P. Cimiano, T. Heath, E. Hyvönen, R. Mizoguchi,E. Oren, M. Sabou, and E. Simperl, editors, The Semantic Web:Research and Applications, 6th European Semantic Web Con-ference, ESWC 2009, volume 6643 of Lecture Notes in Com-puter Science, pages 111–125, Berlin Heidelberg, Germany,2009. Springer. doi:10.1007/978-3-642-02121-3_12.

[23] E. Cabrio, A. P. Aprosio, J. Cojan, B. Magnini, F. Gandon, andA. Lavelli. QAKiS @ QALD-2. In Interacting with LinkedData (ILD 2012), page 61, 2012.

20 Challenges of Question Answering in the Semantic Web

[24] E. Cabrio, P. Cimiano, V. Lopez, A. N. Ngomo, C. Unger, andS. Walter. QALD-3: Multilingual Question Answering overLinked Data. In P. Forner, R. Navigli, D. Tufis, and N. Ferro,editors, CLEF2013 Working Notes, 2013.

[25] E. Cabrio, J. Cojan, F. Gandon, and A. Hallili. Querying mul-tilingual DBpedia with QAKiS. In The Semantic Web: ESWC2013 Satellite Events, pages 194–198. Springer, Berlin Heidel-berg, Germany, 2013. doi:10.1007/978-3-642-41242-4_23.

[26] M. Canitrot, T. de Filippo, P.-Y. Roger, and P. Saint-Dizier.The KOMODO system: Getting recommendations on how torealize an action via Question-Answering. In IJCNLP 2011,Proceedings of the KRAQ11 Workshop: Knowledge and Rea-soning for Answering Questions, pages 1–9, 2011.

[27] L. Cappellato, N. Ferro, M. Halvey, and W. Kraaij, editors.CLEF 2014 Working Notes, volume 1180 of CEUR Work-shop Proceedings, 2014. URL http://ceur-ws.org/Vol-1180. Online Working Notes.

[28] C. Carpineto and G. Romano. A survey of Automatic QueryExpansion in information retrieval. ACM Computing Surveys,44:1:1–1:50, 2012. doi:10.1145/2071389.2071390.

[29] D. Carvalho, C. Callı, A. Freitas, and E. Curry. EasyESA:A low-effort infrastructure for explicit semantic analysis. InProc. of 13th International Semantic Web Conf, pages 177–180, 2014.

[30] G. Cheng, T. Tran, and Y. Qu. RELIN: Relatedness andinformativeness-based centrality for entity summarization. InL. Aroyo, C. Welty, H. Alani, J. Taylor, A. Bernstein, L. Ka-gal, N. F. Noy, and E. Blomqvist, editors, The Semantic Web—ISWC 2011, volume 7031 of Lecture Notes in ComputerScience, pages 114–129, Berlin Heidelberg, Germany, 2011.Springer. doi:10.1007/978-3-642-25073-6_8.

[31] P. Cimiano. Flexible semantic composition with dudes.In Proceedings of the Eighth International Conferenceon Computational Semantics, pages 272–276, Stroudsburg,USA, 2009. Association for Computational Linguistics.doi:10.3115/1693756.1693786.

[32] P. Cimiano and M. Minock. Natural language interfaces: Whatis the problem?—A data-driven quantitative analysis. In Nat-ural Language Processing and Information Systems, 15th In-ternational Conference on Applications of Natural Languageto Information Systems, NLDB 2010, volume 6177 of Lec-ture Notes in Computer Science, pages 192–206, Berlin Hei-delberg, Germany, 2010. Springer. doi:10.1007/978-3-642-12550-8_16.

[33] P. Cimiano, O. Corcho, V. Presutti, L. Hollink, and S. Rudolph,editors. The Semantic Web: Semantics and Big Data, volume7882 of Lecture Notes in Computer Science, Berlin Heidelberg,Germany, 2013. Springer. doi:10.1007/978-3-642-38288-8.

[34] J. Cojan, E. Cabrio, and F. Gandon. Filling the gaps amongdbpedia multilingual chapters for Question Answering. InProceedings of the 5th Annual ACM Web Science Conference,pages 33–42, New York, USA, 2013. Association for Comput-ing Machinery. doi:10.1145/2464464.2464500.

[35] P. Cudré-Mauroux, J. Heflin, E. Sirin, T. Tudorache, J. Eu-zenat, M. Hauswirth, J. X. Parreira, J. Hendler, G. Schreiber,A. Bernstein, and E. Blomqvist, editors. The Semantic Web–ISWC 2012, Berlin Heidelberg, Germany, 2012. Springer.doi:10.1007/978-3-642-35176-1.

[36] D. Damljanovic, M. Agatonovic, and H. Cunningham.FREyA: An interactive way of querying Linked Data usingnatural language. In C. Unger, P. Cimiano, V. Lopez, and

E. Motta, editors, Proceedings of the 1st Workshop on Ques-tion Answering Over Linked Data (QALD-1), Co-located withthe 8th Extended Semantic Web Conference, pages 10–23,2011.

[37] M. Damova, A. Kiryakov, K. Simov, and S. Petrov. Mappingthe central LOD ontologies to PROTON upper-level ontology.In P. Shvaiko, J. Euzenat, F. Giunchiglia, H. Stuckenschmidt,M. Mao, and I. Cruz, editors, Proceedings of the 5th Interna-tional Workshop on Ontology Matching (OM-2010) collocatedwith the 9th International Semantic Web Conference (ISWC-2010), volume 689 of CEUR Workshop Proceedings, 2010.URL http://ceur-ws.org/Vol-689/.

[38] M. Damova, D. Dannells, R. Enache, M. Mateva, and A. Ranta.Natural language interaction with semantic web knowledgebases and LOD. Towards the Multilingual Semantic Web,2013.

[39] H. T. Dang, D. Kelly, and J. J. Lin. Overview of the TREC2007 Question Answering track. In E. M. Voorhees andL. P. Buckland, editors, Proceedings of The Sixteenth TextREtrieval Conference, TREC 2007, Gaithersburg, Maryland,USA, November 5-9, 2007, volume Special Publication 500-274. National Institute of Standards and Technology (NIST),2007.

[40] I. Deines and D. Krechel. A German natural languageinterface for Semantic Search. In H. Takeda, Y. Qu,R. Mizoguchi, and Y. Kitamura, editors, Semantic Technol-ogy, volume 7774 of Lecture Notes in Computer Science,pages 278–289. Springer, Berlin Heidelberg, Germany, 2013.doi:10.1007/978-3-642-37996-3_19.

[41] R. Delmonte. Computational Linguistic Text Processing–Lexicon, Grammar, Parsing and Anaphora Resolution. NovaScience Publishers, New York, USA, 2008.

[42] G. Demartini, B. Trushkowsky, T. Kraska, and M. J. Franklin.Crowdq: Crowdsourced query understanding. In CIDR 2013,Sixth Biennial Conference on Innovative Data Systems Re-search, Asilomar, CA, USA, January 6-9, 2013, Online Pro-ceedings. www.cidrdb.org, 2013.

[43] C. Dima. Intui2: A prototype system for Question Answer-ing over Linked Data. In P. Forner, R. Navigli, D. Tufis, andN. Ferro, editors, CLEF2013 Working Notes, 2013.

[44] C. Dima. Answering natural language questions with Intui3.In L. Cappellato, N. Ferro, M. Halvey, and W. Kraaij, editors,CLEF 2014 Working Notes, 2014.

[45] L. Ding, T. Finin, A. Joshi, R. Pan, R. S. Cost, Y. Peng, P. Red-divari, V. Doshi, and J. Sachs. Swoogle: A search and metadataengine for the Semantic Web. In Proceedings of the ThirteenthACM International Conference on Information and Knowl-edge Management, CIKM ’04, pages 652–659, New York, NY,USA, 2004. ACM. .

[46] K. Elbedweihy, S. N. Wrigley, and F. Ciravegna. Improvingsemantic search using query log analysis. In Interacting withLinked Data (ILD 2012), page 61, 2012.

[47] K. Elbedweihy, S. Mazumdar, S. N. Wrigley, and F. Ciravegna.NL-Graphs: A hybrid approach toward interactively query-ing semantic data. In V. Presutti, C. d’Amato, F. Gandon,M. d’Aquin, S. Staab, and A. Tordai, editors, The SemanticWeb: Trends and Challenges, volume 8465 of Lecture Notesin Computer Science, pages 565–579, New York, USA, 2014.Springer Publishing. doi:10.1007/978-3-319-07443-6_38.

[48] A. Fader, L. Zettlemoyer, and O. Etzioni. Paraphrase-drivenlearning for open Question Answering. In Proceedings of the

Challenges of Question Answering in the Semantic Web 21

51nd Annual Meeting, volume 1, pages 1608–1618, Strouds-burg, USA, 2013. Association For Computational Linguistics.

[49] O. Ferrandez, C. Spurk, M. Kouylekov, I. Dornescu, S. Fer-randez, M. Negri, R. Izquierdo, D. Tomas, C. Orasan,G. Neumann, et al. The QALL-ME framework: Aspecifiable-domain multilingual Question Answering archi-tecture. Journal of Web Semantics, 9:137–145, 2011.doi:10.1016/j.websem.2011.01.002.

[50] S. Ferre. SQUALL2SPARQL: A translator from controlledenglish to full SPARQL 1.1. In P. Forner, R. Navigli, D. Tufis,and N. Ferro, editors, CLEF2013 Working Notes, 2013.

[51] S. Ferré. Squall: a controlled natural language as expressiveas sparql 1.1. In Natural Language Processing and Infor-mation Systems, pages 114–125. Springer, Berlin Heidelberg,Germany, 2013. doi:10.1007/978-3-642-38824-8_10.

[52] P. Forner, R. Navigli, D. Tufis, and N. Ferro, editors.CLEF2013 Working Notes, volume 1179 of CEUR Work-shop Proceedings, 2013. URL http://ceur-ws.org/Vol-1179. Online Working Notes.

[53] A. Freitas and E. Curry. Natural language queries over hetero-geneous Linked Data graphs: A distributional-compositionalsemantics approach. In Proceedings of the 19th interna-tional conference on Intelligent User Interfaces, pages 279–288, New York, USA, 2014. Association for Computing Ma-chinery. doi:10.1145/2557500.2557534.

[54] A. Freitas, J. G. Oliveira, E. Curry, S. O’Riain, and J. C. P.da Silva. Treo: Combining entity-search, spreading activa-tion and semantic relatedness for querying Linked Data. InC. Unger, P. Cimiano, V. Lopez, and E. Motta, editors, Pro-ceedings of the 1st Workshop on Question Answering OverLinked Data (QALD-1), Co-located with the 8th Extended Se-mantic Web Conference, pages 24–37, 2011.

[55] A. Freitas, J. G. Oliveira, S. O’Riain, E. Curry, and J. C. P.Da Silva. Querying Linked Data using semantic relatedness: Avocabulary independent approach. In R. Munoz, A. Montoyo,and E. Metais, editors, Natural Language Processing and In-formation Systems, 16th International Conference on Appli-cations of Natural Language to Information Systems, NLDB2011, volume 6716 of Lecture Notes in Computer Science,pages 40–51, Berlin Heidelberg, Germany, 2011. Springer.doi:10.1016/j.datak.2013.08.003.

[56] A. Freitas, E. Curry, J. Oliveira, and S. O’Riain. Queryingheterogeneous datasets on the Linked Data Web: Challenges,approaches, and trends. IEEE Internet Computing, 16:24–33,2012. doi:10.1109/MIC.2011.141.

[57] T. Furche, G. Gottlob, G. Grasso, C. Schallhart, and A. Sellers.OXPath: A language for scalable data extraction, automation,and crawling on the Deep Web. The VLDB Journal, 22:47–72,2013. doi:10.1007/s00778-012-0286-6.

[58] E. Gabrilovich and S. Markovitch. Computing semantic relat-edness using Wikipedia-based explicit semantic analysis. InM. M. Veloso, editor, IJCAI–07, Proceedings of the 20th Inter-national Joint Conference on Artificial Intelligence, volume 7,pages 1606–1611, Palo Alto, USA, 2007. AAAI Press.

[59] F. Gandon, M. Sabou, H. Sack, C. d’Amato, P. Cudré-Mauroux, and A. Zimmermann, editors. The Semantic Web.Latest Advances and New Domains, volume 9088 of Lec-ture Notes in Computer Science, Cham, Switzerland, 2015.Springer Publishing. doi:10.1007/978-3-319-18818-8.

[60] M. Gao, J. Liu, N. Zhong, F. Chen, and C. Liu. Semanticmapping from natural language questions to OWL queries.

Computational Intelligence, 27:280–314, 2011.[61] D. Gerber and A.-C. N. Ngomo. Extracting multilingual

natural-language patterns for RDF predicates. In Knowl-edge Engineering and Knowledge Management, pages 87–96.Springer, Berlin Heidelberg, Germany, 2012. doi:10.1007/978-3-642-33876-2_10.

[62] C. Giannone, V. Bellomaria, and R. Basili. A HMM-based approach to Question Answering against Linked Data.In P. Forner, R. Navigli, D. Tufis, and N. Ferro, editors,CLEF2013 Working Notes, 2013.

[63] A. M. Gliozzo and A. Kalyanpur. Predicting lexical answertypes in open domain QA. International Journal On SemanticWeb and Information Systems, 8(3):74–88, 2012. .

[64] S. Hakimov, H. Tunc, M. Akimaliev, and E. Dogdu. SemanticQuestion Answering system over Linked Data using relationalpatterns. In Proceedings of the Joint EDBT/ICDT 2013 Work-shops, pages 83–88, New York, USA, 2013. Association forComputing Machinery. doi:10.1145/2457317.2457331.

[65] S. Hakimov, C. Unger, S. Walter, and P. Cimiano. Applying se-mantic parsing to Question Answering over Linked Data: Ad-dressing the lexical gap. In C. Biemann, S. Handschuh, A. Fre-itas, F. Meziane, and E. Métais, editors, Natural Language Pro-cessing and Information Systems: 20th International Confer-ence on Applications of Natural Language to Information Sys-tems, NLDB 2015, Passau, Germany, June 17-19, 2015, Pro-ceedings, pages 103–109, Cham, Switzerland, 2015. SpringerPublishing. doi:10.1007/978-3-319-19581-0_8.

[66] T. Hamon, N. Grabar, F. Mougin, and F. Thiessard. Descrip-tion of the POMELO system for the task 2 of QALD-4. InL. Cappellato, N. Ferro, M. Halvey, and W. Kraaij, editors,CLEF 2014 Working Notes, 2014.

[67] B. Hamp and H. Feldweg. GermaNet—A lexical-semantic netfor German. In I. Mani and M. Maybury, editors, Proceedingsof the ACL/EACL ’97 Workshop on Intelligent Scalable TextSummarization, pages 9–15, Stroudsburg, USA, 1997. Associ-ation for Computational Linguistics.

[68] Z. S. Harris. Papers in structural and transformationallinguistics. Springer, Berlin Heidelberg, Germany, 2013.doi:10.1007/978-94-017-6059-1.

[69] S. He, S. Liu, Y. Chen, G. Zhou, K. Liu, and J. Zhao.CASIA@QALD-3: A Question Answering system overLinked Data. In P. Forner, R. Navigli, D. Tufis, and N. Ferro,editors, CLEF2013 Working Notes, 2013.

[70] S. He, Y. Zhang, K. Liu, and J. Zhao. CASIA@V2: A MLN-based Question Answering system over Linked Data. InL. Cappellato, N. Ferro, M. Halvey, and W. Kraaij, editors,CLEF 2014 Working Notes, 2014.

[71] D. M. Herzig, P. Mika, R. Blanco, and T. Tran. Federatedentity search using on-the-fly consolidation. In The SemanticWeb–ISWC 2013, pages 167–183. Springer, Berlin Heidelberg,Germany, 2013. doi:10.1007/978-3-642-41335-3_11.

[72] L. Hirschman and R. Gaizauskas. Natural language QuestionAnswering: The view from here. Natural Language Engineer-ing, 7:275–300, 2001. doi:10.1017/S1351324901002807.

[73] K. Höffner and J. Lehmann. Towards Question Answeringon statistical Linked Data. In Proceedings of the 10th Inter-national Conference on Semantic Systems, pages 61–64, NewYork, USA, 2014. Association for Computing Machinery. .doi:10.1145/2660517.2660521.

[74] K. Höffner, J. Lehmann, and R. Usbeck. CubeQA—Question Answering on RDF Data Cubes, 2016. URL

22 Challenges of Question Answering in the Semantic Web

http://svn.aksw.org/papers/2016/ISWC_cubeqa/public.pdf. Submitted to ISWC 2016.

[75] I. Horrocks, P. F. Patel-Schneider, H. Boley, S. Tabet,B. Grosof, and M. Dean. SWRl: A Semantic Web rule lan-guage combining OWL and RuleML. Technical report, W3C,2004.

[76] R. Huang and L. Zou. Natural language Question Answeringover RDF data. In Proceedings of the 2013 ACM SIGMOD In-ternational Conference on Management of Data, pages 1289–1290, New York, USA, 2013. Association for Computing Ma-chinery. doi:10.1145/2463676.2463725.

[77] P. Jain, P. Hitzler, A. P. Sheth, K. Verma, and P. Z. Yeh. On-tology alignment for Linked Open Data. In Proceedingsof ISWC, pages 402–417, Berlin, Heidelberg, 2010. Springer.doi:10.1007/978-3-642-17746-0_26.

[78] A. K. Joshi, P. Jain, P. Hitzler, P. Z. Yeh, K. Verma, A. P. Sheth,and M. Damova. Alignment-based querying of Linked OpenData. In On the Move to Meaningful Internet Systems: OTM2012, pages 807–824. Springer, Berlin Heidelberg, Germany,2012. doi:10.1007/978-3-642-33615-7_25.

[79] A. Kalyanpur, J. W. Murdock, J. Fan, and C. Welty. Leverag-ing community-built knowledge for type coercion in QuestionAnswering. In The Semantic Web–ISWC 2011, pages 144–156.Springer, Berlin Heidelberg, Germany, 2011. doi:10.1007/978-3-642-25093-4_10.

[80] J.-T. Lee, S.-B. Kim, Y.-I. Song, and H.-C. Rim. Bridginglexical gaps between queries and questions on large onlineq&a collections with compact translation models. In Pro-ceedings of the Conference on Empirical Methods in NaturalLanguage Processing, EMNLP ’08, pages 410–418, Strouds-burg, USA, 2008. Association for Computational Linguistics.doi:10.3115/1613715.1613768.

[81] J. Lehmann and L. Bühmann. AutoSPARQL: Let users queryyour knowledge base. In The Semantic Web: Research andApplications, 8th European Semantic Web Conference, ESWC2011, volume 6643 of Lecture Notes in Computer Science,2011.

[82] J. Lehmann, C. Bizer, G. Kobilarov, S. Auer, C. Becker, R. Cy-ganiak, and S. Hellmann. DBpedia - a crystallization pointfor the web of data. Journal of Web Semantics, 7(3):154–165,2009. .

[83] J. Lehmann, T. Furche, G. Grasso, A.-C. Ngonga Ngomo,C. Schallhart, A. Sellers, C. Unger, L. Bühmann, D. Gerber,K. Höffner, D. Liu, and S. Auer. DEQA: Deep web extractionfor Question Answering. In The Semantic Web–ISWC 2012,Berlin Heidelberg, Germany, 2012. Springer.

[84] J. Lehmann, R. Isele, M. Jakob, A. Jentzsch, D. Kontokostas,P. N. Mendes, S. Hellmann, M. Morsey, P. van Kleef, S. Auer,et al. DBpedia—A large-scale, multilingual knowledge baseextracted from Wikipedia. Semantic Web Journal, 5:1–29,2014.

[85] V. Lopez, V. Uren, M. Sabou, and E. Motta. Is Question An-swering fit for the Semantic Web?: A survey. Semantic WebJournal, 2:125–155, 2011. doi:10.3233/SW-2011-0041.

[86] V. Lopez, C. Unger, P. Cimiano, and E. Motta. EvaluatingQuestion Answering over Linked Data. Journal of Web Se-mantics, 21:3–13, 2013. doi:10.1016/j.websem.2013.05.006.

[87] E. Marx, R. Usbeck, A.-C. Ngomo Ngonga, K. Höffner,J. Lehmann, and S. Auer. Towards an open Question Answer-ing architecture. In SEMANTiCS 2014, 2014.

[88] D. Melo, I. P. Rodrigues, and V. B. Nogueira. CooperativeQuestion Answering for the Semantic Web. In L. Rato andT. G. calves, editors, Actas das Jornadas de Informática daUniversidade de Évora 2011, pages 1–6. Escola de Ciências eTecnologia, Universidade de Évora, 2011.

[89] E. Métais, F. Meziane, M. Sararee, V. Sugumaran, andS. Vadera, editors. Natural Language Processing and In-formation Systems, volume 7934 of Lecture Notes in Com-puter Science, Berlin Heidelberg, Germany, 2013. Springer.doi:10.1007/978-3-642-38824-8.

[90] E. Métais, M. Roche, and M. Teisseire, editors. Natural Lan-guage Processing and Information Systems, volume 8455 ofLecture Notes in Computer Science, New York, USA, 2014.Springer Publishing. doi:10.1007/978-3-319-07983-7.

[91] P. Mika, T. Tudorache, A. Bernstein, C. Welty, C. Knoblock,D. Vrandecic, P. Groth, N. Noy, K. Janowicz, and C. Goble,editors. The Semantic Web–ISWC 2014, Cham, Switzerland,2014. Springer Publishing. doi:10.1007/978-3-319-11915-1.

[92] G. A. Miller. WordNet: A lexical database for en-glish. Communications of the ACM, 38:39–41, 1995.doi:10.1145/219717.219748.

[93] G. A. Miller and W. G. Charles. Contextual correlates ofsemantic similarity. Language and cognitive processes, 6:1–28, 1991. doi:10.1080/01690969108406936.

[94] A. Mishra and S. K. Jain. A survey on Question Answeringsystems with classification. Journal of King Saud University-Computer and Information Sciences, 2015.

[95] A. M. Moussa and R. F. Abdel-Kader. QASYO: A QuestionAnswering system for YAGO ontology. International Journalof Database Theory and Application, 4, 2011.

[96] R. Muñoz, A. Montoyo, and E. Metais, editors. Natural Lan-guage Processing and Information Systems, volume 6716 ofLecture Notes in Computer Science, Berlin Heidelberg, Ger-many, 2011. Springer. doi:10.1007/978-3-642-22327-3.

[97] N. Nakashole, G. Weikum, and F. Suchanek. PATTY: A tax-onomy of relational patterns with semantic types. In EMNLP-CoNLL 2012, 2012 Joint Conference on Empirical Methodsin Natural Language Processing and Computational Natu-ral Language Learning, Proceedings of the Conference, pages1135–1145, Stroudsburg, USA, 2012. Association for Compu-tational Linguistics.

[98] J.-P. Ng and M.-Y. Kan. QANUS: An open-source Question-Answering platform. arXiv preprint arXiv:1501.00311, 2015.

[99] A.-C. Ngonga Ngomo. Link discovery with guaranteed re-duction ratio in affine spaces with Minkowski measures. InP. Cudré-Mauroux, J. Heflin, E. Sirin, T. Tudorache, J. Euzenat,M. Hauswirth, J. X. Parreira, J. Hendler, G. Schreiber, A. Bern-stein, and E. Blomqvist, editors, The Semantic Web–ISWC2012, Berlin Heidelberg, Germany, 2012. Springer-Verlag.

[100] A.-C. Ngonga Ngomo and S. Auer. LIMES—A time-efficientapproach for large-scale link discovery on the web of data. InIJCAI–11, Proceedings of the 24th International Joint Con-ference on Artificial Intelligence, Palo Alto, California, USA,2011. AAAI Press.

[101] A.-C. Ngonga Ngomo, L. Bühmann, C. Unger, J. Lehmann,and D. Gerber. SPARQL2NL—Verbalizing SPARQL queries.In D. Schwabe, V. Almeida, H. Glaser, R. Baeza-Yates, andS. Moon, editors, Proceedings of the IW3C2 WWW 2013 Con-ference, pages 329–332, 2013.

[102] S. Ou and Z. Zhu. An entailment-based Question Answeringsystem over Semantic Web data. In Digital Libraries: For

Challenges of Question Answering in the Semantic Web 23

Cultural Heritage, Knowledge Dissemination, and Future Cre-ation, pages 311–320. Springer, Berlin Heidelberg, Germany,2011. doi:10.1007/978-3-642-24826-9_39.

[103] G. Paliouras and A.-C. N. Ngomo, editors. BioASQ 2013—Biomedical Semantic Indexing and Question Answering, vol-ume 1094 of CEUR Workshop Proceedings, 2013. OnlineWorking Notes.

[104] S. Park, H. Shim, and G. G. Lee. ISOFT at QALD-4: Seman-tic similarity-based Question Answering system over LinkedData. In L. Cappellato, N. Ferro, M. Halvey, and W. Kraaij,editors, CLEF 2014 Working Notes, 2014.

[105] S. Park, S. Kwon, B. Kim, S. Han, H. Shim, and G. G.Lee. Question Answering system using multiple informa-tion source and open type answer merge. In Proceedingsof NAACL-HLT, pages 111–115, 2015. doi:10.3115/v1/N15-3023.

[106] P. F. Patel-Schneider, Y. Pan, P. Hitzler, P. Mika, L. Zhang,J. Z. Pan, I. Horrocks, and B. Glimm, editors. The SemanticWeb–ISWC 2010, Berlin Heidelberg, Germany, 2010. Springer.doi:10.1007/978-3-642-17746-0.

[107] C. Pradel, G. Peyet, O. Haemmerle, and N. Hernandez.SWIP at QALD-3: Results, criticisms and lesson learned.In P. Forner, R. Navigli, D. Tufis, and N. Ferro, editors,CLEF2013 Working Notes, 2013.

[108] V. Presutti, C. d’Amato, F. Gandon, M. d’Aquin, S. Staab, andA. Tordai, editors. The Semantic Web: Trends and Challenges,volume 8465 of Lecture Notes in Computer Science, Cham,Switzerland, 2014. Springer Publishing. doi:10.1007/978-3-319-07443-6.

[109] M.-M. Rahoman and R. Ichise. An automated template se-lection framework for keyword query over Linked Data. InSemantic Technology, pages 175–190. Springer, Berlin Heidel-berg, Germany, 2013. doi:10.1007/978-3-642-37996-3_12.

[110] M. Rani, M. K. Muyeba, and O. Vyas. A hybrid approachusing ontology similarity and fuzzy logic for semantic Ques-tion Answering. In Advanced Computing, Networking andInformatics-Volume 1, pages 601–609. Springer, Berlin Hei-delberg, Germany, 2014. doi:10.1007/978-3-319-07353-8_69.

[111] A. Ranta. Grammatical framework. Journalof Functional Programming, 14:145–189, 2004.doi:10.1017/S0956796803004738.

[112] K. U. Schulz and S. Mihov. Fast string correction with leven-shtein automata. International Journal on Document Analy-sis and Recognition, 5:67–85, 2002. doi:10.1007/s10032-002-0082-8.

[113] S. Shekarpour, A.-C. Ngonga Ngomo, and S. Auer. Querysegmentation and resource disambiguation leveraging back-ground knowledge. In G. Rizzo, P. N. Mendes, E. Charton,S. Hellmann, and A. Kalyanpur, editors, Proceedings of theWeb of Linked Entities Workshop in conjuction with the 11thInternational Semantic Web Conference (ISWC 2012), 2012.

[114] S. Shekarpour, S. Auer, A.-C. Ngonga Ngomo, D. Gerber,S. Hellmann, and C. Stadler. Generating SPARQL queriesusing templates. Web Intelligence and Agent Systems, 11(3),2013.

[115] S. Shekarpour, K. Höffner, J. Lehmann, and S. Auer. Keywordquery expansion on Linked Data using linguistic and seman-tic features. In ICSC 2013 IEEE Seventh International Con-ference on Semantic Computing, Los Alamitos, USA, 2013.IEEE Computer Society. doi:10.1109/ICSC.2013.41.

[116] S. Shekarpour, A.-C. N. Ngomo, and S. Auer. Question An-swering on interlinked data. In D. Schwabe, V. A. F. Almeida,H. Glaser, R. A. Baeza-Yates, and S. B. Moon, editors, Pro-ceedings of the 22st international conference on World WideWeb, pages 1145–1156. International World Wide Web Confer-ences Steering Committee / ACM, 2013. ISBN 978-1-4503-2035-1.

[117] S. Shekarpour, E. Marx, A.-C. N. Ngomo, and S. Auer. SINA:Semantic interpretation of user queries for Question Answer-ing on interlinked data. Journal of Web Semantics, 30:39–51,2015.

[118] S. Shekarpour, K. M. Endris, A. Jaya Kumar, D. Lukovnikov,K. Singh, H. Thakkar, and C. Lange. Question Answeringon Linked Data: Challenges and future directions. In Pro-ceedings of the 25th International Conference Companionon World Wide Web, WWW ’16 Companion, pages 693–698,Geneva, Switzerland, 2016. International World Wide WebConferences Steering Committee. .

[119] Y. Shen, J. Yan, S. Yan, L. Ji, N. Liu, and Z. Chen. Sparsehidden-dynamics conditional random fields for user intent un-derstanding. In Proceedings of the 20st international con-ference on World Wide Web, pages 7–16, Stroudsburg, USA,2011. Association for Computational Linguistics. .

[120] K. Shirai, K. Inui, H. Tanaka, and T. Tokunaga. An empiricalstudy on statistical disambiguation of japanese dependencystructures using a lexically sensitive language model. In Pro-ceedings of Natural Language Pacific-Rim Symposium, pages215–220, 1997.

[121] E. Simperl, P. Cimiano, A. Polleres, O. Corcho, and V. Pre-sutti, editors. The Semantic Web: Research and applications,volume 7295 of Lecture Notes in Computer Science, BerlinHeidelberg, Germany, 2012. Springer. doi:10.1007/978-3-642-30284-8.

[122] K. Sparck Jones. A statistical interpretation of term specificityand its application in retrieval. Journal of documentation, 28:11–21, 1972.

[123] R. Stewart. A demonstration of a natural language query inter-face to an event-based semantic web triplestore. The Seman-tic Web: ESWC 2014 Satellite Events: ESWC 2014 SatelliteEvents, Anissaras, Crete, Greece, May 25-29, 2014, RevisedSelected Papers, 8798:343, 2014.

[124] H. Sun, H. Ma, W.-t. Yih, C.-T. Tsai, J. Liu, and M.-W.Chang. Open domain Question Answering via seman-tic enrichment. In Proceedings of the 24th InternationalConference on World Wide Web, pages 1045–1055, 2015.doi:10.1145/2736277.2741651.

[125] H. Sun, H. Ma, W.-t. Yih, C.-T. Tsai, J. Liu, and M.-W.Chang. Open domain Question Answering via seman-tic enrichment. In Proceedings of the 24th InternationalConference on World Wide Web, pages 1045–1055, NewYork, USA, 2015. Association for Computing Machinery.doi:10.1145/2736277.2741651.

[126] C. Tao, H. R. Solbrig, D. K. Sharma, W.-Q. Wei, G. K. Savova,and C. G. Chute. Time-oriented Question Answering fromclinical narratives using semantic-web techniques. In TheSemantic Web–ISWC 2010, pages 241–256. Springer, BerlinHeidelberg, Germany, 2010.

[127] G. Tsatsaronis, M. Schroeder, G. Paliouras, Y. Almirantis,I. Androutsopoulos, E. Gaussier, P. Gallinari, T. Artieres, M. R.Alvers, M. Zschunke, et al. BioASQ: A challenge on large-scale biomedical semantic indexing and Question Answering.

24 Challenges of Question Answering in the Semantic Web

In 2012 AAAI Fall Symposium Series, 2012.[128] C. Unger and P. Cimiano. Representing and resolving ambigu-

ities in ontology-based Question Answering. In S. Padó andS. Thater, editors, Proceedings of the TextInfer 2011 Workshopon Textual Entailment, pages 40–49, Stroudsburg, USA, 2011.Association for Computational Linguistics.

[129] C. Unger and P. Cimiano. Pythia: Compositional meaning con-struction for ontology-based Question Answering on the Se-mantic Web. In R. Munoz, A. Montoyo, and E. Metais, editors,16th International Conference on Applications of Natural Lan-guage to Information Systems, NLDB 2011, pages 153–160.Springer, Berlin Heidelberg, Germany, 2011. doi:10.1007/978-3-642-22327-3_15.

[130] C. Unger, P. Cimiano, V. Lopez, and E. Motta, editors. Pro-ceedings of the 1st Workshop on Question Answering OverLinked Data (QALD-1), Co-located with the 8th Extended Se-mantic Web Conference, 2011.

[131] C. Unger, L. Bühmann, J. Lehmann, A.-C. Ngonga Ngomo,D. Gerber, and P. Cimiano. Template-based Question Answer-ing over RDF data. In Proceedings of the 21st internationalconference on World Wide Web, pages 639–648, 2012.

[132] C. Unger, P. Cimiano, V. Lopez, E. Motta, P. Buitelaar, andR. Cyganiak, editors. Interacting with Linked Data (ILD2012), volume 913 of CEUR Workshop Proceedings, 2012.URL http://ceur-ws.org/Vol-913.

[133] C. Unger, C. Forascu, V. Lopez, A.-C. N. Ngomo, E. Cabrio,P. Cimiano, and S. Walter, editors. Question Answering overLinked Data (QALD-4), 2014.

[134] R. Usbeck, A.-C. Ngonga Ngomo, L. Bühmann, and C. Unger.HAWK—Hybrid Question Answering over Linked Data.In F. Gandon, M. Sabou, H. Sack, C. d’Amato, P. Cudré-Mauroux, and A. Zimmermann, editors, The Semantic Web.Latest Advances and New Domains, volume 9088 of LectureNotes in Computer Science, New York, USA, 2015. SpringerPublishing.

[135] P. Vossen. A multilingual database with lexical seman-tic networks. Springer, Berlin Heidelberg, Germany, 1998.doi:10.1007/978-94-017-1491-4.

[136] S. Walter, C. Unger, P. Cimiano, and D. Bär. Evaluation ofa layered approach to Question Answering over Linked Data.In P. Cudré-Mauroux, J. Heflin, E. Sirin, T. Tudorache, J. Eu-zenat, M. Hauswirth, J. X. Parreira, J. Hendler, G. Schreiber,A. Bernstein, and E. Blomqvist, editors, The Semantic Web–ISWC 2012, pages 362–374, Berlin Heidelberg, Germany,2012. Springer. doi:10.1007/978-3-642-35173-0_25.

[137] C. Welty, J. W. Murdock, A. Kalyanpur, and J. Fan. A com-parison of hard filters and soft evidence for answer typing inWatson. In P. Cudré-Mauroux, J. Heflin, E. Sirin, T. Tudo-rache, J. Euzenat, M. Hauswirth, J. X. Parreira, J. Hendler,G. Schreiber, A. Bernstein, and E. Blomqvist, editors, The Se-mantic Web–ISWC 2012, pages 243–256, Berlin Heidelberg,Germany, 2012. Springer. doi:10.1007/978-3-642-35173-0_16.

[138] S. Wolfram. The Mathematica Book. Wolfram Media, Cham-paign, USA, 5 edition, 2004.

[139] www2011. WWW ’11: Proceedings of the 20th InternationalConference on World Wide Web, New York, USA, 2011. Asso-ciation for Computing Machinery. doi:10.1145/1963192.

[140] www2012. WWW ’12: Proceedings of the 21st InternationalConference on World Wide Web, New York, USA, 2012. As-sociation for Computing Machinery. doi:10.1145/2187836.

[141] www2013. WWW ’13 Companion: Proceedings of the 22ndInternational Conference on World Wide Web Companion,New York, USA, 2013. Association for Computing Machinery.doi:10.1145/2488388.

[142] www2014. WWW ’14: Proceedings of the 23rd InternationalConference on World Wide Web, New York, USA, 2014. As-sociation for Computing Machinery. doi:10.1145/2566486.

[143] www2015. WWW ’15 Companion: Proceedings of the 24thInternational Conference on World Wide Web Companion,New York, USA, 2015. Association for Computing Machinery.doi:10.1145/2736277.

[144] K. Xu, Y. Feng, and D. Zhao. Xser@ QALD-4: Answeringnatural language questions via Phrasal Semantic Parsing. InL. Cappellato, N. Ferro, M. Halvey, and W. Kraaij, editors,CLEF 2014 Working Notes, 2014.

[145] M. Yahya, K. Berberich, S. Elbassuoni, M. Ramanath,V. Tresp, and G. Weikum. Deep answers for naturally askedquestions on the Web of Data. In Proceedings of the 21stinternational conference on World Wide Web, pages 445–449,Stroudsburg, USA, 2012. Association For Computational Lin-guistics. doi:10.1145/2187980.2188070.

[146] M. Yahya, K. Berberich, S. Elbassuoni, M. Ramanath,V. Tresp, and G. Weikum. Natural language questions for theWeb of Data. In Proceedings of the 2012 Joint Conferenceon Empirical Methods in Natural Language Processing andComputational Natural Language Learning, EMNLP-CoNLL’12, pages 379–390, Stroudsburg, USA, 2012. Association ForComputational Linguistics.

[147] M. Yahya, K. Berberich, S. Elbassuoni, and G. Weikum. Ro-bust Question Answering over the web of Linked Data. InProceedings of the 22nd ACM international conference onConference on information & knowledge management, pages1107–1116, New York, USA, 2013. Association for Comput-ing Machinery.

[148] Z. Yang, E. Garduno, Y. Fang, A. Maiberg, C. McCormack,and E. Nyberg. Building optimal information systems au-tomatically: Configuration space exploration for biomedi-cal information systems. In Q. He, A. Iyengar, W. Ne-jdl, J. Pei, and R. Rastogi, editors, Proceedings of the 22ndACM international conference on Conference on informa-tion & knowledge management, pages 1421–1430, NewYork, USA, 2013. Association for Computing Machinery.doi:10.1145/2505515.2505692.

[149] E. M. G. Younis, C. B. Jones, V. Tanasescu, and A. I. Abdel-moty. Hybrid geo-spatial query methods on the Semantic Webwith a spatially-enhanced index of DBpedia. In N. Xiao, M.-P. Kwan, M. Goodchild, and S. Shekhar, editors, GeographicInformation Science, pages 340–353, Berlin Heidelberg, Ger-many, 2012. Springer. doi:10.1007/978-3-642-33024-7_25.

[150] N. Zhang, J.-C. Creput, W. Hongjian, C. Meurie, andY. Ruichek. Query Answering using user feedback and con-text gathering for Web of Data. In INFOCOMP 2013: TheThird International Conference on Advanced Communicationsand Computation, pages 33–38, Rockport, USA, 2013. CurranPress.

[151] L. Zou, R. Huang, H. Wang, J. X. Yu, W. He, and D. Zhao.Natural language Question Answering over RDF: A graphdata driven approach. In Proceedings of the 2014 ACM SIG-MOD international conference on Management of data, pages313–324, New York, USA, 2014. Association for Computing

Machinery. doi:10.1145/2588555.2610525.