Embed Size (px)
Citation preview
Question Answering on Linked Data:Challenges and Future Directions
Saeedeh ShekarpourKnoesis Research [email protected]
Kemele M. EndrisUniversity of Bonn, [email protected]
Ashwini Jaya KumarFraunhofer IAIS, Germany
[email protected] Lukovnikov
University of Bonn, [email protected]
Kuldeep SinghFraunhofer IAIS, Germany
Harsh ThakkarUniversity of Bonn, [email protected]
Christoph LangeUniversity of Bonn/[email protected]
ABSTRACTQuestion Answering (QA) systems are becoming the inspir-ing model for the future of search engines. While, recently,datasets underlying QA systems have been promoted fromunstructured datasets to structured datasets with seman-tically highly enriched metadata, question answering sys-tems are still facing serious challenges and are therefore notmeeting users’ expectations. This paper provides an ex-haustive insight of challenges known so far for building QAsystems, with a special focus on employing structured data(i.e. knowledge graphs). It thus helps researchers to easilyspot gaps to fill with their future research agendas.
KeywordsQuestion Answering System, Research Challenge, SpeechInterface, Query Understanding, Data Quality, Distributedand Heterogeneous Datasets, Interoperability of Components.
1. INTRODUCTIONThe Web of Data is growing enormously (currently more
than 84 billion triples1). This figure comprises both struc-tured and unstructured data. Still, taking advantage of thisrapidly growing amount of data is challenging. Traditionalinformation retrieval approaches based on keyword searchare user-friendly but fail to exploit the internal structuresof data due to their bag-of-words semantics. For searchinginformation on the Data Web we need similar user friendlyapproaches, i.e. keyword-based interfaces, which rely on theinternal structure of the data.
1observed on 14 October 2015 at http://stats.lod2.eu/
Copyright is held by the International World Wide Web Conference Com-mittee (IW3C2). IW3C2 reserves the right to provide a hyperlink to theauthor’s site if the Material is used in electronic media.WWW’16 Companion, April 11–15, 2016, Montréal, Québec, Canada.ACM 978-1-4503-4144-8/16/04.http://dx.doi.org/10.1145/2872518.2890571.
Question Answering (QA) is a specialized form of informa-tion retrieval. A Question Answering system retrieves exactanswers to questions posed in natural language by the user.While, recently, datasets underlying QA systems have beenpromoted from unstructured datasets to structured datasetswith semantically highly enriched metadata, question an-swering systems are still facing serious challenges and aretherefore not meeting users’ expectations.
Question Answering systems consists of components thatcan be studied and evolved independently. These compo-nents include (1) an input interface for obtaining a query, (2)components for understanding, interpreting, disambiguatingand parsing the query, (3) components accessing and pro-cessing the datasets employed (facing issues such as hetero-geneity, quality and indexing); thus, there are also issues of(4) interoperability among different interacting components.In the following, we elaborately discuss challenges related toeach aspect and consider future research directions. We closewith a conclusion and a roadmap for future work.
2. CHALLENGESIn this section we present question answering challenges
from four different aspects namely, (i) Speech-based inter-face challenge, (ii) query understanding, interpreting, dis-ambiguating and parsing challenges, (iii) data-oriented chal-lenges (iv) interoperability of QA components challenge.
2.1 Speech InterfaceInterfacing speech to QA systems has become a focus of
research for a long time. But, the main focus of researcheffort so far has been spent on interfacing speech to IR-basedQA systems [29, 31, 20], and much less on interfacing speechinput to QA systems based on KGs (knowledge graphs).Typical state-of-the-art IR approaches integrate a speechrecognition (SR) unit directly with the QA system. An effortbeyond merely interfacing the two units is required to enablenatural conversation in question answering system for bothIR and KG methods.
An SR system mainly consists of an acoustic model and alanguage model, where the main objective is to decode whatis uttered by the user. In contrast, a general IR based QAsystem comprises question processing (to extract the query
693
from the input and to determine the answer type), passageretrieval, document retrieval, passage extraction, and finallyanswer selection depending on the relatedness of the namedentities found to the question keyword. The accuracy of rec-ognizing spoken words has a vital influence on the successof the whole QA process. Ex: if ‘Jamshedpur’ (a city in In-dia) is recognised as ‘game shed poor’ (articulation style andduration of utterance is the key difference), then the wholeQA process is altered. The city name which constitute theimportant answer type is not recognised by the QA system.This can be avoided if there is a rich dataset to train a recog-niser but it is not possible to have acoustic training data foran open-domain. Hence speech recognisers are usually builtfor a specified domain. The same applies for QA systems,developing an open-domain QA is a challenge.
With the evolution of neural network based methods forspeech recognition, the whole conventional approach to speechrecognition has changed. Generally, acoustic model and lan-guage model were built as two independent units. The devel-opment of single neural network architecture to transcribean audio input is a breakthrough in the speech recognitionresearch [19, 17, 2]. The recognition accuracy has beentested for a character level transcription and it is indicatedthat a word/sentence level transcription can be made withthe same architecture. In this type of single neural networkbased speech recognition, language model is applied at theoutput of speech recogniser. Following the same methodol-ogy, it is possible to build an end-to-end speech interfacedQA system with deep neural networks. Current research di-rection is towards exploring the interface of speech to knowl-edge graph using deep neural networks.
2.2 Understanding Questions
2.2.1 DiscussionIn the case of full-fledged QA over structured data, for
example over a knowledge base (KB) such as Freebase [6],the question must be translated into a logical representationthat conveys its meaning in terms of entities, relations, typesas well as logical operators. Simpler forms of QA can also beachieved in other ways, however, approaches without formaltranslation can not express certain constraints (e.g. com-parison). The task of translating from NL to a logical form(semantic parsing (SP)) is characterized by the mismatchbetween natural language (NL) and knowledge base (KB).The semantic parsing problem can be divided into two parts:(1) determining KB constituents mentioned in the NL ex-pression and (2) determining how these constituents shouldbe arranged in a logical structure. The mismatch betweenNL and KB brings several problems. One problem is EntityLinking (EL), recognizing parts of NL input that refer toan entity (NER) and determining which named entities aremeant by that part (disambiguation). A central challengein EL is how to take into account the context of an entitymention in order to find the correct meaning (disambigua-tion). Another challenge is finding an optimal set of suitablecandidates for a mention, where the lexicon (mapping be-tween words/phrases and entities) plays an important role.A problem bordering both disambiguation and candidategeneration is the large number of entities a word can referto (e.g. the thousands of possible “John”’s when confrontedwith “John starred in 1984”).
Another problem is relation detection and classification.Given an NL phrase, we want to determine which KB re-lation is implied by the phrase. Sometimes, the relation isexplicitly denoted by a NL constituent, for example verb-mediated statements (e.g. “X married Y ”), in which case alexicon can help a lot to solve the problem. However, in gen-eral, a lexicon-based approach is not sufficient. Sometimesthere are no relation-specific words in the sentence. Some-times prepositions are used, for example “works by Repin”or “cars from Germany” and sometimes the semantics of therelations and the entities/types they connect are lexicalizedas one, for example, “Russian chemists” or “Tolstoy plays”.Such cases require context-based inference, taking into ac-count the semantics of the entities that would be connectedby the to-be-determined relation (which in turn is related toparsing).
Merely linking entities and recognizing the relations isnot sufficient to produce a logical representation that canbe used to query a data source. The remaining problemis to determine the overall logical structure of the NL in-put. This problem becomes difficult for longer, more com-plex sentences, where different linguistic phenomena, suchas coordination and co-reference, must be handled. Formalgrammars, such as CCG, can help to parse NL input. CCGin particular is well-suited for semantic parsing because ofits transparent interface between syntactic structure and un-derlying semantic form. One problem with grammar-basedsemantic parsers is their rigidity, which is not well-suited forincomplete input as often found in real-world QA scenar-ios. Some works have explored learning relaxed grammarsto handle such input.
The straightforward way of training semantic parsers re-quires training data consisting of NL sentences annotatedwith the corresponding logical representation, which are verycumbersome to obtain. Recent works have explored differentways to reduce the annotation effort in order to bypass thischallenge. One proposed way is to train on question-answerpairs instead [5]. Another way is to automatically gener-ate training data from the KB and/or from entity-linkedcorpora [28] (e.g. ClueWeb). Training with paraphrasingcorpora [5] is another technique explored in several works toimprove the range of expressions the system will be able tocover.
2.2.2 Future directionsRecently, impressive advances in different tasks in Artifi-
cial Intelligence have been achieved using deep learning tech-niques. Embedding-based language models, such as Word2Vec [25]and GloVe [26], have helped to improve performance in manyNLP tasks. One of the most interesting and the most promis-ing future directions for semantic parsing and question an-swering is further exploration of deep learning techniques intheir context.
Using deep learning to better understand questions can bedone by using (possibly custom-trained) word (, word senseand entity) embeddings, which capture their syntactic andsemantic properties, as features to improve existing work-flows. However, a “deeper” approach would be to also devisenew models that provide the machine with more freedomto figure out how to accomplish the task. An excellent andvery recent example in NLP is the Dynamic Memory Net-work (DMN [22]), that does not use any manually engineeredfeatures or problem-tailored models, and yet achieves state-
694
of-the-art performance on all tested tasks, which are disjointenough to leave one impressed (POS tagging, co-referenceresolution, sentiment analysis and Question Answering onthe bAbI dataset). The DMN is one of the works focus-ing on attention and memory in deep learning that enablesthe neural network to reason more freely. We share the be-lief that the investigation and application of more advanceddeep learning models (such as DMN) could yield impressiveresults for different tasks in AI, including question answer-ing.
Recursive, convolutional (CNN) and recurrent (RNN) neu-ral networks are widely used in recent neural network-basedapproaches. Convolutional Neural Networks (CNN), a spe-cial case of recursive NNs are well-explored for computervision. Recursive NNs have also been applied for parsingand sentiment analysis. RNNs produce state-of-the-art re-sults in speech processing as well as in NLP because of theirnatural vigor for processing variable-length sequences. Theyhave been applied for machine translation (SMT), languagegeneration (NLG), language modeling and more and are alsofundamental for the success of the DMN and the NTM.
Even though the DMN has not yet been applied to ourtask of structured QA, some recent works, such as the rel-atively simple embedding-based work of Bordes et al. [8](which outperformed ParaSempre [5] on WebQuestions)and the SMT-like SP approach of [13] seem to acknowledgethe promise of neural approaches with embeddings.
An additional interesting direction is the investigation ofjoint models for the sub-problems involved in question inter-pretation (EL, co-reference resolution, parsing, . . . ). Manytasks in NLP depend on each other to some degree, moti-vating the investigation of efficient approaches to make thedecisions for those tasks jointly. For example, co-referenceresolution and EL can benefit from each other as entity in-formation from a KB can serve as quite powerful featuresfor co-reference resolution and co-reference resolution in turncan improve EL as it transfers KB features to phrases whereanaphora refer to entities. Factor Graphs (and Markov Net-works) are by nature very well-suited for explicit joint mod-els). However, a more internal kind of joint inference couldalso be achieved within a neural architecture (e.g. the DMN).
However, it is worth noting that training advanced neu-ral models and explicit joint models can be a difficult taskbecause of the large number of training parameters and co-dependence of these parameters. Deep learning typically re-lies on the availability of large datasets. However, the wholetask to be solved can be divided in two parts, one focusingon representation learning, which can accomplished in anunsupervised setting (with large amounts of data) and thesecond part relying on and possibly fine-tuning the repre-sentations obtained in the first part in a supervised trainingsetting (requiring annotated task-specific data). For explicitjoint models, data capturing the dependence between differ-ent task-specific parts of the models (e.g. annotated for bothEL and co-reference) are required and the efficient trainingof such models is a very relevant current topic of investiga-tion.
The concluding thought is that the further investigationof language (e.g. [25, 26]) and knowledge modeling (e.g. [16])and powerful deep neural architectures with self-regulatingabilities (attention, memory) as well as implicit or explicitjoint models will continue to push the state of the art in QA.Well-designed deep neural architectures, given proper super-
vision and powerful input models, have the potential to learnto solve many different NLU problems robustly with minimalcustomizations, eliminating the need for carefully engineeredfeatures, strict formalisms to extract complex structures orpipelines arranging problem-tailored algorithms. We believethat these lines of research in QA could be the next yellowbrick in the road to true AI, which has fascinated humanitysince the ancient tales of Talos and Yan Shi’s mechanicalmen.
2.3 Data challenges
Indexing Heterogeneous Datasets:.a typical QA system is empirically only as good as the
performance of its indexing module [14]. The performanceof indexing serves as an upper bound to the overall outputof the QA system, since it can process only as much data asis being presented/served to it from the indices. The preci-sion and recall of the system may be good, but if all or mostof the top relevant documents are not indexed in the sys-tem, the system performance suffers and hence does the enduser. Many researchers have compared effectiveness across avariety of indexing techniques. Their studies show improve-ment if multiple techniques were combined compared to anysingle individual indexing technique [27]. In the present sce-nario, information retrieval systems are carefully tailoredand optimized to deliver highly accurate results for specifictasks. Over the years, efforts of developing such task spe-cific systems have been diversified based on a variety of fac-tors discussed in the following. Based on the type of thedata and the application setting, a wide range of indexingtechniques are deployed. They can broadly be categorizedinto three categories based on the format and type of dataindexed, namely: structured (e.g. RDF, SQL, etc.), semi-structured (e.g. HTML, XML, JSON, CSV, etc.) and/orunstructured data (e.g. text dumps). They are further dis-tinguished by the type of technique they use for indexingand/or also by the type of queries that a particular techniquecan address. The different techniques inherently make use ofa wide spectrum of underlying fundamental data structuresin order to achieve the desirable result. Most of the systemsdealing with unstructured or semi-structured data make useof inverted indices and lists for indexing. For structureddatasets, a variety of data structures such as AVL trees, B-Trees, sparse indices, IR trees, etc., have been developed inthe past decades. Many systems combine two or more datastructures to maintain different indices for different data at-tributes. An exhaustive comparision of indexing techniquesused by a wide vareity of question answering systems canbe found from our publically available spreadsheet2. Ourcurrent work in progress is focusing on benchmarking differ-ent datasets such as Wikidata [36], DBpedia [3], and Free-Base [7] against a wide spectrum of indexing library plat-forms, such as Indri3, Solr4, ElasticSearc5, Sphinx6, Neo4j7,Titan8, Xapian9, and Terrier10.
2Indexing Mastersheet at https://goo.gl/FM1LM93http://www.lemurproject.org/indri.php4http://lucene.apache.org/solr/5https://www.elastic.co/products/elasticsearch6http://sphinxsearch.com/7http://neo4j.com/8http://thinkaurelius.github.io/titan/9http://xapian.org/
10http://www.terrier.org/
695
Data Quality Challenge: .recent advancements in the fields of Web of Data and Data
Science have led to an outburst of standards related to struc-tured data11 such as RDF(a), Linked Data, schema.org, etc.,to an increasing amount of such data, and to a wide rangeof tools to produce, manage and consume such data. Tobe available for ready consumption, especially in open ques-tion answering systems, any such data sources should meet acertain level of quality, e.g., defined by benchmarks. Qual-ity can generally be defined as “fitness for use”, but thereare a lot of concrete factors that influence a dataset’s fit-ness for use in question answering12 settings and in spe-cific application domains. Recently, a number of researchactivities have been concerned with automating the assess-ment of linked data quality. Debattista, who has developedone such tool (Luzzu [12]), provides an overview of otherstate-of-the-art tools [12], including one by Flemming [15],as well as Sieve [24], RDFUnit [21], TripleCheckMate [38],LinkQA [18], and LiQuate [30]. In this section, we summa-rize the concrete criteria by which the quality of linked datacan be assessed, with a special focus on those criteria thatare relevant to question answering.
In a comprehensive review of literature and systems, Sa-vors et al. [37] have identified the dimensions of linked dataquality and categorized them as follows:
• Accessibility dimensions: This category covers as-pects related to retrieving and accessing data, whichincludes full or partial access and different technicalmeans of access (e.g. the possibility to download adata dump vs. the availability of a SPARQL endpoint,i.e. a standardized query interface). (i) Availabilityis generally defined as the ease of access with whichparticular information is obtainable or rapidly retriev-able for readily consumption. In a linked data context,availability can be referred to as the accessibility of aSPARQL endpoint or RDF dumps or dereferenceableURIs. (ii) Interlinking is relevant as it refers to thedata integration and interoperability. The output ofinterlinking is a linkset, i.e. a set of RDF triples link-ing subjects and recognized related objects. (iii) Secu-rity denotes the degree to which a particular datasetis resistant to misuse or alteration without appropri-ate user access rights. (iv) Verifiability, usually by anunbiased third party, addresses the authenticity andcorrectness of the dataset. Verifiability is typically en-abled by provenance metadata.
• Intrinsic dimensions: This category covers aspectsthat are independent of the user’s context, or the outof the application context aAS such as accuracy andconsistency. (i) Accuracy refers to the degree of adataset correctly representing the captured real worldfacts and figures in the form of information with highprecision. (ii) Consistency refers to the independencefrom logical, formal or representational contradictionsof a dataset with respect to others. (iii) Completenessis referred to as the degree to which information in the
11The amount not only of structured, but also of semi-structured andunstructured data available online is also steadily increasing; however,for the purpose of our work we assume that such data has first beentranslated to the RDF data model using standard tools, e.g. from theLinked Data Stack [4].
12In this section, we do not abbreviate “question answering” as “QA”to avoid confusion with “quality assessment”.
dataset is complete or not missing. The dataset shouldhave all the required objects or values for a given taskin order to be considered as complete. Thus, arguingintuitively, completeness is one of the concrete metricsfor linked data quality assessment.
• Contextual dimensions: This category is concernedwith the context of the task being pursued. (i) Timeli-ness is concerned with the freshness of data over timeor timeliness, i.e. the regularity of updates or mergesand so on. (ii) Understandability can be achieved byproviding appropriate human readable annotations toa dataset and its entities, and by consistently followinga certain regular expression as a pattern for formingentity URIs. (iii) Trustworthiness is concerned withthe reliability or trustworthiness of the data and itssource.
• Representational dimensions: This category is con-cerned with the design and representation of the dataand its schema. For instance, understandability andinterpretability. Interpretability refers to adhering tothe standard practice of representing information usingappropriate notations, symbols, units and languages.
Data quality dimensions in all of these categories canbe relevant in question answering scenarios. In a prelim-inary study [34], we evaluated few selected metrics men-tioned above on two popular datasets of linked data namely,Wikidata and DBpedia13. We evaluated the subsections ex-tracted from these datasets on categories such as “politi-cians”, “Movies”, “Restaurants” and “Soccer players”. Theslices have been made available to the common public andcan be found: DBpedia slices14, Wikidata slices15. The de-tailed information regarding the data slice statistics can befound from the work [34] which is selected from a detailedreport made public at https://goo.gl/ignzzI. For thisevaluation we have obtained four slices of both DBpediaand Wikidata, namely Restaurants, Politicians, Films andSoccer players. From this preliminary study, we could sofar observe that Wikidata dominates DBpedia for the con-sidered setting. The detailed scores and discussion can befound from the mentioned work and the spreadsheet.
Our next step is (i) implement and evaluated the pendingmetric from the above work and (ii) to identify more system-atically what other dimensions and metrics of data qualityare specifically relevant in the typical application domains ofquestion answering, or sufficient for determining a dataset’s“fitness” for question answering. Having identified such di-mensions, we have two goals: (a) identifying datasets thatare suitable for question answering at all, and (b) evaluatethese metrics on a major part of the LOD Cloud16 datasets,identifying more specifically what quality problems they stillsuffer from.
Regarding implementation, in our recent study [34], weevaluated the results on DBpedia and Wikidata slices usingthe metrics that the Luzzu linked data quality assessmentframework already provides. We look forward to extendingLuzzu to use in a question answering setting, that further
13Freebase used to be another popular cross-domain dataset but sup-port for it has expired, which is why we did not consider it; cf.https://www.freebase.com/.
14https://goo.gl/Kn6Fom15https://goo.gl/5aTkLp16LOD Cloud: http://lod-cloud.net/
696
existing implementations of metrics in Luzzu can be specif-ically adapted to make them suitable for quality assessmentrelated to question answering, and that, finally, Luzzu’s flex-ible extensibility even enables us to implement new metricsthat may be required. In summary, our near-future work willbe concerned with defining a generally and flexibly applica-ble framework for automating the process of rigorously as-sessing the quality of linked datasets for question answeringby identifying, formalizing and implementing the requiredmetrics.
Distributed Heterogeneous Datasets.The decentralized architecture of the Web has produced a
wealth of knowledge distributed across different data sourcesand different data types. Question answering systems con-sume different types of data: structured, semi-structured orunstructured data. Most question answering systems uses ei-ther of these types of data to answer user queries. Only fewsystems exploit the wealth of data on the Web by combin-ing these types of data. Hybrid question answering systemsare able to answer queries by combining both structuredand unstructured types of data. HAWK [35], for instance,provides entity search for hybrid question answering usingLinked Data and textual data. HAWK is able to achieve anF-measure of up to 0.68 on the QALD-4 benchmark.
Most question answering systems today uses a single sourceto answer users question. It should rather be possible toanswer questions imposed by a user by combining differ-ent interconnected sources. The challenges imposed by thedistributed nature of the Web are, on the one hand, find-ing the right sources that can answer user query and, onthe other hand, integrating partial answers found from dif-ferent sources. Source selection is one of the challenges infederated question answering approaches. In [32], the au-thors presented an approach to construct a federated queryfrom user supplied (natural language) questions using dis-ambiguated resources. Answers may come from differentsources which have different data quality and trust levels,ranking and fusion of data should be applied to select thebest sources. The amount of data to be used to answer users’queries should also be balanced with the response time.
2.4 Interoperability ChallengeThe field of QA is so vast that the list of different QA
systems can go long. Many Question Answering systemsBased on specific domains have been developed. Domain-specific QA systems, for example [1] are limited to a specificknowledge, for example medicine. They are known as closeddomain QA systems. However, when scope is limited toan explicit domain or ontology, there are less chances ofambiguity and high accuracy of answers. It is also difficultand costly to extend closed domain systems to a new domainor reusing it in implementing a new system.To overcome thelimitations of closed domain QA systems, researchers haveshifted their focus to open domain QA systems. FREyA [11],QAKiS [10], and PowerAqua [23] are few examples of opendomain QA systems which use publicly available semanticknowledge for example DBpedia [3].
While many of these system achieved significant perfor-mance for special use cases, a shortage was observed in allof them. We figured out that the existing QA systems suf-fer from the following drawbacks: (1) potential of reusingits components is very weak, (2) extension of the compo-
nents is problematic, and (3) interoperability between theemployed components are not systematically defined. Thereis little, but a work towards interoperable architecture, e.g.QA archiecture developed by OKBQA17. Interoperability ofdifferent QA tools and components is required to enhanceQA process which is still missing at the conceptual level andcurrently more focused on implementation details. There-fore, there is a need for a descriptive approach that define aconceptual view of QA systems. This approach must coverall needs of current QA systems and be abstracted from im-plementation details. Moreover it must be open such that itcan be used in future QA systems. The generalized approachfor architecture or ontology of a QA system and semanticsearch must focus to bring all state-of-the advancement ofQA under a single umbrella [33]. We envisioned that a gen-eralized vocabulary for QA will be an abstraction level ontop of all the existing QA approaches and will provide inter-operability and exchangeability between them. This gener-alized vocabulary can be further used to integrate differentcomponents and web services within a QA system [9].
3. CONCLUSION AND FUTURE ROADMAPIn this paper, we presented an exhaustive overview of all
the open challenges being still controversial for developinga question answering system. The intuition is that LinkedData which provides advantages such as semantic metadataand interlinked dataset can influence all of the four majorelements (i.e. interface, parsing, data and component in-teroperability) which play a key role in Question Answeringsystems. As our future research agenda, we are steeringour research on all of the discussed issues with the focusof employing Linked Data technology to promote questionanswering capabilities.
Acknowledgments Parts of this work received fundingfrom the European Union’s Horizon 2020 research and in-novation program under the Marie Sklodowska-Curie grantagreement No. 642795 (WDAqua project).
References[1] A. B. Abacha and P. Zweigenbaum. “Medical
question answering: translating medical questionsinto sparql queries”. In: ACM International HealthInformatics Symposium, IHI ’12, Miami, FL, USA,January 28-30, 2012. 2012.
[2] D. Amodei et al. “Deep Speech 2: End-to-End SpeechRecognition in English and Mandarin”. In: arXivpreprint arXiv:1512.02595 (2015).
[3] S. Auer et al. “DBpedia: A Nucleus for a Web ofOpen Data”. In: The Semantic Web(ISWC+ASWC). 2007.
[4] S. Auer, J. Lehmann, and A.-C. Ngonga Ngomo.“Introduction to Linked Data and Its Lifecycle on theWeb”. In: Reasoning Web. 2011.
[5] J. Berant and P. Liang. “Semantic parsing viaparaphrasing”. In: ACL. Vol. 7. 1. 2014.
[6] K. Bollacker et al. “Freebase: a collaborativelycreated graph database for structuring humanknowledge”. In: 2008 ACM SIGMOD internationalconference on Management of data. ACM. 2008.
17Open Knowledge base and Question Answering (http://okbqa.org)
697
[7] K. D. Bollacker, R. P. Cook, and P. Tufts. “Freebase:A Shared Database of Structured General HumanKnowledge”. In: Twenty-Second AAAI Conference onArtificial Intelligence, July 22-26, 2007, Vancouver,British Columbia, Canada. 2007.
[8] A. Bordes, S. Chopra, and J. Weston. “Questionanswering with subgraph embeddings”. In: arXivpreprint arXiv:1406.3676 (2014).
[9] A. Both et al. “Qanary – An Extensible Vocabularyfor Open Question Answering Systems”. In: ESWC.2016. Forthcoming.
[10] E. Cabrio et al. “QAKiS: an Open Domain QASystem based on Relational Patterns”. In: Proc. ofthe ISWC 2012 Posters & Demonstrations. 2012.
[11] D. Damljanovic, M. Agatonovic, andH. Cunningham. “FREyA: An Interactive Way ofQuerying Linked Data Using Natural Language”. In:ESWC Workshops. 2011.
[12] J. Debattista, S. Auer, and C. Lange. “Luzzu – AFramework for Linked Data Quality Analysis”. In:Semantic Computing. 2016. url:http://eis.iai.uni-
bonn.de/upload/paper/ICSC_2016_paper_17.pdf.
[13] L. Dong and M. Lapata. “Language to Logical Formwith Neural Attention”. In: arXiv preprint (2016).
[14] X. Dong and A. Halevy. “Indexing dataspaces”. In:2007 ACM SIGMOD international conference onManagement of data. ACM. 2007.
[15] A. Flemming. Quality Characteristics of Linked DataPublishing Datasources. 2010.
[16] A. Garcıa-Duran, A. Bordes, and N. Usunier.“Effective Blending of Two and Three-wayInteractions for Modeling Multi-relational Data”. In:Machine Learning and Knowledge Discovery inDatabases. Springer, 2014.
[17] A. Graves and N. Jaitly. “Towards end-to-end speechrecognition with recurrent neural networks”. In: 31stInternational Conference on Machine Learning(ICML-14). 2014.
[18] C. Gueret et al. “Assessing linked data mappingsusing network measures”. In: The Semantic Web:Research and Applications. Springer, 2012.
[19] A. Y. Hannun et al. “Deep Speech: Scaling upend-to-end speech recognition”. In: CoRRabs/1412.5567 (2014).
[20] G. Hu et al. “SpeechQoogle: An Open-DomainQuestion Answering System with Speech Interface”.In: (2006).
[21] D. Kontokostas et al. “Test-driven evaluation oflinked data quality”. In: 23rd international conferenceon World Wide Web. ACM. 2014.
[22] A. Kumar et al. “Ask me anything: Dynamic memorynetworks for natural language processing”. In: arXivpreprint arXiv:1506.07285 (2015).
[23] V. Lopez et al. “Poweraqua: Supporting users inquerying and exploring the semantic web”. In:Semantic Web 3.3 (2011).
[24] P. N. Mendes, H. Muhleisen, and C. Bizer. “Sieve:linked data quality assessment and fusion”. In: 2012Joint EDBT/ICDT Workshops. ACM. 2012.
[25] T. Mikolov et al. “Efficient estimation of wordrepresentations in vector space”. In: arXiv preprint(2013).
[26] J. Pennington, R. Socher, and C. D. Manning.“Glove: Global vectors for word representation”. In:Empiricial Methods in Natural Language Processing(EMNLP 2014) 12 (2014).
[27] T. Rajashekar and B. W. Croft. “Combiningautomatic and manual index representations inprobabilistic retrieval”. In: Journal of the Americansociety for information science 46.4 (1995).
[28] S. Reddy, M. Lapata, and M. Steedman. “Large-scalesemantic parsing without question-answer pairs”. In:Transactions of the Association for ComputationalLinguistics 2 (2014).
[29] P. Rosso et al. “On the Voice-Activated QuestionAnswering”. In: IEEE Trans. on Systems, Man andCybernetics-Part C: Applications and Reviews 42.1(Jan 2012).
[30] E. Ruckhaus, O. Baldizan, and M.-E. Vidal.“Analyzing Linked Data Quality with LiQuate”. In:On the Move to Meaningful Internet Systems: OTM2013 Workshops. Springer. 2013.
[31] E. Sanchis et al. “Spoken QA based on a passageretrieval engine”. In: Spoken Language TechnologyWorkshop, 2006. IEEE. IEEE. 2006.
[32] S. Shekarpour, A.-C. Ngonga Ngomo, and S. Auer.“Question Answering on interlinked data”. In:WWW. 2013.
[33] K. Singh et al. “Towards a Message-DrivenVocabulary for Promoting the Interoperability ofQuestion Answering Systems”. In: SemanticComputing. 2016.
[34] H. Thakkar et al. “Are Linked Datasets fit forOpen-domain Question Answering? A QualityAssessment”. In: (2016).
[35] R. Usbeck et al. “HAWK – Hybrid QuestionAnswering using Linked Data”. In: (2014).
[36] D. Vrandecic and M. Krotzsch. “Wikidata: a freecollaborative knowledgebase”. In: Communications ofthe ACM 57.10 (2014).
[37] A. Zaveri et al. “Quality Assessment for LinkedData”. In: Semantic Web Journal preprint.preprint(2015).
[38] A. Zaveri et al. “User-driven quality evaluation ofdbpedia”. In: 9th International Conference onSemantic Systems. ACM. 2013.
698
