Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 56–62July 5 - July 10, 2020. c©2020 Association for Computational Linguistics

56

EVIDENCEMINER: Textual Evidence Discovery for Life SciencesXuan Wang1, Yingjun Guan2, Weili Liu1, Aabhas Chauhan1, Enyi Jiang1, Qi Li3,

David Liem4, Dibakar Sigdel4, J. Harry Caufield4, Peipei Ping4, Jiawei Han1

1Department of Computer Science, University of Illinois at Urbana-Champaign2School of Information Sciences, University of Illinois at Urbana-Champaign

3Department of Computer Science, Iowa State University4School of Medicine, University of California, Los Angeles

1,2{xwang174,yingjun2,weilil2,aabhasc2,enyij2,hanj}@illinois.edu, 3qli@iastate.edu,4{dliem,dsigdel,jcaufield,pping}@mednet.ucla.edu

Abstract

Traditional search engines for life sciences(e.g., PubMed) are designed for document re-trieval and do not allow direct retrieval of spe-cific statements. Some of these statementsmay serve as textual evidence that is key totasks such as hypothesis generation and newfinding validation. We present EVIDENCEM-INER, a web-based system that lets users querya natural language statement and automaticallyretrieves textual evidence from a backgroundcorpora for life sciences. EVIDENCEMINERis constructed in a completely automated waywithout any human effort for training data an-notation. It is supported by novel data-drivenmethods for distantly supervised named entityrecognition and open information extraction.The entities and patterns are pre-computedand indexed offline to support fast onlineevidence retrieval. The annotation resultsare also highlighted in the original documentfor better visualization. EVIDENCEMINERalso includes analytic functionalities such asthe most frequent entity and relation summa-rization. EVIDENCEMINER can help scien-tists uncover essential research issues, lead-ing to more effective research and more in-depth quantitative analysis. The system ofEVIDENCEMINER is available at https://evidenceminer.firebaseapp.com/1.

1 Introduction

Search engines on scientific literature have beenwidely used by life scientists for discoveries basedon prior knowledge. Each day, millions of usersquery PubMed2 and PubMed Central3 (PMC) fortheir information needs in biomedicine (Allot et al.,2019). However, traditional search engines for lifesciences (e.g., PubMed) are designed for document

1A brief demo of EVIDENCEMINER is available athttps://youtu.be/iYuQ6gsr--I.

2https://www.ncbi.nlm.nih.gov/pubmed/3https://www.ncbi.nlm.nih.gov/pmc/

retrieval and do not allow direct retrieval of spe-cific statements (Lu, 2011; Ren et al., 2017; Shenet al., 2018). With the results from those searchengines, scientists still need to read a large numberof retrieved documents to find specific statementsas textual evidence to validate the input query. Thistextual evidence is key to tasks such as develop-ing new hypotheses, designing informative experi-ments, or comparing and validating new findingsagainst previous knowledge.

While the last several years have witnessed sub-stantial growth in interests and efforts in evidencemining (Lippi and Torroni, 2016; Wachsmuth et al.,2017; Stab et al., 2018; Chen et al., 2019; Majithiaet al., 2019; Chernodub et al., 2019; Allot et al.,2019), little work has been done for evidence min-ing system development in the scientific literature.A significant difference between evidence in thescientific literature and evidence in other corpora(e.g., the online debate corpus) is that scientificevidence usually does not have a strong sentiment(i.e., positive, negative or neutral) in the opinionit holds. Most scientific evidence sentences areobjective statements reflecting how strongly theysupport a query statement. Therefore, if scien-tists are interested in finding textual evidence for“melanoma is treated with nivolumab”, they mayexpect a ranked list of statements with the top oneslike “bicytopenia in primary lung melanoma treatedwith nivolumab” as the textual evidence that sup-ports the input query.

This paper presents EVIDENCEMINER, a web-based system for textual evidence discovery forlife sciences (Figure 1). Given a query as a nat-ural language statement, EVIDENCEMINER auto-matically retrieves sentence-level textual evidencefrom a background corpora of biomedical litera-ture. EVIDENCEMINER is constructed in a com-pletely automated way without any human effortfor training data annotation. It is supported by

57

novel data-driven methods for distantly supervisednamed entity recognition and open informationextraction. EVIDENCEMINER relies on externalknowledge bases to provide distant supervisionfor named entity recognition (NER) (Shang et al.,2018b; Wang et al., 2018b, 2019). Based on theentity annotation results, it automatically extractsinformative meta-patterns (textual patterns contain-ing entity types, e.g., CHEMICAL inhibit DIS-EASE) from sentences in the background corpora.(Jiang et al., 2017; Wang et al., 2018a; Li et al.,2018a,b). Sentences with meta-patterns that bet-ter match the query statement is more likely to betextual evidence. The entities and patterns are pre-computed and indexed offline to support fast on-line evidence retrieval. The annotation results arealso highlighted in the original document for bet-ter visualization. EVIDENCEMINER also includesanalytic functionalities such as the most frequententity and relation summarization. The contribu-tions and features of the EVIDENCEMINER systemare summarized as follows.

1. We build EVIDENCEMINER, a web-based sys-tem for textual evidence discovery for lifesciences. EVIDENCEMINER is supported bynovel methods for distantly supervised namedentity recognition and pattern-based open in-formation extraction.

2. The retrieved evidence sentences can be easilylocated in the original text. The entity and rela-tion annotation results are also highlighted inthe original document for better visualization.

3. Analytic functionalities are included such asfinding the most frequent entities/relationsfor given entity/relation types and finding themost frequent entities given a relation typewith another entity.

2 Related Work

Search engines performing sentence-level retrievalhave been developed in the biomedical domain.For example, Textpresso (Muller et al., 2004) high-lights the query-related sentences in the retrieveddocuments. However, the sentence highlighting isonly based on query word matching, which doesnot necessarily find sentences semantically relatedto the input query. Another example is LitSense(Allot et al., 2019), which retrieves semanticallysimilar sentences in biomedical literature given

User Query

User

Text Evidence Retrieval

Annotation Result Visualization

Entity/Relation Summarization

Corpora Knowledge Bases

Algorithm Pool

Meta-pattern Discovery: MetaPAD, TruePIE, CPIE, WWPIE

StorageMetadata Pattern

IndexFull-text

Index

Distantly-supervised NER: AutoNER, AutoBioNER, PeNNER

EvidenceMiner

Figure 1: System architecture of EVIDENCEMINER.

a query sentence. It returns best-matching sen-tences using a combined approach of traditionalword matching and neural embedding. However,their neural embeddings are noisy and thus nega-tively impact the effectiveness in retrieving query-specific evidence sentences. EVIDENCEMINER ismore effective compared with LitSense for textualevidence retrieval in biomedical literature.

Similar tools are also developed for other do-mains, such as claim mining and argument miningtools on Twitter or news articles. PerspectroScope(Chen et al., 2019) allows users to query a nat-ural language claim and extract textual evidencein support or against the claim. ClaimPortal (Ma-jithia et al., 2019) is an integrated infrastructure forsearching and checking factual claims on Twitter.TARGER (Chernodub et al., 2019) is an argumentmining framework for tagging arguments in thefree input text and keyword-based retrieval of argu-ments from the argument-tagged corpus. Most ofthese tools rely on fully supervised methods that re-quire human-annotated training data. It is difficultto directly apply these systems to other domains,such as life sciences since it is non-trivial to re-trieve the set of human-annotated articles and theannotations are prone to errors (Levy et al., 2017).

3 System Description

EVIDENCEMINER consists of two major compo-nents: an open information extraction pipeline anda textual evidence retrieval and analysis pipeline.The open information extraction pipeline includestwo functional modules: (1) distantly supervisedNER, and (2) meta-pattern-based open informationextraction; whereas the textual evidence retrievaland analysis pipeline includes three functional mod-ules: (1) textual evidence search, (2) annotation

58

Background corpora Cancers Heart Diseases#PubMed abstracts 48,201 11,766

#PMC full-text papers 7,130 1,151#Sentences in total 1,466,091 246,106#Entity instances 3,315,092 400,327

#Relation instances 29,160 9,576

Table 1: Basic statistics of background corpora. It in-cludes PubMed abstracts and PMC full-text papers re-lated to cancers and heart diseases published in 2019.

result visualization in the original document, and(3) the most frequent entity and relation summa-rization. Figure 1 shows the system architectureof EVIDENCEMINER. The functional modules areintroduced in the following sections.

3.1 Open Information ExtractionThe open information extraction pipeline extractsentities with distant supervision from knowledgebases and relations with automatic meta-patterndiscovery methods. In particular, to extract high-quality entities and relations, we design noise-robust neural models for distantly supervisednamed entity recognition (Shang et al., 2018b;Wang et al., 2019) and wide-window meta-patterndiscovery methods to deal with the long and com-plex sentences in biomedical literature (Wang et al.,2018a; Li et al., 2018b).Data Collection. To obtain the background cor-pora for EVIDENCEMINER, we collect the ti-tles and abstracts of 26M papers from the entirePubMed4 dump, and the full-text contents of 2.2Mpapers from PubMed Central5 (PMC). For thedemonstration purpose, we select a subset of docu-ments published in 2019 that are specifically relatedto two important diseases (cancers and heart dis-eases) to form the background corpora. The subsetof documents are selected by concept matching onMeSH6, a biomedical concept ontology with theconcepts related to cancers (Neoplasms) and heartdiseases (Cardiovascular Diseases). Table 1 sum-marizes the statistics of the background corpora.Distantly Supervised Named Entity Recogni-tion. EVIDENCEMINER relies on UMLS7, a com-prehensive biomedical knowledge base to pro-vide distant supervision for named entity recog-nition. We select 5 major biomedical entity types(Organism, Fully Formed Anatomical Structure,

4https://pubmed.gov/pubmed5https://pubmed.gov/pmc6https://www.nlm.nih.gov/mesh/7https://www.nlm.nih.gov/research/

umls/index.html

Chemical, Physiologic Function, and PathologicFunction) including 17 fine-grained entity types(Archaeon, Bacterium, Eukaryote, Virus, BodyPart/Organ/Organ Component, Tissue, Cell, CellComponent, Gene or Genome, Chemical, Organ-ism Function, Organ or Tissue Function, CellFunction, Molecular Function, Disease or Syn-drome, Cell or Molecular Dysfunction, Experimen-tal Model of Disease, and Pathological Function)from UMLS as the entity types to be annotated. Totackle the problem of limited coverage of the inputdictionary, we first apply a data-driven phrase min-ing algorithm, AutoPhrase (Shang et al., 2018a),to extract high-quality phrases as additional entitycandidates. Then we automatically expand the dic-tionary with a novel dictionary expansion method(Wang et al., 2019). The expanded dictionary isused to label the input corpora with the 17 fine-grained entity types to train a neural model. Weapply AutoNER (Shang et al., 2018b), a state-of-the-art distantly supervised NER method that effec-tively deals with noisy distant supervision. Com-paring with PubTator (Wei et al., 2013), a state-of-the-art BioNER system trained with extensivehuman annotation on 5 biomedical entity types,EVIDENCEMINER can automatically annotate 17fine-grained entity types with high quality withoutany human effort for training data annotation.

Meta-pattern Discovery. Based on the entity an-notation results above, meta-patterns can be auto-matically discovered from the corpora to supporttextual evidence discovery. Meta-patterns are de-fined as sub-sequences in an entity-type-replacedcorpus with at least one entity type token in it. Forexample, “PPAR gamma agonist” and “caspase 1agonist” are two word-sequences in the raw cor-pus. If we replace all the entities (i.e., “PPARgamma” and “caspase 1”) with their correspond-ing entity types (i.e., $GENE) in the raw corpus,“PPAR gamma agonist” and “caspase 1 agonist”are represented as one meta-pattern “$GENE ag-onist” in the entity-type-replaced corpus. Meta-patterns containing at least two entity types (e.g.,“$CHEMICAL induce $DISEASE”) are relationalmeta-patterns. Quality relational meta-patterns canserve as informative textual patterns that guide tex-tual evidence discovery. We apply two state-of-the-art meta-pattern discovery methods, CPIE (Wanget al., 2018a) and WW-PIE (Li et al., 2018b), toextract high-quality meta-patterns from the NER-tagged corpora. Both methods are specifically de-

59

signed to better deal with the long and complexsentence structures in the biomedical literature. InEVIDENCEMINER, we combine the meta-patternextraction results from CPIE and WW-PIE as ourinformative meta-patterns to guide textual evidenceretrieval. We use Elasticsearch8 to create the indexfor each sentence for fast online retrieval. In addi-tion to indexing the keywords, we index each sen-tence with the meta-patterns it matches and the cor-responding entities extracted by the meta-patternsin the sentence.

3.2 Textual Evidence Retrieval and Analysis

The textual evidence retrieval and analysis pipelineretrieves textual evidence given a user-input querystatement and the indexed corpora. The retrievedevidence sentence can be easily located in the orig-inal text. The entity and relation annotation resultsare also highlighted in the text for better visual-ization. EVIDENCEMINER also includes analyticfunctionalities such as finding the most frequententities and relations as summarization.Textual Evidence Search. Given a user-inputquery statement and the indexed corpora, EVI-DENCEMINER retrieves and ranks the candidatesentences with a combined approach of keywordweighting and meta-pattern weighting. Sentenceswith meta-patterns that better match the query state-ment are ranked higher as textual evidence. Thisranking mechanism is more effective comparedwith existing methods (e.g., LitSense) for textualevidence retrieval in biomedical literature (see Sec-tion 4). We use Elasticsearch to support keywordand meta-pattern search over the indexed corpora.

In Figure 2, we show an example of our searchinterface. For example, if scientists are interestedin finding the textual evidence for “melanoma istreated with nivolumab”, they can search it in EVI-DENCEMINER and see the top results such as “bi-cytopenia in primary lung melanoma treated withnivolumab” (Figure 2a). If they click one of thetop results, the retrieved sentence is highlighted inthe original article (Figure 3) on the annotation in-terface. Moreover, EVIDENCEMINER allows moreflexible queries, such as a mixture of keywords andrelational patterns. For example, if scientists areinterested in finding the diseases that can be treatedwith the chemical “nivolumab”, but are not surewhich disease to search, they may input a query like“nivolumab, DISEASEORSYNDROME treat with

8https://www.elastic.co/

CHEMICAL”. EVIDENCEMINER automaticallyfinds all the textual evidence indicating a “treat-ment” relationship with the chemical “nivolumab”(Figure 2b).

Annotation Result Visualization. The annotationinterface shows all the annotated entities and re-lations for better visualization. For example, inFigure 3, we color all the annotated entities withdifferent colors for different types. We use fivedifferent colors for the five major biomedical entitytypes and two additional colors for two specific fine-grained types, “Gene or Genome” and “Disease orSyndrome”, since those two are the most frequentbiomedical entity types. In Figure 3, we see thatthe “melanoma” is colored as a “Disease or Syn-drome” and “nivolumab” is colored as a “Chem-ical”. We also list all the meta-pattern instancesand meta-patterns that match the sentences in thearticle. If the user clicks the meta-pattern instances,the corresponding sentences are also highlightedin the article. In Figure 3, a meta-pattern “DIS-EASEORSYNDROME patient treat with CHEM-ICAL” captures the entity pair “melanoma” and“nivolumab” in the article.

Entity and Relation Summarization. To makeour system more user-friendly and interesting, weadd analytic functionalities for the most frequententity and relation summarization. For example,in Figure 4, if scientists are interested in findingthe most frequent diseases, they can search “en-tity type = DISEASEORSYNDROME” in our an-alytic interface and see the top entities such astumor and breast cancer. Similarly, if scientists areinterested in finding the most frequent chemical-disease pairs with a treatment relation, they cansearch “pattern = DISEASEORSYNDROME treatwith CHEMICAL” in our analytic interface and seethe top entity pairs such as HCC&sorafenib. Moreinterestingly, if researchers are interested in find-ing the most frequent diseases that can be treatedby a specific chemical (e.g., nivolumab), they cansearch “entity = nivolumab & pattern = DISEASE-ORSYNDROME treat with CHEMICAL” in ouranalytic interface and see the most frequent dis-eases, such as melanoma and NSCLC, that can betreated with nivolumab. With these analytic func-tionalities, EVIDENCEMINER can help scientistsuncover important research issues, leading to moreeffective research and more in-depth quantitativeanalysis.

60

(a) Query: melanoma is treated with nivolumab

(b) Query: (nivolumab, DISEASEORSYNDROME treat with CHEMICAL)

Figure 2: The search interface with the textual evidence retrieved. The evidence score indicates the confidence ofeach retrieved sentence being a supporting evidence of the input query.

Figure 3: The annotation interface with all the entity and relation annotation results.

61

(a) (b) (c)

Figure 4: The analytic interface with the entity and relation summarization results. The queries used are (a)entity type=DISEASEORSYNDROME, (b) pattern=DISEASEORSYNDROME treat with CHEMICAL, and (c)entity=nivolumab&pattern=DISEASEORSYNDROME treat with CHEMICAL.

Method / nDCG @1 @5 @10BM25 0.714 0.720 0.746

LitSense 0.599 0.624 0.658EVIDENCEMINER 0.855 0.861 0.889

Table 2: Performance comparison of the textual evi-dence retrieval systems with nDCG@1,5,10.

4 Evaluation

To demonstrate the effectiveness of EVIDENCEM-INER in textual evidence retrieval, we compare itsperformance with the traditional BM25 (Robert-son et al., 2009) and a recent sentence-level searchengine, LitSense (Allot et al., 2019). The back-ground corpus is the same PubMed subset for allthe compared methods. We first ask domain ex-perts to generate 50 query statements based onthe relationships between three biomedical entitytypes (gene, chemical, and disease) in the Com-parative Toxicogenomics Database9. Then we askdomain experts to manually label the top-10 re-trieved evidence sentences by each method withthree grades indicating the confidence of the evi-dence. We use the average normalized DiscountedCumulative Gain (nDCG) score to evaluate the tex-tual evidence retrieval performance. In Table 2, weobserve that EVIDENCEMINER always achievesthe best performance compared with other meth-ods. It demonstrates the effectiveness of usingmeta-patterns to guide textual evidence discoveryin biomedical literature.

5 Further Development

In some cases, a strict query matching may notfind sufficiently high-quality answers due to thestringent search requirements or limited availableentities that match the search queries. In this case, a

9http://ctdbase.org

smart query processor should automatically kick-into do an approximate match, such as a graph-basedapproximate match or an embedding-based seman-tic match. In other cases, a user may query a setof entities (e.g., genes or diseases) or a timeline.We need to conduct a summary of the major dif-ferences among the set of entities or over time byanalyzing large text.

6 Conclusion

We build EVIDENCEMINER, a web-based systemfor textual evidence discovery for life sciences. Theretrieved evidence sentences can be easily locatedin the background corpora for better visualization.EVIDENCEMINER also includes analytic function-alities such as the most frequent entity and relationsummarization. We incorporated another corpuson COVID-19 in EVIDENCEMINER to help boostthe scientific discoveries (Wang et al., 2020b,a).We are further developing EVIDENCEMINER to bea more intelligent system that can assist in moreefficient and in-depth scientific discoveries.

Acknowledgment

Research was sponsored in part by US DARPAKAIROS Program No. FA8750-19-2-1004 andSocialSim Program No. W911NF-17-C-0099,National Science Foundation IIS 16-18481, IIS17-04532, and IIS-17-41317, and DTRA HD-TRA11810026. Any opinions, findings, and con-clusions or recommendations expressed herein arethose of the authors and should not be interpretedas necessarily representing the views, either ex-pressed or implied, of DARPA or the U.S. Gov-ernment. The U.S. Government is authorized toreproduce and distribute reprints for governmentpurposes notwithstanding any copyright annotationhereon.

62

ReferencesAlexis Allot, Qingyu Chen, Sun Kim, Roberto Vera Al-

varez, Donald C Comeau, W John Wilbur, and Zhiy-ong Lu. 2019. Litsense: making sense of biomedicalliterature at sentence level. Nucleic acids research.

Sihao Chen, Daniel Khashabi, Chris Callison-Burch,and Dan Roth. 2019. Perspectroscope: A windowto the world of diverse perspectives. page 129134.

Artem Chernodub, Oleksiy Oliynyk, Philipp Heidenre-ich, Alexander Bondarenko, Matthias Hagen, ChrisBiemann, and Alexander Panchenko. 2019. Targer:Neural argument mining at your fingertips. In ACL:System Demonstrations, pages 195–200.

Meng Jiang, Jingbo Shang, Taylor Cassidy, Xiang Ren,Lance M Kaplan, Timothy P Hanratty, and JiaweiHan. 2017. Metapad: Meta pattern discovery frommassive text corpora. In KDD, pages 877–886.ACM.

Ran Levy, Shai Gretz, Benjamin Sznajder, Shay Hum-mel, Ranit Aharonov, and Noam Slonim. 2017. Un-supervised corpus–wide claim detection. In Proc.Work. Arg. Min., pages 79–84.

Qi Li, Meng Jiang, Xikun Zhang, Meng Qu, Timo-thy P Hanratty, Jing Gao, and Jiawei Han. 2018a.Truepie: Discovering reliable patterns in pattern-based information extraction. In KDD, pages 1675–1684. ACM.

Qi Li, Xuan Wang, Yu Zhang, Qi Li, Fei Ling, CathyWu H, and Jiawei Han. 2018b. Pattern discov-ery for wide-window open information extraction inbiomedical literature. In BIBM. IEEE.

Marco Lippi and Paolo Torroni. 2016. Margot: A webserver for argumentation mining. Expert Systemswith Applications, 65:292–303.

Zhiyong Lu. 2011. Pubmed and beyond: a surveyof web tools for searching biomedical literature.Database, 2011.

Sarthak Majithia, Fatma Arslan, Sumeet Lubal,Damian Jimenez, Priyank Arora, Josue Caraballo,and Chengkai Li. 2019. Claimportal: Integratedmonitoring, searching, checking, and analytics offactual claims on twitter. In ACL: System Demon-strations, pages 153–158.

Hans-Michael Muller, Eimear E Kenny, and Paul WSternberg. 2004. Textpresso: an ontology-based in-formation retrieval and extraction system for biolog-ical literature. PLoS Biol., 2(11):e309.

Xiang Ren, Jiaming Shen, Meng Qu, Xuan Wang, Ze-qiu Wu, Qi Zhu, Meng Jiang, Fangbo Tao, SaurabhSinha, David Liem, et al. 2017. Life-inet: A struc-tured network-based knowledge exploration and an-alytics system for life sciences. In ACL: SystemDemonstrations, pages 55–60.

Stephen Robertson, Hugo Zaragoza, et al. 2009. Theprobabilistic relevance framework: Bm25 and be-yond. FnT Inf. Ret., 3(4):333–389.

Jingbo Shang, Jialu Liu, Meng Jiang, Xiang Ren,Clare R Voss, and Jiawei Han. 2018a. Automatedphrase mining from massive text corpora. TKDE.

Jingbo Shang, Liyuan Liu, Xiang Ren, Xiaotao Gu,Teng Ren, and Jiawei Han. 2018b. Learning namedentity tagger using domain-specific dictionary. InEMNLP. ACL.

Jiaming Shen, Jinfeng Xiao, Xinwei He, Jingbo Shang,Saurabh Sinha, and Jiawei Han. 2018. Entity setsearch of scientific literature: An unsupervised rank-ing approach. In SIGIR, pages 565–574. ACM.

Christian Stab, Johannes Daxenberger, Chris Stahlhut,Tristan Miller, Benjamin Schiller, ChristopherTauchmann, Steffen Eger, and Iryna Gurevych. 2018.Argumentext: Searching for arguments in heteroge-neous sources. In Proceedings of the 2018 confer-ence of the North American chapter of the associa-tion for computational linguistics: demonstrations,pages 21–25.

Henning Wachsmuth, Martin Potthast, KhalidAl Khatib, Yamen Ajjour, Jana Puschmann, JianiQu, Jonas Dorsch, Viorel Morari, Janek Bevendorff,and Benno Stein. 2017. Building an argumentsearch engine for the web. In Proceedings of the 4thWorkshop on Argument Mining, pages 49–59.

Xuan Wang, Weili Liu, Aabhas Chauhan, YingjunGuan, and Jiawei Han. 2020a. Automatic textualevidence mining in covid-19 literature.

Xuan Wang, Xiangchen Song, Yingjun Guan,Bangzheng Li, and Jiawei Han. 2020b. Com-prehensive named entity recognition on cord-19with distant or weak supervision. arXiv preprintarXiv:2003.12218.

Xuan Wang, Yu Zhang, Qi Li, Yinyin Chen, and JiaweiHan. 2018a. Open information extraction with meta-pattern discovery in biomedical literature. In BCB,pages 291–300. ACM.

Xuan Wang, Yu Zhang, Qi Li, Xiang Ren, JingboShang, and Jiawei Han. 2019. Distantly supervisedbiomedical named entity recognition with dictionaryexpansion. In 2019 IEEE International Conferenceon Bioinformatics and Biomedicine (BIBM), pages496–503. IEEE.

Xuan Wang, Yu Zhang, Qi Li, Cathy H. Wu, and JiaweiHan. 2018b. PENNER: pattern-enhanced nestednamed entity recognition in biomedical literature. InBIBM, pages 540–547.

Chih-Hsuan Wei, Hung-Yu Kao, and Zhiyong Lu. 2013.Pubtator: a web-based text mining tool for assist-ing biocuration. Nucleic Acids Res., 41(W1):W518–W522.